Electronic device, and method for editing indicator by electronic device

Abstract

This electronic device may comprise: a processor; a memory for storing instructions; a display; and a camera. The instructions, when executed by the processor, enable the electronic device to: analyze at least one of a color of a background screen displayed on the display, a color of at least one indicator, or a position at which the at least one indicator is displayed in the background screen; and generate a prompt instructing a change of an attribute of the at least one indicator so that the at least one indicator can be recognized as distinct from the background on the basis of analysis information.

Description

How to edit electronic devices and indicators of electronic devices

[0001] This article relates to electronic devices, including wearable devices, and to editing indicators (e.g., app icons or widgets) that appear on the background or lock screen of the electronic device (e.g., images, videos, or the real-world environment of XR devices).

[0002] An artificial neural network (ANN) refers to a computational architecture that models the biological brain. Deep learning and machine learning can be implemented based on ANNs. As an example of an ANN, a deep neural network (DNN) or deep learning can have a multilayer structure containing multiple layers.

[0003] Artificial intelligence (AI) models are being used in a variety of ways to analyze visual and audio data. To ensure effective operation of AI models on mobile devices, active research and development is underway on hardware technologies related to AI models. For example, research is being conducted on optimizing the MAC (multiply-accumulate) operation performed in deep learning AI models, as well as on improving hardware architectures that take AI models into account.

[0004] Additionally, the data processing system may include at least one processor, commonly known as a central processing unit (CPU). Such a data processing system may also include at least one other processor, such as a neural processing unit (NPU), used for various types of specialized processing.

[0005] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0006] Electronic devices, including XR (extended reality) devices, can display indicators (such as app icons, widgets, text, or overlay UI) on screens that change in real time. However, the colors and patterns of these indicators that may appear on top of the background screen as the background changes, as well as the user's vision, can make it difficult to recognize these indicators on the background screen.

[0007] An electronic device according to this document can improve the UI so that the user can easily recognize the indicator on the background screen by adjusting at least one of the color or pattern of the indicator (e.g., app icon, widget, text) based on the properties of the background and the user's vision-related information (e.g., color blindness).

[0008] An electronic device may include a processor, a memory storing instructions, a display, and a camera. The instructions, when executed by the processor, may cause the electronic device to analyze at least one of the color of a background screen displayed on the display, the color of at least one indicator, or the position at which the at least one indicator is displayed within the background screen, and generate a prompt that instructs to change an attribute of the at least one indicator so that the at least one indicator can be recognized as distinct from the background based on the analyzed information.

[0009] A method of operating an electronic device may include an operation of analyzing at least one of a color of a background screen displayed on a display, a color of at least one indicator, or a position at which at least one indicator is displayed within the background screen, an operation of generating a prompt that instructs to change an attribute of at least one indicator so that the at least one indicator can be recognized as distinct from the background based on the analyzed information, and an operation of controlling to change an attribute of the at least one indicator based on the generated prompt.

[0010] An electronic device according to this document can adjust at least one of the color or pattern of an indicator (e.g., an app icon, a widget, a text) based on the properties of the background and information related to the user's eyesight (e.g., color blindness). An electronic device according to this document can improve the UI and enhance user convenience by adjusting at least one of the color or pattern of an indicator (e.g., an app icon, a widget, a text) so that the user can easily recognize the indicator on the background screen.

[0011] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0012] FIG. 2A is a perspective view schematically illustrating a configuration of a wearable electronic device according to one embodiment of the present disclosure.

[0013] FIGS. 2B and 2C are perspective views schematically illustrating the front and back of a wearable electronic device according to one embodiment of the present disclosure.

[0014] FIG. 3 is a diagram schematically illustrating the configuration of an electronic device according to one embodiment of the present disclosure.

[0015] FIG. 4a illustrates a situation in which an indicator (e.g., an application icon, a widget) is displayed on a display of an electronic device according to a comparative example.

[0016] FIG. 4b illustrates a situation in which an electronic device according to a comparative example displays an indicator (e.g., an application icon, a widget) against a background of an actual space.

[0017] FIG. 4c illustrates a situation in which an indicator is displayed on a display of an electronic device according to a comparative example, but the background is changed so that the indicator is not easily recognized by the user.

[0018] FIG. 5a illustrates a situation in which the background screen of an electronic device is changed according to one embodiment.

[0019] FIG. 5b illustrates an embodiment in which an indicator is displayed on a background screen in a situation in which the background screen of an electronic device is changed according to one embodiment.

[0020] FIG. 5c illustrates an embodiment in which, in a situation where the background screen of an electronic device is changed, at least one indicator has a characteristic that is changed so that it can be recognized by a user on the background screen.

[0021] FIG. 6 is a flowchart illustrating a method for editing indicators of an electronic device according to various embodiments.

[0022] FIG. 7 is a flowchart illustrating a method for editing indicators of an electronic device according to various embodiments.

[0023] Figure 8 is a block diagram of a natural language understanding module (800).

[0024] FIG. 9 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0025] Fig. 10 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0026] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0027] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0028] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0029] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0030] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0031] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0032] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0033] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0034] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0035] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0036] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0037] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0038] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0039] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0040] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0041] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0042] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0043] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0044] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0045] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0046] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0047] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0048] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0049] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0050] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0051] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0052] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0053] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0054] FIG. 2A is a perspective view schematically illustrating a configuration of a wearable electronic device according to one embodiment of the present disclosure.

[0055] FIGS. 2B and 2C are perspective views schematically illustrating the front and back of a wearable electronic device according to one embodiment of the present disclosure.

[0056] FIG. 2A is a perspective view schematically illustrating a configuration of a wearable electronic device according to one embodiment of the present disclosure.

[0057] The wearable electronic device (200) of FIG. 2A may include embodiments described in the electronic device (101) of FIG. 1. The wearable electronic device (200) may include augmented reality (AR) glasses or smart glasses in the form of glasses.

[0058] Referring to FIG. 2A, a wearable electronic device (200) according to various embodiments may include a bridge (bridge, 201), a first rim (rim, 210), a second rim (220), a first end piece (end piece, 230), a second end piece (240), a first temple (temple, 250), and / or a second temple (260).

[0059] In one embodiment, the bridge (201) can connect the first rim (210) and the second rim (220). The bridge (201) can be positioned over the user's nose when the user wears the wearable electronic device (200). The bridge (201) can separate the first rim (210) and the second rim (220) based on the user's nose.

[0060] According to various embodiments, the bridge (201) may include a camera module (203), a first gaze tracking camera (205), a second gaze tracking camera (207), and / or an audio module (209).

[0061] According to various embodiments, the camera module (203) (e.g., the camera module (180) of FIG. 1) can capture a front view (e.g., in the -y-axis direction) of a user (e.g., a user of a wearable electronic device (200)) and obtain image data. The camera module (203) can capture an image corresponding to the user's field of view (FoV) or measure a distance to a subject (e.g., an object). The camera module (203) can include an RGB camera, a high resolution (HR) camera, and / or a photo video (PV) camera. The camera module (203) can include a color camera having an auto focus (AF) function and an optical image stabilization (OIS) function to obtain high-quality images.

[0062] According to various embodiments, the first gaze tracking camera (205) and the second gaze tracking camera (207) can determine the gaze of the user. The first gaze tracking camera (205) and the second gaze tracking camera (207) can capture the pupil of the user's eyes in a direction opposite to the capturing direction of the camera module (203). For example, the first gaze tracking camera (205) can partially capture the left eye of the user, and the second gaze tracking camera (207) can partially capture the right eye of the user. The first gaze tracking camera (205) and the second gaze tracking camera (207) can detect the pupil of the user's eyes (e.g., the left eye and the right eye) and track the gaze direction. The tracked gaze direction can be utilized to move the center of a virtual image including a virtual object in response to the gaze direction. The first gaze tracking camera (205) and / or the second gaze tracking camera (207) can track the user's gaze using, for example, at least one of an EOG sensor (electro-oculography or electrooculogram), a coil system, a dual Purkinje system, bright pupil systems, or dark pupil systems.

[0063] According to various embodiments, an audio module (209) (e.g., audio module (170) of FIG. 1) may be positioned between the first gaze tracking camera (205) and the second gaze tracking camera (207). The audio module (209) may convert a user's voice into an electrical signal or convert an electrical signal into sound. The audio module (209) may include a microphone.

[0064] According to one embodiment, the first rim (210) and the second rim (220) may form a frame (e.g., a glasses frame) of a wearable electronic device (200) (e.g., an AR glass). The first rim (210) may be disposed in a first direction (e.g., an x-axis direction) of the bridge (201). The first rim (210) may be disposed at a position corresponding to the user's left eye. The second rim (220) may be disposed in a second direction (e.g., a -x-axis direction) of the bridge (201) opposite to the first direction (e.g., an x-axis direction). The second rim (220) may be disposed at a position corresponding to the user's right eye. The first rim (210) and the second rim (220) may be formed of a metal material and / or a non-conductive material (e.g., a polymer).

[0065] According to various embodiments, the first rim (210) may surround and support at least a portion of the first glass (215) (e.g., the first display) disposed on the inner surface. The first glass (215) may be positioned in front of the user's left eye. The second rim (220) may surround and support at least a portion of the second glass (225) (e.g., the second display) disposed on the inner surface. The second glass (225) may be positioned in front of the user's right eye. The user of the wearable electronic device (200) may view a foreground (e.g., a real image) of an external object (e.g., a subject) through the first glass (215) and the second glass (225). The wearable electronic device (200) may implement augmented reality by superimposing a virtual image on the foreground (e.g., the real image) of the external object and displaying it.

[0066] According to various embodiments, the first glass (215) and the second glass (225) may include a projection type transparent display. The first glass (215) and the second glass (225) may each form a reflective surface as a transparent plate (or transparent screen), and an image generated from the wearable electronic device (200) may be reflected (e.g., total internal reflection) through the reflective surface and incident on the left and right eyes of the user. In one embodiment, the first glass (215) may include an optical waveguide that transmits light generated from a light source of the wearable electronic device (200) to the left eye of the user. For example, the optical waveguide may be formed of a glass, plastic, or polymer material, and may include a nano-pattern (e.g., a grating structure or a mesh structure having a polygonal or curved shape) formed on the inside or the surface of the first glass (215). The optical waveguide may include at least one diffractive element (e.g., a Diffractive Optical Element (DOE), a Holographic Optical Element (HOE)) or at least one reflective element (e.g., a reflective mirror). The optical waveguide may guide display light emitted from a light source toward a user's eye using at least one diffractive element or reflective element included in the optical waveguide. In various embodiments, the diffractive element may include an input / output optical element, and the reflective element may include a total internal reflection (TIR). For example, light emitted from a light source may be guided along an optical path to the optical waveguide through an input optical element, and light traveling inside the optical waveguide may be guided toward a user's eye through an output optical element. The second glass (225) may be implemented in substantially the same manner as the first glass (215).

[0067] According to various embodiments, the first glass (215) and the second glass (225) may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED). Although not shown, when the first glass (215) and the second glass (225) are formed of one of a liquid crystal display, a digital mirror display, or a silicon liquid crystal display, the wearable electronic device (200) may include a light source that irradiates light to the screen output areas of the first glass (215) and the second glass (225). In another embodiment, if the first glass (215) and the second glass (225) are capable of generating light on their own, for example, if they are made of either organic light emitting diodes or micro LEDs, the wearable electronic device (200) can provide a good quality virtual image to the user even without including a separate light source.

[0068] According to various embodiments, the first rim (210) may include a first microphone (211), a first recognition camera (213), a first light-emitting device (217), and / or a first display module (219). The second rim (220) may include a second microphone (221), a second recognition camera (223), a second light-emitting device (227), and / or a second display module (229).

[0069] In various embodiments, the first light emitting device (217) and the first display module (219) may be included in the first end piece (230), and the second light emitting device (227) and the second display module (229) may be included in the second end piece (240).

[0070] According to various embodiments, the first microphone (211) and / or the second microphone (221) may receive the voice of a user of the wearable electronic device (200) and convert it into an electrical signal.

[0071] According to various embodiments, the first recognition camera (213) and / or the second recognition camera (223) can recognize the surrounding space of the wearable electronic device (200). The first recognition camera (213) and / or the second recognition camera (223) can detect a user's gesture within a certain distance (e.g., a certain space) of the wearable electronic device (200). The first recognition camera (213) and / or the second recognition camera (223) can include a global shutter (GS) camera capable of reducing the RS (rolling shutter) phenomenon in order to detect and track the user's rapid hand movements and / or subtle movements of the fingers. The wearable electronic device (200) can detect an eye corresponding to the dominant eye and / or the auxiliary eye among the user's left eye and / or right eye by using the first gaze tracking camera (205), the second gaze tracking camera (207), the first recognition camera (213), and / or the second recognition camera (223). For example, the wearable electronic device (200) can detect the eye corresponding to the dominant eye and / or the auxiliary eye based on the user's gaze direction toward an external object or a virtual object.

[0072] According to various embodiments, the first light emitting device (217) and / or the second light emitting device (227) may emit light to increase the accuracy of the camera module (203), the first gaze tracking camera (205), the second gaze tracking camera (207), the first recognition camera (213), and / or the second recognition camera (223). The first light emitting device (217) and / or the second light emitting device (227) may be used as an auxiliary means to increase the accuracy when capturing the user's pupils using the first gaze tracking camera (205) and / or the second gaze tracking camera (207). The first light emitting device (217) and / or the second light emitting device (227) may be used as an auxiliary means when capturing the user's gesture using the first recognition camera (213) and / or the second recognition camera (223), when it is not easy to detect an object to be captured (e.g., a subject) due to a dark environment or mixing and reflection of multiple light sources. The first light emitting device (217) and / or the second light emitting device (227) may include, for example, an LED, an IR LED, or a xenon lamp.

[0073] According to various embodiments, the first display module (219) and / or the second display module (229) can emit light and transmit the light to the user's left eye and / or right eye using the first glass (215) and / or the second glass (225). The first glass (215) and / or the second glass (225) can display various image information using the light emitted through the first display module (219) and / or the second display module (229). The first display module (219) and / or the second display module (229) can include the display module (160) of FIG. 1. The wearable electronic device (200) can display a foreground for an external object and an image emitted through the first display module (219) and / or the second display module (229) by overlapping them through the first glass (215) and / or the second glass (225).

[0074] In one embodiment, the first end piece (230) may be coupled to a portion (e.g., in the x-axis direction) of the first rim (210). The second end piece (240) may be coupled to a portion (e.g., in the -x-axis direction) of the second rim (220). In various embodiments, the first light-emitting device (217) and the first display module (219) may be included in the first end piece (230). The second light-emitting device (227) and the second display module (229) may be included in the second end piece (240).

[0075] According to various embodiments, the first end piece (230) can connect the first rim (210) and the first temple (250). The second end piece (240) can connect the second rim (220) and the second temple (260).

[0076] In one embodiment, the first temple (250) may be operatively connected to the first end piece (230) using a first hinge portion (255). The first hinge portion (255) may be configured to be rotatable so that the first temple (250) can be folded or unfolded with respect to the first rim (210). The first temple (250) may extend, for example, along the left side of the user's head. A distal portion (e.g., in the y-axis direction) of the first temple (250) may be configured in a curved shape so as to be supported, for example, on the user's left ear when the user wears the wearable electronic device (200). The second temple (260) may be operatively connected to the second end piece (240) using a second hinge portion (265). The second hinge portion (265) may be configured to be rotatable so that the second temple (260) can be folded or unfolded relative to the second rim (220). The second temple (260) may extend, for example, along the right side of the user's head. The distal portion (e.g., in the y-axis direction) of the second temple (260) may be configured in a curved shape so as to be supported, for example, on the user's right ear when the user wears the wearable electronic device (200).

[0077] According to various embodiments, the first temple (250) may include a first printed circuit board (251), a first audio output module (253) (e.g., audio output module (155) of FIG. 1), and / or a first battery (257) (e.g., battery (189) of FIG. 1). The second temple (260) may include a second printed circuit board (261), a second audio output module (263) (e.g., audio output module (155) of FIG. 1), and / or a second battery (267) (e.g., battery (189) of FIG. 1).

[0078] According to various embodiments, various electronic components (e.g., at least some of the components included in the electronic device (101) of FIG. 1), such as the processor (120), memory (130), interface (177), and / or wireless communication module (192) disclosed in FIG. 1, may be disposed on the first printed circuit board (251) and / or the second printed circuit board (261). The processor may include, for example, one or more of a central processing unit, an application processor, a graphics processing unit, an image signal processor, a sensor hub processor, or a communication processor. The first printed circuit board (251) and / or the second printed circuit board (261) may include, for example, a printed circuit board (PCB), a flexible PCB (FPCB), or a rigid-flexible PCB (RFPCB). In some embodiments, the first printed circuit board (251) and / or the second printed circuit board (261) may include a Main PCB, a slave PCB arranged to partially overlap the Main PCB, and / or an interposer substrate between the Main PCB and the slave PCB. The first printed circuit board (251) and / or the second printed circuit board (261) may be electrically connected to other components (e.g., a camera module (203), a first gaze tracking camera (205), a second gaze tracking camera (207), an audio module (209), a first microphone (211), a first recognition camera (213), a first light-emitting device (217), a first display module (219), a second microphone (221), a second recognition camera (223), a second light-emitting device (227), a second display module (229), a first audio output module (253), and / or a second audio output module (263)) using electrical paths such as FPCBs and / or cables.For example, the FPCB and / or cable may be positioned on at least a portion of the first rim (210), the bridge (201), and / or the second rim (220). In some embodiments, the wearable electronic device (200) may include only one of the first printed circuit board (251) and the second printed circuit board (261).

[0079] According to various embodiments, the first audio output module (253) and / or the second audio output module (263) may transmit audio signals to the user's left and / or right ears. The first audio output module (253) and / or the second audio output module (263) may include, for example, a piezo speaker (e.g., a bone conduction speaker) that transmits audio signals without a speaker hole. In some embodiments, the wearable electronic device (200) may include only one of the first audio output module (253) and the second audio output module (263).

[0080] According to various embodiments, the first battery (257) and / or the second battery (267) may supply power to the first printed circuit board (251) and / or the second printed circuit board (261) using a power management module (e.g., the power management module (188) of FIG. 1 ). The first battery (257) and / or the second battery (267) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell. In some embodiments, the wearable electronic device (200) may include only one of the first battery (257) and the second battery (267).

[0081] According to various embodiments, the wearable electronic device (200) may include a sensor module (e.g., the sensor module (176) of FIG. 1). The sensor module may generate an electrical signal or data value corresponding to an internal operating state of the wearable electronic device (200) or an external environmental state. The sensor module may further include, for example, at least one of a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a color sensor, an infrared (IR) sensor, a biometric sensor (e.g., an HRM sensor), a temperature sensor, a humidity sensor, or an illuminance sensor. In some embodiments, the sensor module may recognize biometric information of a user by using various biometric sensors (or biometric recognition sensors), such as an e-nose sensor, an electromyography sensor, an electroencephalogram sensor, an ECG sensor, or an iris sensor.

[0082] According to various embodiments, although the wearable electronic device (200) is described as a device that displays augmented reality using the first glass (215) and the second glass (225) in the above description, it is not limited thereto and may be a device that displays virtual reality (VR).

[0083] In FIG. 2A according to various embodiments, the wearable electronic device (200) is described as a device that displays augmented reality or virtual reality using the first glass (215) and the second glass (225), but is not limited thereto. For example, the wearable electronic device (200) may include a video see-through (VST) device. In this regard, various embodiments will be described in FIGS. 2B and 2C described below.

[0084] FIGS. 2B and 2C are perspective views schematically illustrating the front and back of a wearable electronic device (270) according to one embodiment of the present disclosure.

[0085] Referring to FIGS. 2B and 2C, the wearable electronic device (270) may be provided with a plurality of cameras (e.g., a first camera (273) (e.g., a first recognition camera (213) of FIG. 2A) and a second camera (274) (e.g., a second recognition camera (223) of FIG. 2A)) corresponding to the front direction (e.g., -y direction, a user's gaze direction) of the wearable electronic device (270). For example, the wearable electronic device (270) may include a first camera (273) corresponding to the user's left eye and a second camera (274) corresponding to the user's right eye. The wearable electronic device (270) may capture an external environment in the front direction (e.g., -y direction) of the wearable electronic device (270) using the first camera (273) and the second camera (274). The wearable electronic device (270) may be exposed to the external environment. It may include a first side (271) (e.g., front side) (e.g., see FIG. 2b) and a second side (272) (e.g., back side) (e.g., see FIG. 2c) that is in close contact with the user's skin when worn, but is not exposed to the external environment. For example, when the wearable electronic device (270) is worn on the user's face, the first side (271) of the wearable electronic device (270) may be exposed to the external environment, and the second side (272) of the wearable electronic device (270) may be in a state of at least partially in close contact with the user's face.

[0086] In one embodiment, at least one distance sensor (281, 282, 283, and / or 284) may be disposed on a first surface (271) of the wearable electronic device (270). For example, at least one distance sensor (281, 282, 283, and / or 284) may measure a distance to at least one object disposed around the wearable electronic device (270). The at least one distance sensor (281, 282, 283, and / or 284) may include an infrared sensor, an ultrasonic sensor, and / or a light detection and ranging (LiDAR) sensor. The at least one distance sensor (281, 282, 283, and / or 284) may be implemented based on an infrared sensor, an ultrasonic sensor, and / or a LiDAR sensor.

[0087] In FIG. 2b according to various embodiments, four distance sensors (281, 282, 283, and 284) are shown arranged on the first side (271) of the wearable electronic device (270), but are not limited thereto.

[0088] In one embodiment, the wearable electronic device (270) may have multiple displays (e.g., a first display (275) (e.g., the first glass (215) of FIG. 2A) and a second display (276) (e.g., the second glass (225) of FIG. 2A)) arranged in response to the rear direction of the wearable electronic device (270) (e.g., the +y direction, the direction opposite to the user's gaze direction). For example, the second side (272) (e.g., the rear) of the wearable electronic device (270) may have a first display (275) arranged in response to the user's left eye and a second display (276) arranged in response to the user's right eye. For example, when the wearable electronic device (270) is worn on the user's face, the first display (275) may be arranged in response to the user's left eye, and the second display (276) may be arranged in response to the user's right eye.

[0089] In one embodiment, the wearable electronic device (270) may have a plurality of gaze tracking cameras (e.g., a first gaze tracking camera (291) (e.g., the first gaze tracking camera (205) of FIG. 2A) or a second gaze tracking camera (292) (e.g., the second gaze tracking camera (207) of FIG. 2A) disposed at least partially on the second face (272). For example, the plurality of gaze tracking cameras (291, 292) may track eye movements of the user. The first gaze tracking camera (291) may track eye movements of the user, and the second gaze tracking camera (292) may track eye movements of the user. In one embodiment, the wearable electronic device (270) may determine a direction in which the user is looking based on eye movements tracked using the plurality of gaze tracking cameras (291, 292).

[0090] In one embodiment, a second face (272) of a wearable electronic device (270) may be at least partially provided with a plurality of face recognition cameras (e.g., a first face recognition camera (295) or a second face recognition camera (296)). For example, the plurality of face recognition cameras (295, 296) may recognize a user's face when the wearable electronic device (270) is worn on the user's face. In one embodiment, the wearable electronic device (270) may also use the plurality of face recognition cameras (295, 296) to determine whether the wearable electronic device (270) is worn on the user's face.

[0091] FIG. 3 is a diagram schematically illustrating the configuration of an electronic device according to one embodiment of the present disclosure.

[0092] Referring to FIG. 3, an electronic device (101) according to one embodiment of the present disclosure may include a display module (160), a camera module (180), a memory (130), and / or a processor (120). According to one embodiment, the electronic device (101) may include all or at least a portion of the components of the electronic device (101) as described in the description with reference to FIG. 1.

[0093] According to one embodiment, the display module (160) may include a configuration identical or similar to the display module (160) of FIG. 1. According to one embodiment, the display module (160) may include a display and may visually provide various information to an external party (e.g., a user) of the electronic device (101). According to one embodiment, the display module (160) may visually provide various information (e.g., content, images, or videos) related to an application being executed and its use under the control of the processor (120).

[0094] According to one embodiment, the display module (160) may include a touch sensor, a pressure sensor capable of measuring the intensity of a touch, and / or a touch panel (e.g., a digitizer) for detecting a magnetic stylus pen. According to one embodiment, the display module (160) may detect a touch input and / or a hovering input (or a proximity input) by measuring a change in a signal (e.g., voltage, light intensity, resistance, electromagnetic signal, and / or charge amount) for a specific location of the display module (160) based on the touch sensor, the pressure sensor, and / or the touch panel. According to one embodiment, the display module (160) may include a liquid crystal display (LCD), an organic light emitting diode (OLED), or an active matrix organic light emitting diode (AMOLED). According to one embodiment, the display module (160) may include a flexible display.

[0095] According to one embodiment, the camera module (180) may correspond to the camera module (180) of FIG. 1 or FIG. 2. According to one embodiment, the camera module (180) may, when activated, capture a subject and transmit the relevant result (e.g., captured image) to the processor (120) and / or the display module (160).

[0096] According to one embodiment, the memory (130) may correspond to the memory (130) of FIG. 1. According to one embodiment, the memory (130) may store various data used by the electronic device (101). In one embodiment, the data may include input data or output data for an application (e.g., the program (140) of FIG. 1) and commands related to the application. In one embodiment, the data may include camera image data acquired through a camera module. In one embodiment, the data may include various learning data acquired based on user learning through interaction with the user. In one embodiment, the data may include various schemas (or algorithms, models, networks, or functions) for supporting artificial intelligence-based image processing.

[0097] According to one embodiment, the memory (130) may store instructions that, when executed, cause the processor (120) to operate. For example, the application may be stored as software (e.g., the program (140) of FIG. 1) on the memory (130) and may be executable by the processor (120). According to one embodiment, the application may be a variety of applications that may provide various functions or services (e.g., an image capturing function based on artificial intelligence) in the electronic device (101).

[0098] According to one embodiment, the processor (120) may perform an application layer processing function requested by a user of the electronic device (101). According to one embodiment, the processor (120) may provide control and commands of functions for various blocks of the electronic device (101). According to one embodiment, the processor (120) may perform operations or data processing related to control and / or communication of each component of the electronic device (101). For example, the processor (120) may include at least some of the configurations and / or functions of the processor (120) of FIG. 1. The processor (120) may be operatively connected to, for example, components of the electronic device (101). The processor (120) may load commands or data received from other components of the electronic device (101) into the memory (130), process the commands or data stored in the memory (130), and store result data.

[0099] According to one embodiment, the processor (120) may be an application processor (AP). According to one embodiment, the processor (120) may be a system semiconductor that is responsible for the operation and multimedia driving functions of the electronic device (101). According to one embodiment, the processor (120) may be implemented in the form of a system-on-chip (SoC), and may include a technology-intensive semiconductor chip that integrates multiple semiconductor technologies into one and implements system blocks into a single chip. According to one embodiment, the system blocks of the processor (120) may include a graphics processing unit (GPU) (310), an image signal processor (ISP) (320), a central processing unit (CPU) (330), a neural processing unit (NPU) (340), a digital signal processor (350), a modem (360), a connectivity (370), and / or a security (380) block, as illustrated in FIG. 3.

[0100] In one embodiment, the GPU (310) may be responsible for graphics processing. In one embodiment, the GPU (310) may receive commands from the CPU (330) and perform graphics processing to express the shape, position, color, shading, movement, and / or texture of objects (or objects) on the display.

[0101] In one embodiment, the ISP (320) may be responsible for image processing and correction of images and videos. In one embodiment, the ISP (320) may correct unprocessed data (e.g., input data, raw data, original data, or raw data) transmitted from the image sensor of the camera module (180) (e.g., the image sensor (230) of FIG. 2) to generate an image in a form preferred by the user. In one embodiment, the ISP (320) may correct physical limitations that may occur in the camera module (180) and interpolate R / G / B (red, green, blue) values ​​and remove noise. In one embodiment, the ISP (320) may perform post-processing, such as adjusting partial brightness of the image and emphasizing detailed parts. For example, the ISP (320) may independently perform a process of tuning and correcting the image quality of the image acquired through the camera module (180) to generate a result preferred by the user.

[0102] According to one embodiment, the ISP (320) may support artificial intelligence-based image processing technology to improve image quality, speed up image processing, and reduce power consumption (e.g., low power consumption). For example, the ISP (320) may improve image quality while maintaining low power consumption, and may support artificial intelligence-based image capture for this purpose. According to one embodiment, the ISP (320) may support artificial intelligence-based image processing related to improving the image quality of a video in a dark, low-light environment. According to one embodiment, the ISP (320) may support scene segmentation (e.g., image segmentation) technology to recognize and / or classify parts of a scene being captured in conjunction with the NPU (340). For example, the ISP (320) may include a function to process objects such as the sky, bushes, and / or skin by applying different parameters to each object. According to one embodiment, the ISP (320) can detect and display a human face when capturing an image through an artificial intelligence function, or adjust the brightness, focus, and / or color of the image using the coordinates and information of the face.

[0103] According to one embodiment, the configuration and detailed operation of the ISP (320) of the processor (120) are described with reference to the drawings described below.

[0104] According to one embodiment, the CPU (330) may perform a role corresponding to the processor (120). According to one embodiment, the CPU (330) may decode a user's command, perform arithmetic and logical operations, and / or data processing. For example, the CPU (330) may be responsible for functions such as memory, interpretation, calculation, and control. According to one embodiment, the CPU (330) may control the overall functions of the electronic device (101). For example, the CPU (330) may execute all software (e.g., applications) of the electronic device (101) on top of an operating system (OS) and control hardware devices.

[0105] According to one embodiment, the NPU (340) may be responsible for processing optimized for artificial intelligence deep learning algorithms. According to one embodiment, the NPU (340) is a processor optimized for deep learning algorithm operations (e.g., artificial intelligence operations) and can process big data quickly and efficiently like a human neural network. For example, the NPU (340) may be mainly used for artificial intelligence operations. According to one embodiment, the NPU (340) may recognize objects, environments, and / or people in the background when taking a video through the camera module (180) and automatically adjust the focus, automatically switch the shooting mode of the camera module (180) to food mode when taking a picture of food, and / or be responsible for processing only the deletion of unnecessary subjects from the taken result.

[0106] According to one embodiment, the electronic device (101) can support integrated machine learning processing by interacting with all processors such as the GPU (310), the ISP (320), the CPU (330), and the NPU (340).

[0107] In one embodiment, the DSP (350) may represent an integrated circuit that facilitates rapid processing of digital signals. In one embodiment, the DSP (350) may perform the function of converting analog signals into digital signals and performing high-speed processing.

[0108] According to one embodiment, the modem (360) may perform a role that enables the use of various communication functions in the electronic device (101). For example, the modem (360) may support communications such as phone calls and data transmission and reception by exchanging signals with a base station. According to one embodiment, the modem (360) may include an integrated modem (e.g., a cellular modem, an LTE modem, a 5G modem, a 5G-Advanced modem, and / or a 6G modem) that supports communication technologies such as LTE and 2G to 5G. According to one embodiment, the modem (360) may include an artificial intelligence modem that applies an artificial intelligence algorithm.

[0109] In one embodiment, the connectivity (370) may support wireless data transmission based on IEEE 802.11. In one embodiment, the connectivity (370) may support communication services based on IEEE 802.11 (e.g., Wi-Fi) and / or 802.15 (e.g., Bluetooth, ZigBee, or UWB). For example, the connectivity (370) may support communication services targeting an unspecified number of people in a localized area, such as indoors, using an unlicensed band.

[0110] According to one embodiment, security (380) may provide an independent security execution environment between data or services stored in the electronic device (101). According to one embodiment, security (380) may play a role in preventing external hacking through software and hardware security during the process of user authentication when providing services such as biometrics, mobile identification, and / or payment of the electronic device (101). For example, security (380) may provide an independent security execution environment for device security for reinforcing the security of the electronic device (101) itself and for security services based on user information such as mobile identification, payment, and car keys in the electronic device (101).

[0111] According to one embodiment of the present disclosure, the processor (120) (e.g., the ISP (320)) may include processing circuitry and / or executable program elements. According to one embodiment, the processor (120) (e.g., the ISP (320)) may control (or process) operations related to supporting image processing of a video in an AI manner during video capturing (e.g., recording) based on the processing circuitry and / or executable program elements.

[0112] According to one embodiment, the processor (120) (e.g., ISP (320)) may perform an operation of receiving raw data (e.g., input data, unprocessed data, or original data) through the camera module (180) while a user is capturing a video. According to one embodiment, the processor (120) (e.g., ISP (320)) may perform an operation of determining a parameter corresponding to a setting for capturing the video or a change in the setting. According to one embodiment, the processor (120) (e.g., ISP (320)) may perform an image processing operation including demosaicing and scaling the raw data based on the parameter. According to one embodiment, the processor (120) (e.g., ISP (320)) may perform an operation of outputting image data according to the image processing.

[0113] According to one embodiment, the detailed operation of the processor (120) and / or the ISP (320) of the electronic device (101) is described with reference to the drawings described below.

[0114] According to one embodiment, the operations performed by the processor (120) (e.g., ISP (320)) may be implemented as a recording medium (or a computer program product). For example, the recording medium may include a non-transitory computer-readable recording medium having recorded thereon a program for executing various operations performed by the processor (120).

[0115] The embodiments described in the present disclosure may be implemented in a computer-readable recording medium using software, hardware, or a combination thereof. In a hardware implementation, the operations described in one embodiment may be implemented using at least one of Application Specific Integrated Circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and / or other electrical units for performing functions.

[0116] LLM refers to a language model based on an artificial neural network that has learned from a large amount of text data through pre-training. LLMs can contain significantly more parameters (e.g., over 10 billion) than conventional language models. LLMs can utilize a transformer artificial neural network structure based on the attention mechanism.

[0117] An attention mechanism can refer to a technology that helps an AI model focus (attention) on important parts of input data. The attention mechanism can be used to predict output data by predicting the degree to which a portion of time-series input data (e.g., input data such as voice or video, or input data of a neural network layer) contributes to the intermediate or final output of the neural network. While recurrent neural networks (RNNs), which sequentially process each element of a sequence, exhibit poor prediction performance when there is information dependency between long time-series distances, the attention mechanism can consider information dependency between long time-series distances by controlling the degree of weight concentration (attention) within the overall (or partial) context of the input data.

[0118] A transformer can be structured as an encoder-decoder. The encoder processes input data and outputs compressed information (e.g., a contextual representation), and the decoder processes the compressed information and outputs token-based data. Each encoder and decoder can include an independent attention network, and a cross-attention network connecting the encoder and decoder can also be included.

[0119] For example, LLM learning may involve pre-training and / or fine-tuning. Pre-training is the process by which LLM acquires general linguistic knowledge using large amounts of text data. For example, this may involve self-supervised learning, where the LLM predicts the next word using the previous word sequence in the text string. Fine-tuning is the process by which LLM is trained to be suitable for a specific domain (e.g., chatbot, translation, summarization, or Q&A) or task. LLM may further undergo supervised learning (or adaptive learning) based on the pre-trained model using a dataset tailored to the domain's purpose. LLM can perform tasks with text inputs containing natural language, called prompts.

[0120] For example, fine-tuning can be omitted when learning LLMs. Users can control the prompts provided to LLMs to enhance their performance on desired tasks. Similar to in-context learning or zero-shot / few-shot learning, prompts can be supplemented with examples of tasks and / or guidance for performing them. Publicly available LLMs include BERT (bidirectional encoder representations from transformer) and GPT (generative pre-trained transformer).

[0121] The term "LLM" can refer to the language neural network model itself, but it can also refer to the model of an LLM-based application (e.g., chatbot, translation, summarization, text classification, or sentence generation). For example, an LLM-based chatbot or LLM-based translator could also be referred to as "LLM."

[0122] "LLM" may also include an inference engine utilizing the LLM neural network model. For example, "entering an input prompt into the LLM" may mean "entering the input prompt into an inference engine based on the LLM." For example, "the output of the LLM for the input prompt" may refer to the output information of the last neural network layer of the LLM obtained when the input prompt is entered into the LLM-based inference engine.

[0123] FIG. 4a illustrates a situation in which an indicator (e.g., an application icon or widget) is displayed on a display of an electronic device (e.g., the electronic device (101) of FIG. 3) according to a comparative example.

[0124] An electronic device according to a comparative embodiment can display a background on a display. The background screen can include either a still image or a moving image.

[0125] An indicator may include at least one of the following: an icon corresponding to the application, a widget displaying the application's content, or text corresponding to the application. The types of indicators are merely examples and are not limited thereto.

[0126] An electronic device according to a comparative embodiment can display a background on a display and display at least one indicator on the background screen.

[0127] FIG. 4b illustrates a situation in which an electronic device according to a comparative example displays an indicator (e.g., an application icon or widget) against a background of an actual space.

[0128] In FIG. 4b, the electronic device (e.g., the wearable electronic device (200, 270) of FIG. 2) may include, for example, an XR (extended reality) device. For example, the XR (extended reality) device may be used in a form worn on a user's head. In one embodiment, the XR (extended reality) device may use a camera to capture an environment around the electronic device and display an indicator (e.g., an application icon or a widget) with the captured screen as the background. When the electronic device uses a video or captured screen as the background, the background screen may change in real time. In this case, at least one indicator may have a different color from the background on the first screen, making it relatively recognizable to the user. On the other hand, at least one indicator may have a similar color to the background on the second screen, making it relatively less recognizable to the user. This will be described in FIG. 4c.

[0129] FIG. 4c illustrates a situation in which an indicator is displayed on a display of an electronic device according to a comparative example, but the background is changed so that the indicator is not easily recognized by the user.

[0130] Electronic devices can change their backgrounds in real time when using a video or video capture screen as the background. In this case, at least one indicator may have a different color from the background on the first screen (410), making it relatively recognizable to the user. Conversely, at least one indicator may have a similar color to the background on the second screen (420), making it relatively less recognizable to the user.

[0131] In FIG. 4C, the electronic device can display multiple indicators on a background screen. The first indicator group (422) has a different color from the background on the first screen (410) with a forest as the background, making it relatively easily recognizable to the user. On the other hand, the first indicator group (422) has a similar color to the background on the second screen (420) with an ocean as the background, making it relatively difficult for the user to recognize it.

[0132] The second indicator group (424) may be relatively easily recognized by the user in the first screen (410) with a tree background because its color is different from the background. On the other hand, the first indicator group (422) may be relatively difficult to recognize by the user in the second screen (420) with a sand background because its color is similar to the background.

[0133] According to one embodiment, when the background changes over time, the degree to which the indicator is recognized by the user may change as the color or position of the background changes. The background may include, for example, an image or video generated using a machine learning model (e.g., a generative artificial intelligence model). The machine learning model may generate an image or video corresponding to a user input (e.g., a text input or a voice input) and provide it as the background. An electronic device according to the present document (e.g., the electronic device (101) of FIG. 1) may analyze the changing background and change the properties of the indicator based on the analysis information. For example, the electronic device (101) may analyze at least one of the color of the background displayed on the display, the color of at least one indicator, or the position at which the at least one indicator is displayed within the background. According to one embodiment, the electronic device (101) may generate a prompt instructing a change in the properties of the at least one indicator based on the analysis information so that the at least one indicator can be recognized as distinct from the background.

[0134] In an AI learning model, a prompt can represent input for a user's question or request. The AI ​​learning model can generate a corresponding response based on the prompt. The information provided in the prompt can play a crucial role in shaping the AI ​​learning model's response. For example, if a user asks, "What's the weather like today?", the AI ​​learning model can use the information in the prompt to generate a response such as, "It's sunny today."

[0135] FIG. 5a illustrates a situation in which the background screen of an electronic device is changed according to one embodiment.

[0136] In FIG. 5A, an electronic device (e.g., the electronic device (101) of FIG. 1) can display multiple screens (e.g., a first screen (510), a second screen (520), or a third screen (530)) as a background. This document assumes and describes a case where there are three background screens, but the background screens may change in real time, and the type or number of background screens may not be limited. For example, the background screen may be generated based on a pre-stored video file, or may be generated based on a screen captured in real time using a camera of the electronic device (101). For example, the background screen may include an image or video generated using a machine learning model (e.g., a generative artificial intelligence model). The machine learning model can generate an image or video corresponding to a user input (e.g., a text input or a voice input) and provide it as a background screen.

[0137] An electronic device (101) according to various embodiments of this document can change the characteristics of indicators displayed on a background screen so that the indicators can be easily recognized by a user in a situation where the background screen changes.

[0138] FIG. 5b illustrates an embodiment in which an indicator is displayed on a background screen in a situation in which the background screen of an electronic device is changed according to one embodiment.

[0139] For example, in the first screen (510), the colors of the 1-1 indicator group (512) and the 1-2 indicator (514) may be similar to the color of the background screen. The color may be expressed as at least one of an RGB value, HSL (hue, saturation, lightness), or HSV (hue, saturation, value). An electronic device (e.g., the electronic device (101) of FIG. 3) may use an RGB value, which is a method of expressing a color by combining the intensities of the three primary colors of light, red, green, and blue. In one embodiment, the electronic device (101) may express the colors of the indicators and the color of the background screen as values ​​from 0 to 255. For example, the electronic device (101) may determine that (255, 0, 0) indicates red, (0, 255, 0) indicates green, and (0, 0, 255) indicates blue. The electronic device (101) may express various colors by combining RGB values. For example, in a situation where the colors of the 1-1 indicator group (512) and the 1-2 indicator (514) are determined to be similar to the colors of the background screen, the electronic device (101) may change the colors of icons or texts included in the 1-1 indicator group (512) and the 1-2 indicator (514). The electronic device (101) may change the colors based on user settings (e.g., comfortable viewing) within the electronic device (101) when changing the colors of icons or texts. For example, the electronic device (101) can display to the user all candidates for colors that can be changed when changing the color of an icon or text, and perform the color change based on the user's selection.

[0140] For example, the electronic device (101) can control the edges of the first-first indicator group (512) and the first-second indicator (514) to be displayed with relatively thicker lines than before so as to be distinguished from the background screen.

[0141] In one embodiment, similar colors may mean that the difference in RGB values ​​is within a specified range (e.g., 20). The specified range (e.g., 20) is only an example and may vary depending on the settings.

[0142] HSL (hue, saturation, lightness) or HSV (hue, saturation, value) can refer to a method of expressing color. Hue is expressed as a value from 0 to 360 degrees, and can represent various colors by rotating along the color wheel. Saturation is the vividness of a color and can be expressed as a value from 0 to 100%. Lightness (or value) is the brightness of a color and can be expressed as a value from 0 to 100%. The electronic device (101) can display colors more intuitively than RGB by using HSL and HSV. The electronic device (101) can express similar or similar colors by using HSL and HSV.

[0143] In one embodiment, similar or similar colors may mean that the difference in hue values ​​is within a specified range (e.g., 10 degrees). The specified range (e.g., 10 degrees) is only an example and may vary depending on the configuration.

[0144] According to one embodiment, the electronic device (101) may compare the RGB values ​​of the 1-1 indicator group (512) with the RGB values ​​of the screen corresponding to the position of the 1-1 indicator group (512) on the first screen (510). The electronic device (101) may determine that the colors of the 1-1 indicator group (512) and the background screen are similar if the difference in the RGB values ​​is within a specified range (e.g., 20). For example, the electronic device (101) may determine that the colors of the 1-1 indicator group (512) and the background screen are not similar if the difference in the RGB values ​​exceeds a specified range (e.g., 20). The specified range is merely an example and may vary depending on the settings. Although described herein based on RGB values, the electronic device (101) may also compare HSL (hue, saturation, lightness) or HSV (hue, saturation, value) to determine whether the colors are similar.

[0145] For example, in the second screen (520), the colors of the 2-1 indicator group (522), the 2-2 indicator group (524), and the 2-3 indicator (526) may be similar to the colors of the background screen. For example, in the third screen (530), the colors of the 3-1 indicator group (532), the 3-2 indicator (534), and the 3-3 indicator (536) may be similar to the colors of the background screen.

[0146] According to one embodiment, the electronic device (101) can display special effects on a display (e.g., the display module (160) of FIG. 1). The special effects may include, for example, effects that display additional objects (e.g., fallen leaves, raindrops, or snowflakes) on a background screen. The special effects may be generated based on, for example, the weather. For example, if it is snowing, a special effect that makes it look like snow is falling on the background screen may be displayed. If it is raining, a special effect that makes it look like water droplets are falling on the background screen may be displayed. The special effects are merely examples and may vary depending on the type of additional object, and the type of additional object is not limited thereto.

[0147] In one embodiment, the color of an indicator may be changed based on the screen content modified by a special effect. For example, if an animation of falling leaves is displayed at a specific location, the indicator displayed at a location corresponding to the location of the leaves may be difficult for the user to recognize due to its color. In this case, the electronic device (101) may change the color of the indicator to distinguish it from the fallen leaves, taking into account the special effect for the falling leaves. While the special effect is described herein as falling leaves, the special effect is not limited to this.

[0148] In one embodiment, when a fallen leaf passes a first point, the electronic device (101) may change the color of an indicator located at the first point to distinguish it from the fallen leaf. The indicator located at the first point may overlap, in whole or in part, with an area where an animation corresponding to the fallen leaf is displayed. In this case, the electronic device (101) may change the color of an indicator that partially or completely overlaps with an area where the fallen leaf is displayed, based on the color of the fallen leaf.

[0149] For example, if the fallen leaves are red, the electronic device (101) can set the color of the indicator located at the first point to a different color that is distinct from red. If the fallen leaves pass the first point and move to a second point, the electronic device (101) can restore the color of the indicator located at the first point to its original color. In addition, the electronic device (101) can set the color of the indicator located at the second point to a different color that is distinct from red. If the fallen leaves pass the second point and move to a third point, the electronic device (101) can restore the color of the indicator located at the second point to its original color. In addition, the electronic device (101) can set the color of the indicator located at the third point to a different color that is distinct from red. The number of points and the colors of the special effects are merely examples and are not limited thereto.

[0150] In one embodiment, the electronic device (101) can control the color of the indicator to not continuously change, but rather not to exceed a specified number of times. For example, the specified number of times (e.g., three times) may vary depending on the settings. If the color of the indicator continuously changes, it may cause eye fatigue to the user and increase battery consumption of the electronic device (101). Therefore, the electronic device (101) can limit the number of times the color of the indicator changes.

[0151] FIG. 5c illustrates an embodiment in which, in a situation where the background screen of an electronic device is changed, at least one indicator has a characteristic that is changed so that it can be recognized by a user on the background screen.

[0152] In FIG. 5c, an electronic device (101) according to an embodiment may generate a prompt requesting a change in the color of the 1-1 indicator group (512) so that the color of the 1-1 indicator group (512) can be distinguished from the color of the background screen.

[0153] The fact that the color of the first indicator group (512) is distinct from the color of the background screen may mean that the difference between the RGB value of the first indicator group (512) and the RGB value of the background screen exceeds a specified level (e.g., 30). The specified level (e.g., 30) is only an example and may vary depending on the settings. Although described herein based on RGB values, the electronic device (101) may also compare HSL (hue, saturation, lightness) or HSV (hue, saturation, value) to determine whether the colors are similar. This has been described above in FIG. 5b.

[0154] In an AI learning model, a prompt can represent input for a user's question or request. The AI ​​learning model can generate a corresponding response based on the prompt. The information provided through the prompt can play a crucial role in shaping the AI ​​learning model's response.

[0155] In one embodiment, the electronic device (101) may generate a prompt and provide it to an artificial intelligence learning model. The artificial intelligence learning model may generate color-related information to change the color of an indicator with a color similar to the background image to a color different from the background image based on the prompt input from the electronic device (101). For example, the electronic device (101) may change the color of an indicator based on the color-related information output from the artificial intelligence.

[0156] Alternatively, for example, the artificial intelligence learning model may change the color of an indicator having a color similar to the background to a color not similar to the background based on a prompt input from the electronic device (101).

[0157] In one embodiment, the electronic device (101) may analyze at least one of the color of the background screen, the color of at least one indicator, or the position at which the at least one indicator is displayed within the background screen. For example, the electronic device (101) may generate a prompt instructing a change in the properties of at least one indicator so that the at least one indicator can be recognized as distinct from the background based on the analysis information.

[0158] According to one embodiment, the electronic device (101) may analyze the background screen and express the color of the background screen as a number. For example, the number may include at least one of an RGB value, an HSL (hue, saturation, lightness) value, or an HSV (hue, saturation, value) value. The electronic device (101) may analyze the color of at least one indicator and express it as a number. The electronic device (101) may analyze the position where the at least one indicator is displayed within the background screen. In one embodiment, the electronic device (101) may generate a prompt requesting to change the color of the at least one indicator so that the color of the at least one indicator differs from the numerical value corresponding to the color of the background screen by more than a specified level (e.g., 20%). The information provided in the prompt may play a significant role in forming the response of the artificial intelligence learning model.

[0159] According to one embodiment, the electronic device (101) can determine the color of the background screen, the color and position of at least one indicator, and generate a prompt based on these values. For example, the prompt can be generated in the following form: "Check the positions where indicators are displayed on the background screen, and change the color of each indicator so that it can be distinguished from the background screen."

[0160] This is just one example, and the electronic device (101) can generate various prompts based on the color of the background screen, the position and color of the indicator displayed on the background screen.

[0161] According to one embodiment, the electronic device (101) may, under the control of a processor (e.g., processor (120) of FIG. 1), analyze the color of the background screen, the position and color of an indicator displayed on the background screen, and generate a prompt based on the analysis information.

[0162] According to one embodiment, the electronic device (101) may insert a background image (e.g., an image or video) onto the prompt and provide it to the artificial intelligence learning model. The electronic device (101) may insert images of indicators to be displayed onto the prompt and provide them to the artificial intelligence learning model. Instead of analyzing the color of the background image and the position and color of the indicators displayed on the background image using the processor (120), the electronic device (101) may provide the images to the artificial intelligence learning model for analysis.

[0163] In one embodiment, the electronic device (101) may generate a prompt to request the AI ​​learning model to analyze the color of the background screen, the location and color of the indicator displayed on the background screen. The electronic device (101) may also generate a prompt to request the AI ​​learning model to provide change information on the indicator so that the color of the indicator can be distinguished from the background screen. The change information on the indicator may refer to information on the color of the indicator. For example, the electronic device (101) may use the change information on the indicator output from the AI ​​learning model to change the characteristics (e.g., color or location) of the indicator.

[0164] According to one embodiment, the electronic device (101) may generate a prompt requesting a change in the color of the first-second indicator (514) so ​​that the color of the first-second indicator (514) can be distinguished from the color of the background screen.

[0165] According to one embodiment, the electronic device (101) may generate a prompt requesting a change in the color of the second-second indicator group (524) so ​​that the color of the second-second indicator group (524) can be distinguished from the color of the background screen.

[0166] In one embodiment, the electronic device (101) may generate a prompt requesting a change in the color of the second-third indicator (526) so that the color of the second-third indicator (526) can be distinguished from the color of the background screen.

[0167] In one embodiment, the electronic device (101) may generate a prompt requesting a change in the color of the 3-2 indicator (524) so ​​that the color of the 3-2 indicator (524) can be distinguished from the color of the background screen. The electronic device (101) may generate a prompt requesting a change in the color of the 3-3 indicator (526) so that the color of the 3-3 indicator (526) can be distinguished from the color of the background screen.

[0168] FIG. 5c illustrates a situation in which an artificial intelligence learning model changes the characteristics of an indicator based on a prompt generated by an electronic device (101) to output a result and displays the output result on a display.

[0169] According to one embodiment, the electronic device (101) can change the internal color of at least one indicator. Additionally, the electronic device (101) can change the color and / or thickness of the border of at least one indicator. For example, the indicator may include at least one of an icon corresponding to an application, text describing the application, or a widget displaying the contents of the application.

[0170] FIG. 6 is a flowchart illustrating a method for editing indicators of an electronic device according to various embodiments.

[0171] The operations described through FIG. 6 can be implemented based on instructions that can be stored in a computer recording medium or memory (e.g., memory (130) of FIG. 1). The illustrated method can be executed by an electronic device (e.g., electronic device (101) of FIG. 1) described above through FIGS. 1 to 5C, and the technical features described above will be omitted below. The order of each operation of FIG. 6 can be changed, some operations can be omitted, and some operations can be performed simultaneously.

[0172] In operation 610, the electronic device (101) may analyze at least one of the color of the background screen, the color of the indicator, or the display position of the indicator under the control of a processor (e.g., the processor (120) of FIG. 1 ). For example, the background screen may include an image or video generated using a machine learning model (e.g., a generative artificial intelligence model). The machine learning model may generate an image or video corresponding to a user input (e.g., a text input or a voice input) and provide it as the background screen.

[0173] According to one embodiment, the electronic device (101) can analyze at least one of a background screen (e.g., a home screen composed of an image, a video, an actual environment of an XR device, an application screen, or a dynamic lock screen), an application icon, an AOD (always on display) screen, or a widget. For example, the electronic device (101) can analyze the background screen when the screen changes or is likely to change in real time, such as a dynamic lock screen, a video, or a real-time shooting screen, and perform an operation of analyzing an application icon, widget, and / or text. AOD (always on display) may refer to a function of displaying information (e.g., date, notification, remaining battery level, or time) even when the screen displayed on the display of the electronic device (101) is set to off. The electronic device (101) can reduce power consumption while displaying information to the user by using the AOD function.

[0174] According to one embodiment, the electronic device (101) may include an extended reality (XR) device. When operating in MR mode and / or AR mode, the electronic device (101) may display an indicator (e.g., an application icon, widget, or text) against the background of the surrounding environment acquired through the camera. The indicator may include a virtual object.

[0175] According to one embodiment, the electronic device (101) can set a dynamic screen or video that changes at a specified interval as the background screen. If there is movement in the background screen, the electronic device (101) can analyze the background screen and perform an operation to analyze the application's icons, widgets, and / or text.

[0176] According to one embodiment, the electronic device (101) may perform operations to analyze various backgrounds, app icons, widgets, or texts described above based on specified conditions.

[0177] In one embodiment, the XR device can acquire information in real time using a camera (e.g., the camera module (180) of FIG. 3) or a sensor. The electronic device (101) can perform analysis operations based on the operating mode of the electronic device (101) (e.g., VR, MR, or AR mode) or whether the electronic device (101) is moving.

[0178] According to one embodiment, the electronic device (101) may acquire information about the surrounding environment of the electronic device (101) using a camera based on detecting that a user is wearing the electronic device (101), and may control the electronic device (101) to acquire the information about the surrounding environment based on moving the electronic device (101) to a designated location or changing the electronic device (101) from a virtual reality (VR) mode to either a mixed reality (MR) mode or an augmented reality (AR) mode. Since the mixed reality (MR) mode or the augmented reality (AR) mode displays a virtual object or indicator on a real background captured by the camera, it is necessary to acquire information about the surrounding environment. On the other hand, when the electronic device (101) operates in a virtual reality (VR) mode, the electronic device (101) may not acquire information about the surrounding environment in order to prevent waste of power and / or storage space because it displays a virtual object or indicator on a virtual background screen.

[0179] According to one embodiment, the electronic device (101) may acquire information about the surrounding environment of the electronic device (101) using a camera based on the detection that the user is wearing the electronic device (101). When a new movement exceeding a specified range is detected in a situation where no movement is detected for a specified period of time or a new object is detected while the electronic device (101) is moving, the electronic device (101) may determine that the electronic device (101) will change to either a mixed reality (MR) mode or an augmented reality (AR) mode and control the acquisition of the surrounding environment information.

[0180] In one embodiment, the electronic device (101) can analyze a set background (e.g., an image, animation, or video). For example, if the background comprises a specified number of images that change, such as a dynamic background, the electronic device (101) can analyze all images that change. Unlike real-time captured images, videos or dynamic backgrounds stored in the memory (130) have a periodic nature, so the electronic device (101) can relatively easily predict how the background will change. The electronic device (101) can utilize this prediction to generate a prompt for changing an indicator.

[0181] In one embodiment, the electronic device (101) can recognize a repeating pattern when the background image changes in the same way repeatedly. In the case of a background image captured in real time without a repeating pattern, the electronic device (101) can analyze the location of the electronic device and the displayed color based on the changes over a certain period of time (e.g., 1 minute) to determine an average color. The electronic device (101) can use information about the determined average color to generate a prompt for changing the indicator.

[0182] In operation 620, the electronic device (101) may generate prompt text instructing a change in the properties of the indicator so that it can be recognized as distinct from the background.

[0183] According to one embodiment, the electronic device (101) may generate a prompt that instructs a change in the properties of an icon, widget, or text of an application based on analysis information, movement information of the device, vision-related information of the user, and preference information of the user.

[0184] Vision-related information may include, for example, terminal settings related to screen mode, color and clarity, size and zoom, color blindness, color weakness, visual impairment, visual field impairment, light-sensitivity impairment, color vision impairment, refractive impairment, accommodation impairment, binocular vision problems, age, or gender. These are examples only, and vision-related information may vary depending on settings.

[0185] According to one embodiment, the electronic device (101) may perform an operation of generating a prompt for improving visibility based on at least one of the analysis result obtained in operation 610, movement-related information of the electronic device (101), a state of the electronic device (101) (e.g., VR, MR AR mode), and vision-related information of the user.

[0186] Improving visibility may refer to the action of changing the properties of an indicator so that the colors of the background and the indicator are different when the colors are similar. Here, similar colors may mean that the difference in RGB values ​​is within a specified range (e.g., 20). The specified range (e.g., 20) is only an example and may vary depending on the settings. Alternatively, similar or similar colors may mean that the difference in hue values ​​is within a specified range (e.g., 10 degrees). The specified range (e.g., 10 degrees) is only an example and may vary depending on the settings. RGB values, HSL (hue, saturation, lightness), or HSV (hue, saturation, value) are described in FIGS. 5A to 5C.

[0187] The user's eyesight-related information may include, for example, electronic device (101) setting values ​​related to the color of the background screen, the sharpness of the background screen, the size of the background screen, and the zoom in or out of the indicator of the screen mode (e.g., VR, AR, or MR mode) of the electronic device (101) input by the user. For example, the user's eyesight-related information may include personal information including at least one of color blindness, color weakness, visual acuity, field of vision, wide angle, color vision, refraction, accommodation, binocular vision, age, or gender.

[0188] In one embodiment, the electronic device (101) may determine that the user is walking through a dense forest based on the screen displayed for a specified period of time (e.g., 3 minutes) using an artificial intelligence learning model. The electronic device (101) may change at least one of the color, border, or display location of the indicator, while avoiding colors similar to green and removing red tones for users with red-green color blindness. This is merely an example, and the user characteristics or location are not limited thereto.

[0189] According to one embodiment, the electronic device (101) can determine the color of the background screen, the color and position of at least one indicator, and generate a prompt based on these values. For example, the prompt can be generated in the following form: "Check the positions where indicators are displayed on the background screen, and change the color of each indicator so that it can be distinguished from the background screen."

[0190] A prompt could take the following form, for example: "Analyze the change in the background, isolate the current background from the objects on it, and then tell me which images or settings need to be updated for each object so that I can change objects that may be problematic for the user with red-green color blindness. The user's vision and color blindness information is the same as what was recently requested, so please refer to it if necessary." This is just an example and the form of the prompt is not limited to this. The form of the prompt can be any form that requests a change in the properties of the indicator so that the background and the indicator can be recognized differently.

[0191] According to one embodiment, the electronic device (101) may generate a prompt requesting a color change of the indicator based on at least one of: a change in ambient brightness, a change in brightness according to a battery level of the electronic device, a rotation of the electronic device (101), a movement of the electronic device (101), or a color change information of the indicator.

[0192] The electronic device (101) can change the properties of at least one indicator based on the generated prompt text. The action of changing the properties of the indicator may not be essential in the indicator editing action of the electronic device (101).

[0193] According to one embodiment, the electronic device (101) can change the properties of at least one indicator using information output from an artificial intelligence learning model. The information output from the artificial intelligence learning model may include, for example, information regarding the color of the indicator. For example, the electronic device (101) may generate a prompt to request the artificial intelligence learning model to change the indicator's color so that it can be distinguished from the background screen. For example, the artificial intelligence learning model may output multiple change information items according to settings and provide them to the electronic device (101). The color that makes the indicator's color distinguishable from the background screen may be multiple, not just one. The electronic device (101) may generate multiple screens as candidates based on the multiple change information items and ultimately display one of the candidates on the display based on a user's selection.

[0194] According to one embodiment, the electronic device (101) can capture the surroundings using a camera. In this case, an indicator displayed on a background screen may not be easily recognized by the user compared to the background screen due to changes in the surrounding lighting or the color of objects. The electronic device (101) according to this document can improve usability by changing the border or color of an indicator that is not easily recognized by the user depending on the state of the background, so that the indicator can be relatively easily recognized by the user.

[0195] FIG. 7 is a flowchart illustrating a method for editing indicators of an electronic device according to various embodiments.

[0196] The operations described through FIG. 7 can be implemented based on instructions that can be stored in a computer recording medium or memory (e.g., memory (130) of FIG. 1). The illustrated method (700) can be executed by an electronic device (e.g., electronic device (101) of FIG. 1) described above through FIGS. 1 to 5C, and the technical features described above will be omitted below. The order of each operation of FIG. 7 can be changed, some operations can be omitted, and some operations can be performed simultaneously.

[0197] In operation 702, the electronic device (101) may receive information about the displayed background and indicator under the control of a processor (e.g., processor (120) of FIG. 1).

[0198] In operation 704, the electronic device (101) may acquire real-world environmental information using a camera. For example, the electronic device (101) may include an extended reality (XR) device. When operating in MR mode and / or AR mode, the electronic device (101) may display an indicator (e.g., an application icon, widget, or text) against the background of the surrounding environment acquired through the camera. The indicator may include a virtual object.

[0199] In operation 706, the electronic device (101) can analyze the background and indicator.

[0200] According to one embodiment, the electronic device (101) can analyze at least one of a color of a background screen displayed on the display, a color of at least one indicator, or a position at which at least one indicator is displayed within the background screen.

[0201] In operation 708, the electronic device (101) may generate a prompt to distinguish between the background and the indicator. According to one embodiment, the electronic device (101) may generate a prompt to instruct a change in the properties of the at least one indicator so that the at least one indicator can be recognized as distinct from the background based on the analysis information.

[0202] According to one embodiment, the electronic device (101) may generate a prompt indicating a property change using at least one of analysis information, location information of the electronic device (101), or vision information of a user of the electronic device (101). The vision information of the user of the electronic device may include at least one of screen mode input by the user, screen color, screen clarity, screen size, whether the screen is enlarged or reduced compared to the base screen, color blindness, color weakness, the user's vision, the user's field of vision, wide angle, color vision, refraction, binocular vision, or age.

[0203] In operation 710, the electronic device (101) can determine whether the background screen displayed on the display is an image or a video. Furthermore, the electronic device (101) can determine whether the background screen displayed on the display includes virtual information or an image or video corresponding to a real environment. The virtual information may include, for example, an image or video generated by a machine learning model (e.g., a generative AI model).

[0204] Action 710 may be performed to determine whether the background of the electronic device (101) is moving or changing or is static.

[0205] In operation 712, the electronic device (101) may update an existing prompt to generate a prompt by considering at least some of the changing screen components (e.g., the color of the background) while the background changes based on whether the background is a video or a real environment. If the background changes, even if a prompt is generated for the first screen, it may be difficult to apply it to the changed second screen. The prompt may mean a command requesting a change in the color of an indicator displayed on the background screen to be different from the background screen. A prompt generated for the first screen may not be valid for the second screen, which is different from the first screen. Valid may mean requesting a change in the color of the indicator displayed on the background screen to be different from the background screen. Invalid may mean that it is difficult to request a change in the color of the indicator displayed on the background screen to be different from the color of the background screen. For example, a difference in the color of the indicator from the background screen may mean that there is a certain level or more of difference based on RGB values, so that the indicator and the background screen are recognized as distinct. The electronic device (101) can generate a single prompt by considering both the first and second screens when the first and second screens are different. Thereafter, in operation 714, the electronic device (101) can change the properties of the indicator using an artificial intelligence learning model.

[0206] In operation 714, the electronic device (101) may change the properties of the indicator using an artificial intelligence learning model. For example, the electronic device (101) may input the generated prompt into the artificial intelligence learning model and output information about changes in the properties of the indicator. The electronic device (101) may change the properties of the indicator based on the information about changes in the properties of the indicator output from the artificial intelligence learning model. At this time, the electronic device (101) may change the properties of the indicator so that the colors of the background screen and the indicator are different. The different colors may mean that the RGB values ​​of the background screen and the RGB values ​​of the indicator differ by more than a specified level (e.g., 20). The specified level (e.g., 20) is merely an example and may vary depending on the settings.

[0207] According to one embodiment, the electronic device (101) may determine the properties of at least one indicator so as to distinguish the background screen from the indicator using the generated prompt. The electronic device (101) may change the properties of at least one indicator based on at least one of a change in the ambient brightness of the electronic device (101), a battery status of the electronic device (101), a location of the electronic device (101), or an update status of the indicator.

[0208] According to one embodiment, the electronic device (101) may control, using the generated prompt, at least one of the colors of the background screen and the at least one indicator, the color of a portion adjacent to the at least one indicator, the thickness of the border of the at least one indicator, or the color of text displayed together with the at least one indicator to be changed. The electronic device (101) may change the properties of the at least one indicator based on the generated prompt.

[0209] According to one embodiment, the electronic device (101) may acquire information about the surrounding environment of the electronic device (101) using a camera based on detecting that the user is wearing the electronic device (101). The electronic device (101) may classify zones based on the direction in which the camera is facing or the direction in which the user of the electronic device (101) is facing. The electronic device (101) may classify, for example, four zones: top, bottom, left, and right. The classified zones are merely an example and may vary depending on the settings. The electronic device (101) may generate a background based on the surrounding environment information acquired using the camera for each classified zone, and determine the properties of at least one indicator so as to be distinguished from the generated background. The electronic device (101) may display the background generated based on the direction in which the camera is facing or the direction in which the user of the electronic device (101) is facing, and may change the properties of at least one indicator and display it on the background.

[0210] According to one embodiment, the electronic device (101) can determine the characteristics of the location where the electronic device (101) is located by using the surrounding environment information acquired through the camera and the location information acquired using the global positioning system (GPS).

[0211] According to one embodiment, the electronic device (101) can determine the location where the electronic device (101) is located using GPS, and can determine whether the user of the electronic device (101) is moving or stationary based on the GPS signal. In addition, the electronic device (101) can infer whether the electronic device (101) is walking or moving using another means of transportation based on the GPS signal when the electronic device (101) is moving. In a situation where a screen captured in real time using a camera is displayed as a background, the electronic device (101) can predict a background screen to be displayed in the future based on GPS. For example, if the electronic device (101) is moving on a highway, the electronic device (101) can determine that the user is moving using a vehicle. In this case, the electronic device (101) can predict a background screen to be displayed in the future based on the user's movement path. In a situation where the electronic device (101) displays an indicator with trees in the background, the electronic device (101) can predict that the electronic device (101) will pass through a tunnel. In this case, the electronic device (101) may determine that the background screen will be relatively darker than the current background screen and prepare to display the indicator relatively brightly.

[0212] According to one embodiment, the electronic device (101) can estimate the level of movement of the user of the electronic device (101) based on objects identified in the current background screen. For example, the electronic device (101) can determine that the user is in a study or a library or a bookstore based on the detection of multiple books in the background screen. The electronic device (101) can also more specifically identify the user's location using GPS. If multiple books are detected in the background screen, the electronic device (101) can estimate that the user is reading a book or moving to find a book, and that the user's movement is relatively small, so that the background screen will not change significantly. In this case, the electronic device (101) can maintain the properties (e.g., color or position) of the indicator displayed appropriately in the current background screen. Displaying appropriately in the background screen can mean that the indicator is displayed so as to be distinguishable from the background screen. The study is just one example; the electronic device (101) may determine, based on its settings, that the user's movements will be relatively large in some locations and relatively small in other locations.

[0213] In one embodiment, the electronic device (101) can determine the location of the user of the electronic device (101) based on the type of IoT device with which a communication connection has been established. For example, the electronic device (101) can determine that the user is located at the user's home based on a communication connection established with an appliance (e.g., a refrigerator or a robot vacuum cleaner). The appliance is merely an example, and the IoT device is not limited thereto.

[0214] According to one embodiment, the electronic device (101) may generate a prompt that instructs to change the properties of at least one indicator so that the at least one indicator is distinguished from the background, taking into account the characteristics of the location where the electronic device (101) is located (e.g., the sea or a forest) and the vision information of the user of the electronic device (101) (e.g., color blindness).

[0215] According to one embodiment, the electronic device (101) may analyze at least one of a color of a background screen displayed on a display, a color of at least one indicator, or a position at which at least one indicator is displayed within the background screen, and generate a prompt for instructing a property change using at least one of the analysis information, location information of the electronic device (101), or eyesight information of a user of the electronic device (101). The eyesight information of the user may include at least one of a screen mode input by the user, a color of the screen, a sharpness of the screen, a size of the screen, whether the screen is enlarged or reduced compared to a basic screen, color blindness, color weakness, the user's eyesight, the user's field of vision, a wide angle, color vision, refraction, binocular vision, or age.

[0216] According to one embodiment, the electronic device (101) may control, using the generated prompt, at least one of the colors of the background screen and the at least one indicator, the color of a portion adjacent to the at least one indicator, the thickness of the border of the at least one indicator, or the color of text displayed together with the at least one indicator to be changed. The electronic device (101) may change the properties of the at least one indicator based on the generated prompt.

[0217] According to one embodiment, the electronic device (101) may determine the properties of at least one indicator so as to distinguish the background screen from the indicator using the generated prompt. The electronic device (101) may change the properties of at least one indicator based on at least one of a change in the ambient brightness of the electronic device (101), a battery status of the electronic device (101), a location of the electronic device (101), or an update status of the indicator.

[0218] According to one embodiment, the electronic device (101) may acquire information about the surrounding environment of the electronic device (101) using a camera. The electronic device (101) may determine whether a background screen on which at least one indicator is displayed will change based on a location where a user of the electronic device (101) is located. Based on a determination that the background screen on which at least one indicator is displayed will not change, the electronic device (101) may generate a prompt to change the properties of at least one indicator so that at least one indicator can be recognized in the currently displayed background screen.

[0219] According to one embodiment, the electronic device (101) may obtain information about the colors of the background screen at a first time in the past and the background screen at a second time in the past based on the present based on the expectation that the background screen on which at least one indicator is displayed will change. The electronic device (101) may generate a prompt so that the color of the at least one indicator is different from the color of the background screen at the first time and different from the color of the background screen at the second time.

[0220] According to one embodiment, the electronic device (101) can acquire information about the surrounding environment of the electronic device (101) using a camera based on the detection of a user's body coming into contact with the electronic device (101). The electronic device (101) can control not to acquire information about the surrounding environment based on the electronic device (101) moving to a designated location or the electronic device (101) changing from a virtual reality (VR) mode to either a mixed reality (MR) mode or an augmented reality (AR) mode.

[0221] According to one embodiment, when a new movement beyond a specified range is detected in a situation where no movement of the electronic device (101) is detected for a specified period of time, or a new object is detected while the electronic device (101) is moving, the electronic device (101) may determine to change to either a mixed reality (MR) mode or an augmented reality (AR) mode and control the electronic device (101) not to acquire surrounding environment information.

[0222] According to one embodiment, the electronic device (101) may acquire information about the surrounding environment of the electronic device (101) using a camera based on detecting that the user is wearing the electronic device (101). The electronic device (101) may classify zones based on the direction in which the camera is facing or the direction in which the user of the electronic device (101) is facing. The electronic device (101) may classify, for example, four zones: top, bottom, left, and right. The classified zones are merely an example and may vary depending on the settings. The electronic device (101) may generate a background based on the surrounding environment information acquired using the camera for each classified zone, and determine the properties of at least one indicator so as to be distinguished from the generated background. The electronic device (101) may display the background generated based on the direction in which the camera is facing or the direction in which the user of the electronic device (101) is facing, and may change the properties of at least one indicator and display it on the background.

[0223] According to one embodiment, the electronic device (101) may generate a prompt that instructs to change the properties of at least one indicator so that the at least one indicator is distinguished from the background, taking into account both the characteristics of the location where the electronic device (101) is located and the vision information of the user of the electronic device (101).

[0224] Figure 8 is a block diagram of a natural language understanding module (800).

[0225] Referring to FIG. 8, the natural language understanding module (800) may include a natural language understanding model (810), a dispatcher (820), a domain classifier (830), or a combination thereof. For example, the natural language understanding module (800) may be a software module implemented by executing instructions by a processor. Hereinafter, the operations performed by the natural language understanding module (800) may be referred to as operations of the processor of the device implementing the natural language understanding module (800).

[0226] The natural language understanding module (800) can obtain a speech recognition result (e.g., data converted from one or more phonemes included in a user's speech) from the automatic speech recognition module (802). The natural language understanding module (800) can provide a processing result of the speech recognition result to the planner module (804). The processing result of the speech recognition result can include an intent, a target device (e.g., information about the target device), a capsule, or a combination thereof.

[0227] The natural language understanding model (810) can determine intent by interpreting (e.g., syntactic analysis and / or semantic analysis) the speech recognition result from the automatic speech recognition module (802). The natural language understanding model (810) can use linguistic features (e.g., grammatical elements) of morphemes or phrases to understand the meaning of words extracted from the speech recognition result, and can determine the user's intent based on the meaning of the understood word and / or other parameters (e.g., domains or categories associated with the word). Grammatical analysis can include an act of dividing user input (e.g., user's utterance) into grammatical units (e.g., words, phrases, and / or morphemes) and identifying grammatical elements of the divided units. Semantic analysis can be performed through semantic matching, rule matching, and / or formula matching.

[0228] Here, data converted from one or more phonemes may represent one or more words included in the user's utterance, and / or tokens of each of one or more words. The intent may be data used by the natural language platform to generate a plan. The intent may include a goal and / or parameters. The goal may be used to specify the final goal of the plan in the planner module (804). The parameters may be values ​​input to one or more actions included in the plan in the planner module (804).

[0229] The dispatcher (820) can determine a target device and / or capsule (or domain, application) associated with one or more words included in the speech recognition result.

[0230] The dispatcher (820) may include a device dispatcher (821), a named dispatcher (823), a meta command dispatcher (825), or a combination thereof.

[0231] The device dispatcher (821) can determine one or more target devices based on the device names included in the voice recognition results. For example, the target devices may be IoT (Internet of Things) devices. The target devices may include, but are not limited to, devices that perform the operations included in the plan and / or devices that receive the results of performing the operations included in the plan.

[0232] The named dispatcher (823) can determine one or more capsules based on the names of the capsules included in the speech recognition results.

[0233] The meta-command dispatcher (825) may determine one or more capsules based on specific commands included in the speech recognition result. For example, the specific commands may be commands designated for specific situations. The specific situations may include situations in which the speech recognition service prompts the user, and / or situations in which content selection is possible from a content list including one or more contents. For example, in a prompting situation, the specific commands may include select (e.g., "first"), cancel (e.g., "cancel"), confirm (e.g., "understood"), or a combination thereof. For example, in situations in which content selection is possible, the specific commands may include repeat (e.g., "again"), next (e.g., "next"), previous (e.g., "back to previous"), or a combination thereof.

[0234] The domain classifier (830) can determine one or more capsules required to perform a task based on the speech recognition results. The domain classifier (830) can determine one or more capsules using defined classification rules and / or artificial intelligence models. The defined classification rules and / or artificial intelligence models can be learned using a learning algorithm.

[0235] FIG. 9 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0236] Referring to FIG. 9, an integrated intelligent system of one embodiment may include a first electronic device (901) (e.g., the electronic device (100) of FIG. 1), a second electronic device (902) (e.g., any device including a headset, earbuds, or microphone), an intelligent server (1000), and a service server (1099).

[0237] According to the illustrated embodiment, the first electronic device (901) may include a communication interface (910), an input / output (I / O) interface (920), a processor (930), and / or a memory (940). The components listed above may be operatively or electrically connected to each other. For example, the electronic device (901) may include at least some of the components of the electronic device (100) of FIG. 1.

[0238] The communication interface (910) can be connected to an external device (e.g., an intelligent server (1000) and / or a service server (1099)) via a first network (999) (e.g., any network including a cellular network and / or a wireless local area network (WLAN)) to transmit and receive data. For example, the communication interface (910) can correspond to the CP (118) and / or the communication circuit (160) of FIG. 1. The communication interface (910) can support data transmission and reception with an external device (e.g., a second electronic device (902)) via a second network (998) (e.g., a short-range wireless communication network).

[0239] The I / O interface (920) may receive user input, process received user input, and / or output results processed by the processor (930) using input / output devices (not shown) (e.g., a microphone, a speaker, and / or a display (e.g., the display (140) of FIG. 1)).

[0240] The processor (930) may be operatively or electrically connected to a communication interface (910), an I / O interface (920), and / or a memory (940) (e.g., memory (120) of FIG. 1) to perform a designated operation. For example, the processor (930) may correspond to the processor (110) of FIG. 1. The processor (930) may execute a program (or one or more instructions) stored in the memory (940) to perform a designated operation. For example, the processor (930) may receive a user's voice input (e.g., user speech) through the I / O interface (920). For example, the processor (930) may receive a user's voice input received by the second electronic device (902) from the second electronic device (902) through the communication interface (910). The processor (930) can transmit voice input received through the communication interface (910) to the intelligent server (1000). For example, the processor (930) can include one or more processors.

[0241] The processor (930) may receive a result corresponding to a voice input from the intelligent server (1000). For example, the processor (930) may receive a plan corresponding to the voice input and / or a result calculated using the plan from the intelligent server (1000). For example, the plan may include, but is not limited to, information regarding a plurality of sequential operations to be executed by the first electronic device (901) and / or another electronic device in relation to the voice input. The processor (930) may receive a request from the intelligent server (1000) to obtain information (e.g., entities, slots, and / or parameters) necessary to generate a plan corresponding to the voice input. The processor (930) may transmit the necessary information to the intelligent server (1000) in response to the request.

[0242] The processor (930) can visually, tactilely, and / or audibly output the results of executing the operations specified according to the plan through the I / O interface (920). For example, the processor (930) can sequentially display the execution results of multiple operations on the display. As an example, the processor (930) can display only the execution results of executing multiple operations (e.g., the execution result of one of the multiple operations or the execution result of the last operation) on the display. The processor (930) can provide feedback through the second electronic device (902) by transmitting the execution results of the multiple operations or the execution results of at least some of the multiple operations to the second electronic device (902).

[0243] The processor (930) can recognize voice input. For example, the processor (930) can execute an intelligent app (or a voice recognition app) to process the voice input in response to a specified voice input (e.g., "Wake up!"). The processor (930) can provide a voice recognition service through the intelligent app. The processor (930) can transmit the voice input to the intelligent server (1000) through the intelligent app and receive a result corresponding to the voice input from the intelligent server (1000).

[0244] In one example, the second electronic device (902) may include a communication interface (911), an input / output (I / O) interface (921), a processor (931), and / or a memory (941). The components listed above may be operatively or electrically connected to each other. In one example, the second electronic device (902) may be a set of multiple electronic devices configured as a single set (e.g., a left earbud and a right earbud).

[0245] The communication interface (911) may support connection with an external device (e.g., the first electronic device (901)) via a second network (998). The I / O interface (921) may receive user input, process received user input, and / or output a result processed by the processor (931) using input / output devices (not shown) (e.g., at least one microphone, at least one speaker, and / or button).

[0246] The processor (931) may be operatively and / or electrically connected to the communication interface (911), the I / O interface (921), and / or the memory (941) to perform a designated operation. The processor (931) may execute a program (or one or more instructions) stored in the memory (941) to perform the designated operation. For example, the processor (931) may receive a user's voice input (e.g., a user's speech) through the I / O interface (921). In one example, the processor (931) may perform voice activity detection (VAD) using at least one sensor (not shown) of the second electronic device (902). The processor (931) may detect a user's speech of the second electronic device (902) using an acceleration sensor and / or a microphone.

[0247] The processor (931) can transmit voice input received through the second network (998) to the first electronic device (901) using the communication interface (911).

[0248] The processor (931) may receive a result corresponding to a voice input from the first electronic device (901). For example, the processor (931) may receive data (e.g., text data) corresponding to the result corresponding to the voice input from the first electronic device (901). The processor (931) may output the received result through the I / O interface (921).

[0249] The processor (931) can recognize a voice input. For example, the processor (931) can request the first electronic device (901) to execute an intelligent app (or a voice recognition app) to process the voice input in response to a specified voice input (e.g., wake up!).

[0250] An intelligent server (1000) of one embodiment can receive a user's voice input from a first electronic device (901) via a first network (999). The intelligent server (1000) can convert audio data corresponding to the received voice input into text data. The intelligent server (1000) can generate at least one plan for performing a task corresponding to the user's voice input based on the text data. The intelligent server (1000) can transmit the generated plan or a result according to the generated plan to the first electronic device (901) via the first network (999).

[0251] An intelligent server (1000) of one embodiment may execute one or more programs including a front end (1010), a natural language platform (1020), a capsule database (1030), an execution engine (1040), and / or an end user interface (1050).

[0252] The front end (1010) can receive a voice input from the first electronic device (901) or the second electronic device (902). The front end (1010) can transmit a response corresponding to the voice input to the first electronic device (901).

[0253] The natural language platform (1020) may include an automatic speech recognition (ASR) module (1021), a natural language understanding (NLU) module (1023), a planner module (1025), a natural language generator (NLG) module (1027), and / or a text-to-speech (TTS) module (1029).

[0254] The automatic speech recognition module (1021) can convert the voice input received from the first electronic device (901) into text data. The natural language understanding module (1023) can identify the user's intent and / or parameters (e.g., entities and / or slots) based on the text data of the voice input. The user's intent corresponds to the voice input and may include information indicating an action (or function) that the user wishes to perform using the device. For example, the slot may be detailed information related to the user's intent. The slot may be acquired based on a domain corresponding to the utterance. The slot may be variable information required to perform the action. In one embodiment, the variable information constituting the slot may include a named entity.

[0255] The planner module (1025) can generate a plan using the intent and / or parameters determined by the natural language understanding module (1023). For example, the planner module (1025) can determine at least one domain necessary to perform a task based on the determined intent. The domain may correspond to a category (or service) associated with an action (or function) that the user wishes to perform using the device. The domain may be classified according to a service (e.g., an app) related to the text. The domain may be related to the user's intent corresponding to the text. The domain may be classified according to, for example, the type of application that received the voice input and / or the type of service to be provided based on the voice input, but is not limited thereto. In one example, the determination of the domain may be performed by another module (e.g., the natural language understanding module (1023)). The planner module (1025) may determine a plurality of actions included in each of the at least one domain determined based on the intent. The planner module (1025) can determine parameters required to execute a plurality of determined actions or result values ​​output by the execution of the plurality of actions. The parameters and result values ​​can be defined as concepts of a specified format (or class). For example, the plan can include a plurality of actions and / or a plurality of concepts determined by the user's intention. The planner module (1025) can determine the relationship between the plurality of actions and / or the plurality of concepts in a step-by-step (or hierarchical) manner. For example, the planner module (1025) can identify the execution order of the plurality of actions (e.g., the plurality of actions determined based on the user's intention) based on the plurality of concepts (e.g., parameters required to execute the plurality of actions and results output by the execution of the plurality of actions). The planner module (1025) can generate a plan including association information (e.g., ontology) between the plurality of actions and the plurality of concepts.The planner module (1025) can create a plan using information (e.g., at least one capsule) stored in a capsule database (1030) in which a set of relationships between concepts and actions is stored.

[0256] The planner module (1025) can generate a plan based on an artificial intelligence (AI) system. For example, the AI ​​system can include one or more electronic devices and / or one or more processing circuits to execute a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN) and / or a recurrent neural network (RNN)), or a combination thereof. The AI ​​system described above is exemplary, and the AI ​​system can be an AI system based on any machine learning-based model. The planner module (1025) can select a plan corresponding to a user request from a set of predefined plans, or generate a plan in real time in response to a user request.

[0257] The natural language generation module (1027) can convert specified information into text. The information converted into text may be in the form of natural language speech. The text-to-speech conversion module (1029) can convert text-to-speech information into speech information.

[0258] The capsule database (1030) can store information on the relationship between multiple concepts and actions corresponding to multiple domains (e.g., applications). The capsule database (1030) can store at least one capsule (e.g., capsule (1031) and / or capsule (1033)) in the form of a concept action network (CAN). For example, the capsule database (1030) can store an action for processing a task corresponding to a user's voice input and / or parameters required for the action in the form of a CAN. A capsule can include multiple action objects (or action information) and / or concept objects (or concept information) included in a plan. For example, capsules (1031, 1033) can be created for each domain and stored in the capsule database (1030), but are not limited thereto.

[0259] The execution engine (1040) can produce results using the generated plan. The end user interface (1050) can transmit the produced results to the first electronic device (901).

[0260] According to one embodiment, some functions (e.g., the natural language platform (1020)) or all functions of the intelligent server (1000) may be implemented in the first electronic device (901). For example, the first electronic device (901) may execute one or more programs including the natural language platform (e.g., the natural language platform (950) of FIG. 10) separately from the intelligent server (1000). For example, the electronic device (901) may directly perform at least some of the operations of the natural language platform (1020) of the intelligent server (1000) (e.g., the automatic speech recognition module (1021), the natural language understanding module (1023), the planner module (1025), the natural language generation module (1027), and / or the text-to-speech module (1029)).

[0261] In one embodiment, a service server (1099) may provide a service (e.g., food ordering or hotel reservation) designated to a first electronic device (901). The service server (1099) may be a server operated by a different operator than the intelligent server (1000). The service server (1099) may communicate with the intelligent server (1000) and / or the first electronic device (901) via a first network (999). The service server (1099) may communicate with the intelligent server (1000) via a separate connection (not shown). The service server (1099) may provide the intelligent server (1000) with information for generating a plan corresponding to a voice input received by the first electronic device (901) (e.g., operation information and / or concept information for providing a designated service). The provided information may be stored in a capsule database (1030). The service server (1099) can provide the result information according to the plan received from the first electronic device (901) to the intelligent server (1000).

[0262] FIG. 10 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0263] Referring to FIG. 10, the integrated intelligence system may include a first electronic device (901), a second electronic device (902), and an intelligent server (1002). The first electronic device (901) and the intelligent server (1002) may be connected to each other via a network to transmit and receive data. For example, the first electronic device (901) and the second electronic device (902) may be connected to each other via a short-range network to transmit and receive data. According to one embodiment, the integrated intelligence system may be composed of a single device or multiple devices. For example, each device may include configurations having the same or similar functions, and the configuration of one device may be replaced with the configuration of another device.

[0264] According to one embodiment, the intelligent server (1002) may include the entire configuration or at least a portion of the configuration of the intelligent server (1000) illustrated in FIG. 9. For example, the intelligent server (1002) may execute one or more programs including the natural language platform (1020) of the intelligent server (1000) of FIG. 9 and / or store the capsule database (1030) of FIG. 9. The configuration of the intelligent server (1002) is not limited to that illustrated in FIG. 10. For example, at least a portion of the natural language platform (1020) (e.g., the automatic speech recognition module (1021), the natural language understanding module (1023), the planner module (1025), the natural language generation module (1027), and / or the text-to-speech module (1029)) may be omitted from the intelligent server (1002). For example, the intelligent server (1002) may further include some components of the intelligent server (1000) of FIG. 9 (e.g., the front end (1010), the execution engine (1040), and / or the end user interface (1050)).

[0265] The first electronic device (901) may execute one or more programs including a natural language platform (950) and / or store a capsule database (960). For example, the first electronic device (901) may further execute one or more programs including a natural language platform (950) and / or store a capsule database (960) while including components of the first electronic device (901) of FIG. 9.

[0266] The natural language platform (950) may include an automatic speech recognition module (951), a natural language understanding module (953), a planner module (955), a natural language generation module (957), and / or a text-to-speech module (959). The automatic speech recognition module (951), the natural language understanding module (953), the planner module (955), the natural language generation module (957), and the text-to-speech module (959) may perform functions identical to or similar to those of the automatic speech recognition module (1021), the natural language understanding module (1023), the planner module (1025), the natural language generation module (1027), and the text-to-speech module (1029) of FIG. 9, respectively.

[0267] The capsule database (960) may perform functions identical to or similar to those of the capsule database (1030) of the intelligent server (1000, 1002). The capsule database (960) may store information about relationships between multiple operations and multiple concepts included in a plan generated by the planner module (955). For example, the capsule database (960) may store at least one capsule (e.g., capsule (961) and / or capsule (963)).

[0268] According to one embodiment, the first electronic device (901) (e.g., the natural language platform (950) and / or the capsule database (960)) and the intelligent server (1002) (e.g., the natural language platform (1020) and / or the capsule database (1030)) may perform at least one function (or operation) in conjunction with each other, or may independently perform at least one function (or operation). For example, the first electronic device (901) may perform voice recognition on its own without transmitting the received user's voice input to the intelligent server (1002). As an example, the first electronic device (901) may convert the received voice input into text data through the automatic voice recognition module (951). The first electronic device (901) may transmit the converted text data to the intelligent server (1002). The intelligent server (1002) can determine (or identify) the user's intent and / or parameters from text data through the natural language understanding module (1023). The intelligent server (1002) can generate a plan through the planner module (1025) based on the determined intent and parameters and transmit the plan to the first electronic device (901), or can transmit the determined intent and parameters to the first electronic device (901) so that the plan is generated through the planner module (955) of the first electronic device (901). The planner module (955) of the first electronic device (901) can generate at least one plan for performing a task corresponding to a voice input using information stored in the capsule database (960).

[0269] For example, the first electronic device (901) can convert voice input received through the automatic speech recognition module (951) into text data, and determine (or identify) the user's intention and / or parameters based on the text data through the natural language understanding module (953). The first electronic device (901) can generate a plan through the planner module (955) based on the determined intention and parameters, or transmit the determined intention and parameters to the intelligent server (1002) so that the planner module (1025) of the intelligent server (1002) can generate a plan. For example, if the planner module (955) and / or the capsule database (960) are not included in the first electronic device (901), the first electronic device (901) can generate a plan through the intelligent server (1002).

[0270] For example, the first electronic device (901) can detect a speech pattern that is difficult to learn in an automatic speech recognition module (951) or a natural language understanding module (953), and transmit a voice input corresponding to the detected speech pattern to an intelligent server (1002) so that the automatic speech recognition module (1021) or the natural language understanding module (1023) of the intelligent server (1002) can process it.

[0271] Embodiments of the present disclosure are not limited to the examples described above. For example, the first electronic device (901) may process the received voice input only within the terminal and produce a result corresponding to the voice input. For example, the first electronic device (901) and the intelligent server (1002) may not only divide the voice input into modules and process it, but may also collaborate with each other to process it. For example, the natural language understanding module (953) of the first electronic device (901) and the natural language understanding module (1023) of the intelligent server (1002) may work together to produce a single result value (e.g., the user's intention and / or parameters).

[0272] The second electronic device (902) can execute one or more programs including an automatic speech recognition (ASR) module (952) and / or a text-to-speech (TTS) module (954). For example, the second electronic device (902) can include components of the second electronic device (902) of FIG. 9 and execute one or more programs including an automatic speech recognition module (952) and / or a text-to-speech module (954). The automatic speech recognition module (952) and the text-to-speech module (954) can perform functions identical to or similar to the automatic speech recognition module (1021) and the text-to-speech module (1029) of FIG. 9, respectively.

[0273] According to one embodiment, the first electronic device (901) and the second electronic device (902) may perform at least one function (or operation) in conjunction with each other, or may independently perform at least one function (or operation). For example, the second electronic device (902) may perform voice recognition for a voice input using an automatic voice recognition module (952). The second electronic device (902) may perform a function corresponding to the voice input based on the voice recognition. For example, the second electronic device (902) may transmit a command corresponding to the recognized voice command to the first electronic device (901). The second electronic device (902) may output data received from the first electronic device (901). For example, the second electronic device (902) may convert data received from the first electronic device (901) into voice using a text-to-speech conversion module (954) and output the converted voice.

[0274] According to one embodiment, the electronic device can analyze at least one of a color of a background screen displayed on a display, a color of at least one indicator, or a position at which at least one indicator is displayed within the background screen. The electronic device generates a prompt for instructing a property change using at least one of the analysis information, location information of the electronic device, or eyesight information of a user of the electronic device, wherein the eyesight information of the user of the electronic device can include at least one of a screen mode input by the user, a color of the screen, a sharpness of the screen, a screen size, whether the screen is enlarged or reduced compared to a base screen, color blindness, color weakness, the user's eyesight, the user's field of vision, a wide angle, color vision, refraction, binocular vision, or age.

[0275] According to one embodiment, the electronic device can use the generated prompt to determine an attribute of the at least one indicator to change at least one of a color of the background screen and the at least one indicator, a color of a portion adjacent to the at least one indicator, a thickness of a border of the at least one indicator, or a color of text displayed with the at least one indicator, and can change the attribute of the at least one indicator based on the generated prompt.

[0276] According to one embodiment, the electronic device can use the generated prompt to determine an attribute of at least one indicator so that the background screen and the indicator are distinguished, and change the attribute of the at least one indicator to the determined attribute based on at least one of a change in ambient brightness of the electronic device, a battery status of the electronic device, a location of the electronic device, or an update status of the indicator.

[0277] According to one embodiment, the electronic device may obtain information about the surrounding environment of the electronic device using a camera, determine whether a background screen on which at least one indicator is displayed will change based on a location where a user of the electronic device is located, and generate a prompt to change a property of at least one indicator so that the at least one indicator can be recognized in the currently displayed background screen based on a determination that the background screen on which the at least one indicator is displayed will not change.

[0278] According to one embodiment, the electronic device may obtain information about the colors of the background screen at a first time in the past and the background screen at a second time in the past based on the present based on the expectation that the background screen on which at least one indicator is displayed will change, and may generate a prompt such that the color of the at least one indicator is different from the color of the background screen at the first time and also different from the color of the background screen at the second time.

[0279] According to one embodiment, the electronic device may acquire information about the surrounding environment of the electronic device using a camera based on detecting that a user's body is in contact with the electronic device, and may control the electronic device not to acquire the information about the surrounding environment based on moving the electronic device to a designated location or changing the electronic device from a virtual reality (VR) mode to either a mixed reality (MR) mode or an augmented reality (AR) mode.

[0280] According to one embodiment, the electronic device acquires information about the surrounding environment of the electronic device using a camera based on the detection of a user's body coming into contact with the electronic device, and when the electronic device detects a new movement beyond a specified range while no movement has been detected for a specified period of time or a new object is detected while the electronic device is moving, the electronic device determines to change to either a mixed reality (MR) mode or an augmented reality (AR) mode and controls the electronic device not to acquire the information about the surrounding environment.

[0281] According to one embodiment, the electronic device may acquire information about the surrounding environment of the electronic device using a camera based on detecting that a user's body is in contact with the electronic device, and classify zones based on a direction in which the camera of the electronic device is facing or a direction in which the user of the electronic device is facing. The electronic device may generate a background based on the information about the surrounding environment acquired using the camera for each zone, determine an attribute of at least one indicator so as to be distinguished from the generated background, display the background generated based on the direction in which the camera of the electronic device is facing or the direction in which the user of the electronic device is facing, and change an attribute of at least one indicator to display it on the background.

[0282] According to one embodiment, the electronic device may acquire information about the surrounding environment of the electronic device using a camera based on detecting that a user's body is in contact with the electronic device, determine characteristics of a location where the electronic device is located using the surrounding environment information acquired through the camera and location information acquired using a global positioning system (GPS), and generate a prompt that instructs to change the properties of at least one indicator so that the at least one indicator is distinguished from the background by considering both the characteristics of the location where the electronic device is located and the vision information of the user of the electronic device.

[0283] The embodiments of this document disclosed in this specification and drawings are merely specific examples presented to easily explain the technical contents according to the embodiments of this document and to facilitate the understanding of the embodiments of this document, and are not intended to limit the scope of the embodiments of this document. Therefore, the scope of one embodiment of this document should be interpreted to include all changes or modified forms derived based on the technical idea of ​​one embodiment of this document, in addition to the embodiments disclosed herein.

Claims

1. In electronic devices, At least one processor; Memory that stores instructions; display; and Including a camera, The above instructions, when executed by the at least one processor, cause the electronic device to: Analyzing at least one of a color of a background screen displayed on the display, a color of at least one indicator, or a position at which at least one indicator is displayed within the background screen, An electronic device that controls generating a prompt to change an attribute of at least one indicator so that the at least one indicator can be recognized as distinct from the background based on the analysis information.

2. In paragraph 1, The above electronic device Analyzing at least one of a color of a background screen displayed on the display, a color of at least one indicator, or a position at which at least one indicator is displayed within the background screen, Generate a prompt for instructing a property change using at least one of the analysis information, the location information of the electronic device, or the vision information of the user of the electronic device, wherein the vision information of the user of the electronic device is An electronic device that includes at least one of the following information: screen mode input by the user, screen color, screen sharpness, screen size, screen magnification or reduction relative to the native screen, color blindness, color weakness, user's eyesight, user's field of vision, wide angle, color vision, refraction, binocular vision, or age.

3. In paragraph 1, The above electronic device Using the generated prompt, determine a property of the at least one indicator so that at least one of the color of the background screen and the at least one indicator, the color of a portion adjacent to the at least one indicator, the thickness of the border of the at least one indicator, or the color of text displayed with the at least one indicator is different; An electronic device controlling change of a property of at least one indicator based on the generated prompt.

4. In paragraph 1, The above electronic device Using the generated prompt, determine the properties of at least one indicator so that the background screen and the indicator are distinguished, An electronic device that changes a property of at least one indicator based on at least one of a change in ambient brightness of the electronic device, a battery status of the electronic device, a location of the electronic device, or an update status of the indicator.

5. In paragraph 1, The above electronic device Obtaining information about the surrounding environment of the electronic device using a camera, determining whether the background screen on which at least one indicator is displayed will change based on the location of the user of the electronic device; An electronic device that generates a prompt to change a property of said at least one indicator so that said at least one indicator can be recognized on a currently displayed background screen based on a determination that the background screen on which said at least one indicator is displayed will not change.

6. In paragraph 5, The above electronic device Based on the expectation that the background screen on which at least one indicator is displayed will change, information about the color of the background screen at a first point in time in the past and the background screen at a second point in time in the past is acquired based on the present, An electronic device that generates a prompt such that the color of said at least one indicator is different from the color of the background screen at said first point in time and also different from the color of the background screen at said second point in time.

7. In paragraph 1, The above electronic device Acquire information about the surrounding environment of the electronic device using a camera based on detecting that the user's body is in contact with the electronic device, If the above electronic device is moved to a designated location or An electronic device that controls not to acquire surrounding environment information based on the electronic device changing from VR (virtual reality) mode to either MR (mixed reality) mode or AR (augmented reality) mode.

8. In paragraph 1, The above electronic device Acquire information about the surrounding environment of the electronic device using a camera based on detecting that the user's body is in contact with the electronic device, When the electronic device detects a new movement beyond a specified range while no movement has been detected for a specified period of time, or a new object is detected while the electronic device is moving, the electronic device determines that the electronic device will change to either MR (mixed reality) mode or AR (augmented reality) mode, An electronic device that controls the acquisition of information about the surrounding environment.

9. In paragraph 1, The above electronic device Acquire information about the surrounding environment of the electronic device using a camera based on detecting that the user's body is in contact with the electronic device, Classify the area based on the direction in which the camera of the electronic device is looking or the direction in which the user of the electronic device is looking, For each area, a background is created based on the surrounding environment information acquired using the above camera. Determine the properties of at least one indicator so as to be distinguished from the generated background, An electronic device that displays a background generated based on a direction viewed by the camera of the electronic device or a direction viewed by a user of the electronic device, and changes the properties of at least one indicator to display it on the background.

10. In paragraph 1, The above electronic device Acquire information about the surrounding environment of the electronic device using a camera based on detecting that the user's body is in contact with the electronic device, The characteristics of the location where the electronic device is located are determined using the surrounding environment information acquired through the camera and the location information acquired using the global positioning system (GPS). An electronic device that generates a prompt for instructing a change in the properties of at least one indicator so that the at least one indicator is distinguished from the background, taking into account both the characteristics of the location where the electronic device is located and the vision information of a user of the electronic device.

11. In the method of operating an electronic device, An action of analyzing at least one of a color of a background screen displayed on a display, a color of at least one indicator, or a position at which at least one indicator is displayed within the background screen; An action of generating a prompt that instructs a change in an attribute of said at least one indicator so that said at least one indicator can be recognized as distinct from the background based on the analysis information; and A method comprising an action for controlling changing a property of at least one indicator based on the generated prompt.

12. In paragraph 11, An operation that analyzes at least one of a color of a background screen displayed on a display, a color of at least one indicator, or a position at which at least one indicator is displayed within the background screen; and Further comprising an action of generating a prompt that instructs a property change using at least one of the analysis information, location information of the electronic device, or vision information of a user of the electronic device; The user's vision information of the above electronic device A method including at least one of the following information: screen mode input by the user, screen color, screen sharpness, screen size, whether the screen is enlarged or reduced compared to the default screen, color blindness, color weakness, the user's eyesight, the user's field of vision, wide angle, color vision, refraction, binocular vision, or age.

13. In paragraph 11, An operation of determining a property of said at least one indicator so that the color of said background screen and said at least one indicator is different using said generated prompt; and A method further comprising an action of changing a property of at least one indicator based on the determined property.

14. In paragraph 11, An operation of determining a property of at least one indicator so that the background screen and the indicator are distinguished using the generated prompt; and A method further comprising an action of changing a property of the at least one indicator to a property determined based on at least one of a change in ambient brightness of the electronic device, a battery status of the electronic device, a location of the electronic device, or an update status of the indicator.

15. In paragraph 11, An action of obtaining information about the surrounding environment of the electronic device using a camera; An action of determining whether the background screen on which the at least one indicator is displayed is to change based on a location where a user of the electronic device is located; A method further comprising the action of generating a prompt to change a property of the at least one indicator so that the at least one indicator can be recognized on the currently displayed background screen based on a determination that the background screen on which the at least one indicator is displayed will not change.

Images