Electronic device for displaying additional object in augmented reality image and method for driving the electronic device

Abstract

An electronic device includes a communication module, a display, and at least one processor operatively connected to the communication module and the display. The at least one processor may be configured to: receive an augmented reality image from at least one external device performing wireless communication with the communication module via the communication module; display an execution screen of an application associated with the augmented reality image on the display; determine whether an object associated with input information input to the electronic device is included in the augmented reality image; and display an additional object on the object based at least in part on the input information. In addition, various embodiments may be recognized through the specification.

Description

Technical Field

[0001] Embodiments disclosed in the present disclosure relate to a technology of displaying an additional object, such as an emoticon, on an object included in an augmented reality image obtained by an electronic device. Background Art

[0002] Electronic devices each having a display have been widely used. The electronic device can display a screen obtained by the electronic device using the display. Alternatively, when running an application, the electronic device can display the running screen on the display.

[0003] The screen may include one or more objects. The electronic device may display additional objects on the objects. For example, the electronic device may display emoticons with facial expression shapes on the objects. Summary of the invention

[0004] [Technical issues]

[0005] The electronic device may obtain an augmented reality image representing a real environment from an augmented reality device. The augmented reality image may include one or more objects. The electronic device may display a specified additional object on any object. However, the electronic device may not be able to identify whether any object is an object meaningful to the user. Therefore, there is a problem in which an inappropriate additional object is displayed on an object meaningful to the user, or an unnecessary additional object is displayed on an object meaningless to the user.

[0006] In addition, when an additional object is displayed on an object, the electronic device may not determine whether to display any positive or negative emotions. Therefore, there is a problem in that since it can only display a specified additional object, it is not possible to provide the user with an emotional expression according to the object using the additional object.

[0007] [Technical solution]

[0008] According to aspects of the present disclosure, an electronic device is provided. The electronic device may include a communication module, a display, and at least one processor operatively connected to the communication module and the display. The at least one processor may be configured to: receive an augmented reality image from at least one external device performing wireless communication with the communication module via the communication module; display an execution screen of an application associated with the augmented reality image on the display; determine whether an object associated with input information input to the electronic device is included in the augmented reality image; and display an additional object on the object based at least in part on the input information.

[0009] According to another aspect of the present disclosure, an electronic device is provided. The electronic device may include a communication module, a display, a memory, and at least one processor operatively connected to the communication module, the display, and the memory. The at least one processor may be configured to: receive an augmented reality image from at least one external device performing wireless communication with the communication module via the communication module; display an execution screen of an application associated with the augmented reality image on the display; determine whether an object associated with stored information stored in the memory is included in the augmented reality image; and display an additional object on the object based at least in part on the stored information.

[0010] According to another aspect of the present disclosure, a driving method of an electronic device is provided. The driving method may include: receiving an augmented reality image from at least one external device performing wireless communication with the electronic device; displaying a running screen of an application associated with the augmented reality image on a display; determining whether an object associated with at least one of first information input by a user and second information stored in a memory is included in the augmented reality image; and performing at least one of the following items on the object: generating a first additional object based on the first information, and generating a second additional object based on the second information.

[0011] [Beneficial Effects]

[0012] According to the embodiments disclosed in the present disclosure, an object meaningful to a user may be accurately recognized, and an additional object required by the user may be displayed on the meaningful object.

[0013] Furthermore, according to the embodiments disclosed in the present disclosure, an emotion expression, such as a positive emotion or a negative emotion, may be provided to the user according to the type or state of an object.

[0014] Furthermore, various effects directly or indirectly determined by the present disclosure can be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram illustrating an electronic device in a network environment according to various embodiments;

[0016] Figure 2 is a block diagram showing a display device according to various embodiments;

[0017] Figure 3 is a simplified diagram of an electronic device and external devices according to an embodiment.

[0018] Figure 4 is a flowchart illustrating a driving method of an electronic device according to an embodiment;

[0019] Figure 5 is a diagram showing that an electronic device according to an embodiment is configured to display an additional object on an object recognized by an external device;

[0020] Figure 6 is a diagram showing that an electronic device according to another embodiment is configured to display an additional object on an object recognized by an external device;

[0021] Figure 7 is a diagram illustrating displaying an additional object on a recognized object in an electronic device according to an embodiment;

[0022] Figure 8 is a simplified diagram showing that an electronic device according to an embodiment is configured to display an additional object on an augmented reality image in response to movement of an external device;

[0023] Fig.9A and Fig. 9B is a simplified diagram illustrating recognizing an object using an external device in an electronic device or recognizing an object using the electronic device according to an embodiment;

[0024] Fig.10 is a simplified diagram illustrating a process of viewing reality using an external device and generating an additional object in response to a shape of an object in an electronic device according to an embodiment;

[0025] FIG. 11A to FIG. 11C is a diagram illustrating displaying an additional object on an object of an augmented reality image viewed by an external device based on input information in an electronic device according to an embodiment; and

[0026] Fig.12 is a diagram illustrating displaying an additional object on an object of an augmented reality image viewed by an external device based on information stored in the electronic device according to an embodiment.

[0027] With regard to the description of the drawings, the same or similar reference numerals may be used to refer to the same or similar components. DETAILED DESCRIPTION

[0028] Hereinafter, various embodiments of the present disclosure may be described with reference to the accompanying drawings. However, it should be understood that this is not intended to limit the present disclosure to a specific implementation form, but includes various modifications, equivalent forms and / or alternative forms of the embodiments of the present disclosure.

[0029] Figure 1 1 is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments. Figure 1, the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input device 150, a sound output device 155, a display device 160, an audio module 170, a sensor module 176, an interface 177, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a subscriber identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the components (e.g., the display device 160 or the camera module 180) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some of the components may be implemented as a single integrated circuit.For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented to be embedded in the display device 160 (eg, a display).

[0030] The processor 120 may run, for example, software (e.g., program 140) to control at least one other component (e.g., hardware component or software component) of the electronic device 101 connected to the processor 120, and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, the processor 120 may load a command or data received from another component (e.g., sensor module 176 or communication module 190) into the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resultant data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 123 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is independent of or combined with the main processor 121 in operation. Additionally or alternatively, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to be specifically used for a specified function. The auxiliary processor 123 may be implemented separately from the main processor 121 , or as part of the main processor 121 .

[0031] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., display device 160, sensor module 176, or communication module 190) among the components of the electronic device 101 (not the main processor 121), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., display device 160, sensor module 176, or communication module 190) among the components of the electronic device 101 together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., camera module 180 or communication module 190) that is functionally related to the auxiliary processor 123.

[0032] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

[0033] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0034] The input device 150 may receive commands or data from outside the electronic device 101 (e.g., a user) to be used by other components of the electronic device 101 (e.g., the processor 120). The input device 150 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (e.g., a stylus).

[0035] The sound output device 155 can output a sound signal to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records, and the receiver can be used for incoming calls. Depending on the embodiment, the receiver can be implemented as a separate part from the speaker, or as a part of the speaker.

[0036] The display device 160 may visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display device 160 may include a touch circuit adapted to detect a touch or a sensor circuit (e.g., a pressure sensor) adapted to measure the strength of a force caused by a touch.

[0037] The audio module 170 may convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 170 may obtain sound via the input device 150, or output sound via the sound output device 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0038] The sensor module 176 may detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a state of a user) outside the electronic device 101, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0039] The interface 177 may support one or more specific protocols to be used to connect the electronic device 101 directly (e.g., wired) or wirelessly to an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0040] The connection end 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. According to an embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0041] The haptic module 179 may convert the electrical signal into mechanical stimulation (eg, vibration or motion) or electrical stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0042] The camera module 180 may capture still images or moving images. According to an embodiment, the camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0043] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0044] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0045] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently from the processor 120 (e.g., an application processor (AP)) and supporting direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a Wide Area Network (WAN))). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multiple components (e.g., multiple chips) separated from each other. The wireless communication module 192 may identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the subscriber identification module 196.

[0046] The antenna module 197 may transmit or receive a signal or power to or from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a radiating element formed of a conductive material or a conductive pattern formed in or on a substrate (e.g., a PCB). According to an embodiment, the antenna module 197 may include a plurality of antennas. In this case, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 198 or the second network 199) may be selected from the plurality of antennas by, for example, the communication module 190 (e.g., the wireless communication module 192). A signal or power may then be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna. According to an embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiating element may be additionally formed as a part of the antenna module 197.

[0047] At least some of the above components may be connected to each other via an inter-peripheral communication scheme (e.g., a bus, general purpose input output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and communicatively transmit signals (e.g., commands or data) therebetween.

[0048] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via the server 108 connected to the second network 199. Each of the electronic device 102 and the electronic device 104 may be a device of the same type as the electronic device 101, or a device of a different type from the electronic device 101. According to an embodiment, all or some operations to be executed in the electronic device 101 may be executed in one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices that receive the request may execute the requested at least part of the function or service, or execute another function or another service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a partial reply to the request with or without further processing the result. To this end, for example, cloud computing technology, distributed computing technology, or client-server computing technology may be used.

[0049] Figure 2 2 is a block diagram showing a display device 160 according to various embodiments. Figure 2, the display device 160 may include a display 210 and a display driver integrated circuit (DDI) 230 that controls the display 210. The DDI 230 may include an interface module 231, a memory 233 (e.g., a buffer memory), an image processing module 235, or a mapping module 237. The DDI 230 may receive image information from another component of the electronic device 101 via the interface module 231, the image information including image data or an image control signal corresponding to a command to control the image data. For example, according to an embodiment, the image information may be received from a processor 120 (e.g., a main processor 121 (e.g., an application processor)) or an auxiliary processor 123 (e.g., a graphics processing unit) that operates independently of the function of the main processor 121. The DDI 230 may communicate with the touch circuit 250 or the sensor module 176, for example, via the interface module 231. The DDI 230 may also store at least a portion of the received image information in the memory 233, for example, frame by frame. The image processing module 235 may perform pre-processing or post-processing (e.g., adjustment of resolution, brightness, or size) with respect to at least a portion of the image data. According to an embodiment, the pre-processing or post-processing may be performed, for example, at least in part based on one or more characteristics of the image data or one or more characteristics of the display 210. The mapping module 237 may generate a voltage value or a current value corresponding to the image data pre-processed or post-processed by the image processing module 235. According to an embodiment, the generation of the voltage value or the current value may be performed, for example, at least in part based on one or more attributes of the pixel (e.g., an array of pixels (such as RGB stripes or a pentile structure), or the size of each sub-pixel). At least some pixels of the display 210 may be driven, for example, at least in part based on the voltage value or the current value, so that visual information corresponding to the image data (e.g., text, an image, or an icon) may be displayed via the display 210.

[0050] According to an embodiment, the display device 160 may further include a touch circuit 250. The touch circuit 250 may include a touch sensor 251 and a touch sensor IC 253 for controlling the touch sensor 251. The touch sensor IC 253 may control the touch sensor 251 to sense a touch input or a hovering input relative to a specific position on the display 210. For example, to achieve this, the touch sensor 251 may detect (e.g., measure) a change in a signal (e.g., voltage, light amount, resistance, or one or more charge amounts) corresponding to a specific position on the display 210. The touch circuit 250 may provide input information (e.g., position, area, pressure, or time) indicating a touch input or a hovering input detected via the touch sensor 251 to the processor 120. According to an embodiment, at least a portion of the touch circuit 250 (e.g., the touch sensor IC 253) may be formed as a part of the display 210 or the DDI 230, or as a part of another component (e.g., the auxiliary processor 123) provided outside the display device 160.

[0051] According to an embodiment, the display device 160 may further include at least one sensor (e.g., a fingerprint sensor, an iris sensor, a pressure sensor, or an illumination sensor) of the sensor module 176 or a control circuit of the at least one sensor. In this case, the at least one sensor or the control circuit of the at least one sensor may be embedded in a portion of a component of the display device 160 (e.g., the display 210, the DDI 230, or the touch circuit 250). For example, when the sensor module 176 embedded in the display device 160 includes a biometric sensor (e.g., a fingerprint sensor), the biometric sensor may obtain biometric information (e.g., a fingerprint image) corresponding to a touch input received via a portion of the display 210. As another example, when the sensor module 176 embedded in the display device 160 includes a pressure sensor, the pressure sensor may obtain pressure information corresponding to a touch input received via a portion or the entire area of ​​the display 210. According to an embodiment, the touch sensor 251 or the sensor module 176 may be disposed between pixels in a pixel layer of the display 210, or above or below the pixel layer.

[0052] Figure 3 1 is a simplified diagram of an electronic device 101 and an external device 301 according to an embodiment. The external device 301 may be an electronic device that displays an image or generates image data and performs wireless communication with the electronic device 101. For example, the external device 301 may be an augmented reality device such as a wearable device or smart glasses. Figure 3 Schematically shown as smart glasses, but not limited to this. In addition, Figure 3Only one external device 301 is shown, but is not limited thereto, and the electronic device 101 may perform wireless communication with one or more external devices 301 .

[0053] In an embodiment, the electronic device 101 may obtain an image from the outside using the communication module 190. For example, when the external device 301 is an augmented reality device, the electronic device 101 may obtain an augmented reality image from the external device 301. The processor 120 of the electronic device 101 may analyze the obtained image. The processor 120 may modify the obtained image based at least in part on information input to the input device 150 of the electronic device 101 or information stored in the memory 130. The processor 120 may display the modified image on the display 210 using the display device 160.

[0054] In an embodiment, the electronic device 101 may obtain image data from the outside using the communication module 190. For example, the electronic device 101 may obtain augmented reality image data from the external device 301. The processor 120 of the electronic device 101 may analyze the obtained image.

[0055] In an embodiment, the external device 301 may generate image data. For example, when the external device 301 is an augmented reality device, the external device 301 may generate an augmented reality image in which an object actually visible to the wearer of the external device 301 and additional information are combined with each other. The external device 301 may display the augmented reality image to the wearer of the external device 301, and may convert the augmented reality image into image data. For another example, the external device 301 may obtain a real screen using a separate input device (such as a camera), and may convert this real screen into image data.

[0056] In an embodiment, the external device 301 may output image data showing an augmented reality image. The external device 301 may obtain information about at least one object included in the augmented reality image. The external device 301 may transmit the information about the at least one object to the electronic device 101.

[0057] In an embodiment, the processor 120 of the electronic device 101 may modify the obtained image based at least in part on input information input to the input device 150 of the electronic device 101 or stored information stored in the memory 130. The processor 120 may modify data about at least one object included in the image data based at least in part on the input information or the stored information. For example, the processor 120 may add an additional object (such as an emoticon) to the augmented reality image to reflect the content of the input information or the stored information in the image data.

[0058] In an embodiment, the communication module 190 of the electronic device 101 may send the modified image data to the external device 301. The external device 301 may display an augmented reality image corresponding to the modified image data so that the user can view at least a portion of the modified augmented reality image. For example, the external device 301 may use the image data to display additional objects such as emoticons on the augmented reality image.

[0059] Figure 4 is a flowchart illustrating a driving method of the electronic device 101 according to an embodiment.

[0060] In operation 410, the electronic device 101 according to an embodiment may receive a signal from an external device (eg, Figure 3 The electronic device 101 may receive the augmented reality image from the external device 301. For example, the electronic device 101 may receive image data corresponding to the entire augmented reality image from the external device 301. For another example, the electronic device 101 may receive data associated with at least one object included in the augmented reality image from the external device 301. The data associated with the object may include information about the type of the object, the shape of the object, and the relationship between the object and the user.

[0061] In operation 420, the electronic device 101 according to the embodiment may display an execution screen of an application associated with the augmented reality image on the display 210. The electronic device 101 may execute an application corresponding to the obtained augmented reality image or an object included in the obtained augmented reality image. For example, when the augmented reality image has content associated with the health of the user of the electronic device 101, or includes food or fitness equipment as an object associated with the health of the user, the electronic device 101 may execute a health-related application (e.g., Samsung Health TM For another example, when the augmented reality image is an image associated with an object, the electronic device 101 may run an application related to payment (eg, Samsung Pay). TM ). The electronic device 101 may display an execution screen of the executed application on the display 210.

[0062] In operation 430, the electronic device 101 according to the embodiment may determine whether an object associated with at least one of the first information input by the user and / or the second information stored in the memory 130 is included in the augmented reality image. The electronic device 101 may identify a plurality of objects included in the augmented reality image. The electronic device 101 may determine whether each of the plurality of objects is associated with at least one of the first information and / or the second information. In the present disclosure, the first information may be referred to as input information, and the second information may be referred to as stored information.

[0063] In an embodiment, the first information may be information directly set by the user. The electronic device 101 may determine whether to include an object associated with the information directly set by the user. For example, when the user inputs health-related information to the input device 150 of the electronic device 101 using touch or voice, the electronic device 101 may determine whether food or fitness equipment as an object associated with the user's health is included in the augmented reality image.

[0064] In an embodiment, the second information may be usage pattern data in which the usage pattern of the user is accumulated. The electronic device 101 may determine whether to include an object associated with the usage pattern data. For example, when the search frequency of a keyword associated with health is high in the user's information search record stored in the memory 130 of the electronic device 101, the electronic device 101 may determine whether food or fitness equipment as an object associated with the user's health is included in the augmented reality image.

[0065] In operation 440, when an object is included in the augmented reality image, the electronic device 101 according to an embodiment may identify the object as meaningful to the user. The processor 120 of the electronic device 101 may be configured to identify the object as meaningful to the user and allow the external device 301 to recognize the object when the object associated with at least one of the first information and / or the second information is included in the augmented reality image. The external device 301 may perform wireless communication with the electronic device 101, and may recognize the object associated with at least one of the first information and / or the second information on the augmented reality image.

[0066] In operation 450, the electronic device 101 according to the embodiment may generate at least one of the following objects on the object of the augmented reality image: a first additional object based on the first information, and a second additional object based on the second information. For example, the electronic device 101 may generate an emoticon by reflecting the content of the first information or the second information, the emoticon representing the first information or the second information as a facial expression.

[0067] In an embodiment, the electronic device 101 may combine additional object data capable of generating and displaying an additional object with the image data. The electronic device 101 may send the image data combined with the additional object data to the external device 301. The external device 301 may use the image data combined with the additional object data to view an image in which the additional object is combined with the original augmented reality image.

[0068] Figure 5 is a diagram showing that the electronic device 101 is configured to display an additional object 521 on an object 511 recognized by the external device 301 according to an embodiment.

[0069] In an embodiment, the reality 501 may include at least one object 511. The electronic device 101 may store information associated with the at least one object 511. For example, the electronic device 101 may store information associated with the type of the at least one object 511, the shape of the at least one object 511, or the priority of the user to the at least one object 511.

[0070] In an embodiment, when viewing reality 501 through external device 301, external device 301 may display an augmented reality image in which additional object 521 is generated on at least one object 511 included in reality 501. Processor 120 of electronic device 101 may be configured to display additional object 521 on at least one object 511.

[0071] In an embodiment, the additional object 521 may be an emoticon that represents a facial expression on the object 511 by at least partially reflecting the first information input to the electronic device 101 or the second information stored in the electronic device 101. The additional object 521 may display the meaning of the object 511 to the user using the first information or the second information. The additional object 521 may be implemented as a positive facial expression or a negative facial expression according to the meaning of the object 511 to the user. The additional object 521 may emotionally convey the meaning of the object 511 to the user depending on the first information or the second information.

[0072] In an embodiment, the user may input the recommended daily water intake and today's water intake to the electronic device 101. The processor 120 of the electronic device 101 may be configured to allow health-related objects to have a high priority in the external device 301, and may be configured to identify health-related objects in the augmented reality image. For another example, the memory 130 of the electronic device 101 may store information about the recommended daily water intake and record it when the user performs a search related to water intake.

[0073] In an embodiment, when the reality 501 is viewed using the external device 301 and the water in the bottle is visible, the electronic device 101 may identify the water in the bottle as an object 511 that is meaningful to the user, and may generate additional object data capable of generating an additional object 521, thereby sending the additional object data to the external device 301. The external device 301 may use the additional object data to display an emoticon representing a positive recommendation emotion as at least partially overlapping the bottle to recommend the user viewing the augmented reality image to drink the liquid.

[0074] In an embodiment, the processor 120 of the electronic device 101 may be configured to cause at least one or more of the electronic device 101 or the external device 301 to provide a visual guide 531 associated with the additional object 521. For example, the external device 301 may generate text information indicating a current value compared to a target value that can be visually identified and / or a graph corresponding to the text information. The external device 301 may display the visual guide 531 as at least partially overlapping at least one object 511. The user can accurately identify the intention that the additional object 521 wants to express or the extent to which the additional object 521 wants to recommend an action with the aid of the visual guide 531. For example, when the object 511 is water in a bottle, the external device 301 may receive the recommended daily water intake and the user's water intake today from the electronic device 101. The external device 301 may display today's water intake compared to the recommended daily water intake in the form of text information and a circular graph corresponding to the text information.

[0075] In an embodiment, the processor 120 of the electronic device 101 may be configured to enable at least one or more of the electronic device 101 or the external device 301 to provide a voice guidance 541 associated with the additional object 521. For example, when the object 511 is water in a bottle, the electronic device 101 or the external device 301 may use a voice application (e.g., Bixby TM ) outputs the voice “Please drink me! (Drink me)” 541.

[0076] In an embodiment, when the external environment of at least one or more of the electronic device 101 or the external device 301 changes, the processor 120 of the electronic device 101 may change the shape of the additional object 522 in response to the changed situation. For example, when the reality 502 or the object 512 changes, the processor 120 of the electronic device 101 may modify the additional object data by reflecting the changed situation. For example, when the object 512 is a bottle after the water is drunk, the processor 120 of the electronic device 101 may generate the additional object 522 by reflecting the situation in which the increase in water intake today is the same as the amount of water in the bottle, and the additional object 522 may be sent to the external device 301. The external device 301 may display an emoticon representing a positive, satisfactory emotion as at least partially overlapping the bottle to indicate that the water intake is a satisfactory need for the user.

[0077] In an embodiment, the processor 120 of the electronic device 101 may be configured to: when the external environment of at least one or more of the electronic device 101 or the external device 301 changes, provide visual guidance 532 in response to the changed situation. For example, when the object 512 is a bottle after the water is drunk, the processor 120 of the electronic device 101 may send data in which the increase in water intake today is the same as the amount of water in the bottle to the external device 301. The external device 301 may increase and display the increase in the user's water intake today, which is the same as the amount of water in the bottle.

[0078] In an embodiment, the processor 120 of the electronic device 101 may be configured to provide voice guidance 542 in response to the changed situation when the external environment of at least one or more of the electronic device 101 or the external device 301 changes. For example, when the object 512 is a bottle of water that has been drunk up, the electronic device 101 or the external device 301 may output the voice "Great job!" using a voice utterance application.

[0079] Figure 6 6 is a simplified diagram in which the electronic device 101 is configured to display an additional object 621 on the object 611 recognized by the external device 301 according to an embodiment.

[0080] In an embodiment, the reality 601 may include at least one object 611. The electronic device 101 may determine whether the at least one object 611 is an object meaningful to the user based on input information directly input by the user or stored information stored in the memory 130.

[0081] In an embodiment, when viewing the reality 601 through the external device 301, the external device 301 may display an augmented reality image including generating an additional object 621 on at least one object 611 in the reality 601. The additional object 621 may represent a facial expression on the object 611 by at least partially reflecting the first information input to the electronic device 101 or the second information stored in the electronic device 101. The additional object 621 may emotionally convey the meaning of the object 611 to the user based on the first information or the second information.

[0082] In an embodiment, the user may input the allowed daily caffeine intake and today's caffeine intake into the electronic device 101. The processor 120 of the electronic device 101 may be configured to allow health-related objects to have a high priority in the external device 301, and may be configured to identify health-related objects in the augmented reality image. For another example, the memory 130 of the electronic device 101 may store information about the allowed daily caffeine intake and record it when the user performs a search related to caffeine intake.

[0083] In an embodiment, when the reality 601 is viewed using the external device 301 and water in the caffeine drink is visible, the electronic device 101 may identify the caffeine drink as an object 611 that is meaningful to the user, and may generate additional object data capable of generating an additional object 621, thereby sending the additional object data to the external device 301. The external device 301 may use the additional object data to display an emoticon that expresses a worrying emotion in a manner that at least partially overlaps the caffeine drink, so as to alert the user viewing the augmented reality image of excessive caffeine intake.

[0084] In an embodiment, the processor 120 of the electronic device 101 may be configured to cause at least one or more of the electronic device 101 or the external device 301 to provide a visual guide 631 associated with the additional object 621. For example, when the object 611 is a caffeine drink, the external device 301 may receive the allowed daily caffeine intake and the user's today's caffeine intake from the electronic device 101. The external device 301 may display the user's today's caffeine intake compared to the allowed daily caffeine intake in the form of text information and a circular graph corresponding to the text information.

[0085] In an embodiment, the processor 120 of the electronic device 101 may be configured to enable at least one or more of the electronic device 101 or the external device 301 to provide a voice guidance 641 associated with the additional object 621. For example, when the object 611 is a caffeinated drink, the electronic device 101 or the external device 301 may output a voice “Oh, I'm highly caffeinated” 641 using a voice utterance application.

[0086] In an embodiment, when the external environment of at least one or more of the electronic device 101 or the external device 301 changes, the processor 120 of the electronic device 101 may change the shape of the additional object 622 in response to the changed situation. For example, when the reality 602 or the object 612 changes, the processor 120 of the electronic device 101 may modify the additional object data by reflecting the changed situation. For example, when the object 612 is a cup from which caffeine has been drunk, the processor 120 of the electronic device 101 may generate the additional object 622 by reflecting the situation in which the increase in today's caffeine intake is the same as the amount of caffeine contained in the caffeine drink, and may send the additional object 622 to the external device 301. The external device 301 may display an emoticon representing a negative, more worrying emotion in a manner that at least partially overlaps the cup to convey to the user the meaning of avoiding drinking more caffeine.

[0087] In an embodiment, the processor 120 of the electronic device 101 may be configured to provide the visual guidance 632 in response to the changed situation when the external environment of at least one or more of the electronic device 101 or the external device 301 changes. For example, when the object 612 is a cup after the caffeine is drunk, the processor 120 of the electronic device 101 may send data in which the increase amount of today's caffeine intake is the same as the amount of caffeine contained in the caffeine drink to the external device 301. The external device 301 may increase and display the amount of today's caffeine intake being the same as the amount of caffeine contained in the caffeine drink.

[0088] In an embodiment, the processor 120 of the electronic device 101 may be configured to provide voice guidance 642 in response to the changed situation when the external environment of at least one or more of the electronic device 101 or the external device 301 changes. For example, when the object 612 is a cup after the caffeine drink is drunk, the electronic device 101 or the external device 301 may use a voice application to output a voice "Hope this can be your last caffeine of the day!".

[0089] Figure 7 is a diagram of displaying an additional object 721 on a recognized object 711 in the electronic device 101 according to an embodiment.

[0090] In an embodiment, the electronic device 101 may obtain an image including the object 712 using a camera included in the camera module 180. The electronic device 101 may display the obtained image on the display 210. The electronic device 101 may obtain information associated with the object 712. For example, when the object 712 is a food, the processor 120 of the electronic device 101 may obtain calorie information of the object 712.

[0091] In an embodiment, the electronic device 101 may determine whether the object 712 is meaningful to the user based at least in part on information directly input by the user or information according to a situation outside the electronic device 101. For example, when the user inputs the calories of a food or diet into the input device 150, the electronic device 101 may determine that the object 712 corresponding to the food is meaningful to the user. For another example, when the user has recently frequently searched for information about the calories of a food or diet or selected information about the calories of a food or diet with high priority, the electronic device 101 may determine that the object 712 corresponding to the food is meaningful to the user.

[0092] In an embodiment, the electronic device 101 may determine whether the object 712 is meaningful to the user based at least in part on usage pattern data in which the user's usage pattern is accumulated. For example, the electronic device 101 may accumulate a pattern in which the user has recently frequently searched for information about food or dietary calories, or selected information about food or dietary calories with high priority, and store the pattern in the memory 130. When the electronic device 101 views the object 712, the processor 120 of the electronic device 101 may refer to the usage pattern data stored in the memory 130 to determine that the object 712 corresponding to the food is meaningful to the user.

[0093] In an embodiment, the electronic device 101 may run an application 701 associated with the object 712. For example, when the object 712 is a food, the electronic device 101 may run a food application 701 associated with the type of food or the calories of the food. When the user selects a meaningful object 712, the application 701 may be executed according to the user's input. Alternatively, when the electronic device 101 recognizes the object 712 based on the storage information stored in the memory 130, the processor 120 of the electronic device 101 may automatically run the application 701.

[0094] In an embodiment, the processor 120 of the electronic device 101 may display an additional object 721 on the object 712 based on information input by the user or information stored in the memory 130. The processor 120 may determine that the object 712 has any one of positive or negative meanings to the user based on the information input by the user or the information stored in the memory 130. The processor 120 may display an additional object 721 representing the emotion associated with the object 712 by reflecting the meaning that the object 712 has to the user. For example, when the object 712 is a cake, the processor 120 may use the food application 701 to measure the calories of the cake. When the user inputs information related to calories to the electronic device 101, or when a search record related to calories is stored in the memory 130, because the calories of the cake are high, the processor 120 may generate an additional object 721 for representing a worrying emotion. The display 210 of the electronic device 101 may display an image showing the additional object 721 on the object 712.

[0095] In an embodiment, the processor 120 of the electronic device 101 may be configured to cause at least one or more of the electronic device 101 or the external device 301 to provide a visual guide 731 associated with the additional object 721. For example, when the object 712 is a cake, the processor 120 may load the recommended daily calorie intake and the user's today's calorie intake from the memory 130. The display 210 may display the user's today's calorie intake compared to the recommended daily calorie intake in the form of text information and a circular graph corresponding to the text information.

[0096] In an embodiment, the electronic device 101 may provide a voice guidance 741 associated with the additional object 721. For example, when the object 712 is a cake, the electronic device 101 may output a voice “Oh, I'm packed with unwanted calories” 741 using a voice utterance application.

[0097] In an embodiment, when the external environment of the electronic device 101 changes, the processor 120 of the electronic device 101 may change the shape of the additional object 722 in response to the changed situation. For example, when the state of the object 711 changes, the processor 120 of the electronic device 101 may modify the additional object data by reflecting the changed state of the object 711. For example, when the object 711 is a wrapping paper after eating a cake, the processor 120 of the electronic device 101 may generate the additional object 722 by reflecting a situation in which the increase in today's calorie intake is the same as the calories contained in the cake. The electronic device 101 may display an emoticon representing a positive emotion of eating and feeling full and satisfied in a manner that at least partially overlaps with the wrapping paper of the cake to convey to the user the meaning that it can be assured because the calories consumed so far are less than the recommended daily calorie intake.

[0098] In an embodiment, the processor 120 of the electronic device 101 may be configured to provide visual guidance 732 in response to the changed situation when the external environment of the electronic device 101 changes. For example, when the object 711 is a wrapping paper after eating a cake, the processor 120 of the electronic device 101 may generate data in which the increase in today's calorie intake is the same as the calories contained in the cake. The display 210 may increase and display the same amount of today's calorie intake as the calories contained in the cake.

[0099] In an embodiment, the processor 120 of the electronic device 101 may be configured to provide voice guidance 742 in response to the changed situation when the external environment of the electronic device 101 changes. For example, when the object 711 is the wrapping paper after eating the cake, the electronic device 101 may use a voice application to output the voice "It's OK. We haven't undo everything!".

[0100] Figure 8 8 is a simplified diagram in which the electronic device 101 is configured to display additional objects 821 and 822 on an augmented reality image in response to movement of the external device 301 according to an embodiment.

[0101] In an embodiment, a user may view reality using an external device 301. The external device 301 may display an augmented reality image 801 based on reality. The external device 301 may identify an object 811 on the augmented reality image 801. The external device 301 may determine whether the object 811 is meaningful to the user based on input information or stored information provided from the electronic device 101. For example, when the object 811 is a shoe, the external device 301 may determine that the object 811 is meaningful to the user based on information about the shoe provided from the electronic device 101 and input by the user and information storing a search history related to the shoe.

[0102] In an embodiment, the external device 301 may determine whether the object 811 has a positive emotion or a negative emotion to the user based on the input information provided from the electronic device 101 and the stored information. The external device 301 may analyze the emotion that the user has for the object 811 in the input information provided from the electronic device 101 or the stored information and humanize it to generate and display the additional object 821. For example, when the object 811 is the same shoe as the shoe stored in the memory 130 of the electronic device 101, the external device 301 may determine that the user wants to use the object 811. The external device 301 may generate the additional object 821 so that the object 811 also represents an expression of welcoming the user in the form of a facial expression. The external device 301 may display the additional object 821 in a form that at least partially overlaps with the object 811.

[0103] In an embodiment, the external device 301 may provide a voice guide 841 associated with the additional object 821. For example, when the object 811 is a pair of shoes that the user frequently searches for, the external device 301 may output a voice “You finally found us!” 841 using a voice utterance application.

[0104] In an embodiment, the external device 301 and the augmented reality image 801 displayed by the external device 301 may move with the movement of the user. When the augmented reality image 801 is moved, the state of the object 811 displayed on the augmented reality image 801 may be changed. For example, when the augmented reality image 801 changes its display position to display a position where the object 811 is not set in reality, the object 811 may be separated from the augmented reality image 801.

[0105] In an embodiment, the external device 301 may change and display the additional object 822 displayed on the object 811 in response to the movement of the augmented reality image 801. When the object 811 is out of the augmented reality image 801, the external device 301 may change and display the shape of the additional object 822 relative to the meaning of the user to the object 811. For example, when the object 811 is a pair of shoes that the user frequently searches for, and when the augmented reality image 801 changes its display position to display a position where the object 811 is not set in reality, the external device 301 may determine that the user has not found and missed the object 811. The external device 301 may change the additional object 822 to express an emotion in the form of a facial expression on the object 811 that allows the user to recognize whether the object 811 is missed by chance. The external device 301 may display the additional object 822 in a form that at least partially overlaps with the object 811.

[0106] In an embodiment, the external device 301 may provide a voice guidance 842 associated with the additional object 822. For example, when the object 811 is a pair of shoes that the user frequently searches for, and when the augmented reality image 801 changes its display position to display a position where the object 811 is not set on reality, the external device 301 may output a voice “I’m here!” 842 using a voice utterance application.

[0107] Fig.9A and Fig. 9B is a simplified diagram of recognizing an object 911 using an external device 300 in the electronic device 101 or recognizing an object 912 using the electronic device 101 according to another embodiment.

[0108] In an embodiment, Fig.9AAs shown, the external device 301 can view the object 911. The electronic device 101 can determine whether the object 911 viewed through the external device 301 is meaningful to the user. The processor 120 of the electronic device 101 can determine whether the object 911 is meaningful to the user based on the input information input by the user or the storage information stored in the memory 130. When it is determined that the object 911 is meaningful to the user, the processor 120 can combine the additional object data of the additional object 921 displayed on the object 911 with the image data. The communication module 190 of the electronic device 101 can send the image data combined with the additional object data to the external device 301 through wireless communication. The external device 301 can display an augmented reality image indicating the additional object 921 to the user on the object 911.

[0109] For example, the external device 301 may view the shoes 911. When the user inputs the shoes into the input device 150 of the electronic device 101, or when the information associated with the shoes is stored in the memory 130 of the electronic device 101, the processor 120 of the electronic device 101 may identify the shoes 911 as an object meaningful to the user. The processor 120 may determine that the user wants the shoes 911, and may combine the emoticon data with the image data to generate a facial expression 921 that can express the emotion that the user finds the shoes very good. The external device 301 may use the image data combined with the emoticon data to display an augmented reality image indicating the facial expression 921 in a form that at least partially overlaps with the shoes 911.

[0110] In an embodiment, the external device 301 may provide a voice guide 941 associated with the additional object 921. For example, when the object 911 is a pair of shoes that the user needs, the external device 301 may output a voice “I'm yours!” 941 using a voice utterance application.

[0111] In an embodiment, Fig. 9B As shown, the electronic device 101 can view the object 912 using the camera module 180. The electronic device 101 can determine whether the viewed object 912 is meaningful to the user. The processor 120 of the electronic device 101 can determine whether the object 912 is meaningful to the user based on the input information input by the user or the storage information stored in the memory 130. When it is determined that the object 912 is meaningful to the user, the processor 120 can combine the additional object data of displaying the additional object 922 on the object 912 with the image data. The electronic device 101 can display an image indicating the additional object 922 to the user on the object 912.

[0112] For example, the electronic device 101 may view the shoes 912. When the user inputs the shoes into the input device 150 of the electronic device 101, or when the information associated with the shoes is stored in the memory 130 of the electronic device 101, the processor 120 of the electronic device 101 may identify the shoes 912 as an object meaningful to the user. The processor 120 may determine that the user wants the shoes 912, and may combine the emoticon data with the image data to generate a facial expression 922 that can represent the emotion that the user finds the shoes very good. The electronic device 101 may use the image data combined with the emoticon data to display an image indicating the facial expression 922 on the display 210 in a form that at least partially overlaps with the shoes 912.

[0113] In an embodiment, the electronic device 101 may provide a voice guide 942 associated with the additional object 922. For example, when the object 912 is a pair of shoes that the user needs, the electronic device 101 may output a voice “I'm yours!” 942 using a voice utterance application.

[0114] Fig.10 10 is a simplified diagram of a process of viewing the reality 1001 using the external device 301 and generating additional objects 1031 , 1032 , and 1033 in response to the shape of the object 1021 in the electronic device 101 according to an embodiment.

[0115] In an embodiment, the external device 301 may display a reality 1001 included in a field of view (FOV) 1011 visible to the external device 301. The external device 301 may identify an object 1021 on the reality 1001 included in the visible FOV 1011. The external device 301 may determine whether the object 1021 is meaningful to the user based on input information sent from the electronic device 101 or stored information. When the object 1021 is meaningful to the user, the external device 301 may send information associated with the shape of the object 1021 to the electronic device 101. For example, when the user inputs information associated with a cup to the electronic device 101 or stores the information in the memory 130, the external device 301 may identify the cup as an object 1021 meaningful to the user.

[0116] In an embodiment, the processor 120 of the electronic device 101 may determine whether the additional image 1031 can be displayed on the object 1021 based on the shape of the object 1021. For example, the processor 120 may determine whether the additional object 1031 can be overlapped on the object 1021 according to the size of the object 1021, the color of the object 1021, or whether there is a symbol (such as a symbol character or pattern) on the surface of the object 1021.

[0117] In an embodiment, the processor 120 of the electronic device 101 may be configured to: when it is determined that the additional object 1031 can be superimposed on the object 1021, display the facial expression 1031 based on the input information or stored information associated with the object 1021. The processor 120 may send the image data combined with the emoticon data to the external device 301, so that the external device 301 superimposes and displays the additional object 1031 on the object 1021. For example, when the object 1021 is a cup, the processor 120 may determine that the user finds the cup in the early morning based on information about the time when the user uses the cup. The processor 120 may be configured to generate a facial expression indicating that the cup as the additional object 1031 greets the user good morning.

[0118] In an embodiment, the processor 120 may be configured to enable the electronic device 101 or the external device 301 to use a voice-generating application (eg, Bixby TM ) outputs a voice corresponding to the additional object 1031. For example, when a facial expression 1031 greeting good morning is generated, the electronic device 101 or the external device 301 may output a voice "Good morning sir" 1041 using a voice utterance application.

[0119] In an embodiment, the processor 120 may be configured to change the shape of the additional object 1032 in response to the shape of the object 1021. The processor 120 may be configured to cause the additional object 1032 to protrude from the edges of both sides of the object 1021 when the width of the object 1021 satisfies the specified range condition. The external device 301 may display the additional object 1032 whose shape is changed. For example, when the object 1021 is a cup, the processor 120 may determine whether the width of the cup satisfies the specified range condition. When the width of the cup satisfies the specified range condition, the external device 301 may display the additional object 1032 in the form of arms attached to both side edges of the cup under the control of the processor 120.

[0120] In an embodiment, the processor 120 may change the shape of the additional object 1033 in response to whether the edge of the object 1021 is in contact with another thing. The external device 301 may display the additional object 1033 whose shape is changed. For example, the processor 120 may be configured to humanize the cup so that the cup sits on the table when the object 1021 is a cup on the table. The external device 301 may display the additional object 1033 in the form of the lower part of the cup sitting with legs.

[0121] FIG. 11A to FIG. 11Cis a diagram illustrating displaying additional objects 1121 , 1122 , and 1123 on objects 1111 , 1112 , and 1113 of augmented reality images 1101 , 1102 , and 1103 viewed through an external device 301 based on input information in the electronic device 101 according to an embodiment.

[0122] In an embodiment, the electronic device 101 may receive information directly input by the user, or information about the surrounding conditions of the electronic device 101, with the aid of the sensor module 176 to have input information. The electronic device 101 may receive augmented reality images 1101, 1102, and 1103 from the external device 301 using the communication module 190. The processor 120 of the electronic device 101 may recognize the meaning between the user and the objects 1111, 1112, and 1113 of the augmented reality images 1101, 1102, and 1103 based on the input information. The processor 120 may be configured to display additional objects 1121, 1122, and 1123 to correspond to the meaning between the user and the objects 1111, 1112, and 1113.

[0123] For example, Fig.11A As shown, bread whose calories are greater than or equal to a specified range can be viewed on an augmented reality image 1101 viewed through an external device 301. When a user inputs calorie-related information to the electronic device 101, or when calorie-related information is set to have a high priority, the electronic device 101 can identify the bread as an object 1111. The processor 120 of the electronic device 101 can be configured to display a negative facial shape as an additional object 1121 to indicate that the calories of this break are too high. In an embodiment, the external device 301 can display an additional object 1121 on the object 1111 of the augmented reality image 1101.

[0124] For another example, Fig. 11B As shown, a water bottle filled with water can be viewed on an augmented reality image 1102 viewed through an external device 301. When a user inputs water intake related information to the electronic device 101, or when water intake related information is set to have a high priority, the electronic device 101 can identify the water bottle filled with water as an object 1112. The processor 120 of the electronic device 101 can be configured to display a frontal facial shape as an additional object 1122 to indicate that drinking water is beneficial. The external device 301 can display the additional object 1122 on the object 1112 of the augmented reality image 1102.

[0125] For another example, Fig. 11CAs shown, fruit can be viewed on the augmented reality image 1103 viewed through the external device 301. When the user inputs vitamin-related information to the electronic device 101, or when vitamin-related information is set to have a high priority, the electronic device 101 can identify this fruit as an object 1113. The processor 120 of the electronic device 101 can be configured to display a frontal face shape as an additional object 1123 to indicate that this fruit is rich in vitamins. The external device 301 can display the additional object 1123 on the object 1113 of the augmented reality image 1103.

[0126] In an embodiment, the processor 120 of the electronic device 101 may be configured to output voices 1141, 1142, and 1143 corresponding to the contents of the displayed additional objects 1121, 1122, and 1123 via the electronic device 101 or the external device 301. Fig.11A In the illustrated case, the processor 120 may use a voice-generating application (e.g., Bixby TM ) outputs a voice message "Not recommended. Too much carbohydrate." 1141 to indicate that bread has too high calories. For another example, in Fig. 11B In the case shown in FIG. 1 , the processor 120 may use the voice application to output a voice message “Pick me to drink more! More water means more hydrated skin!” 1142 to indicate that drinking water is beneficial. For another example, in FIG. Fig. 11C In the illustrated case, the processor 120 may use the voice application to output a voice message "Pick me! I'm full of vitamin C!" 1142 to indicate that the fruit is rich in vitamins.

[0127] Fig.12 is a diagram illustrating displaying an additional object 1221 on an object 1210 of an augmented reality image viewed through an external device 301 based on stored information in the electronic device 101 according to an embodiment.

[0128] In an embodiment, the memory 130 of the electronic device 101 may store stored information 1220 including usage pattern data in which the usage pattern of the user is accumulated. The processor 120 of the electronic device 101 may determine whether the object 1210 of the augmented reality image viewed through the external device 301 is an object 1210 meaningful to the user based on the stored information 1220 stored in the memory 130. For example, when the object 1210 of the augmented reality image viewed through the external device 301 is an item, the processor 120 may determine whether information associated with the item previously purchased by the user is stored in the memory 130. When the information associated with the previously purchased item is stored in the memory 130, the processor 120 may set the information associated with the previously purchased item as the stored information 1220, and may determine that the object 1210 of the augmented reality image is an object 1210 meaningful to the user.

[0129] In an embodiment, the processor 120 of the electronic device 101 may configure the external device 301 to display the additional object 1221 on the object 1210 based on the stored information. The external device 301 may provide the user with an augmented reality image in which the additional object 1221 is displayed on the object 1210. For example, when the stored information 1220 is information associated with a previously purchased item, the processor 120 may configure the external device 301 to generate a facial expression as the additional object 1221, thereby providing a notification that the object 1210 is similar to the previously purchased item.

[0130] In an embodiment, the processor 120 of the electronic device 101 may run an application associated with the object 1210 based on the object 1210 and the stored information 1220. The external device 301 may display an execution screen 1230 of the application associated with the object 1210 on the augmented reality image. For example, when the object 1210 is an item and when the stored information 1220 is information associated with a previously purchased item, the processor 120 may run a payment-related application (e.g., Samsung Pay). TM ) to pay for the item. The external device 301 may display the price of the object 1210 and the target expenditure set by the application on the execution screen 1230.

[0131] In an embodiment, the processor 120 of the electronic device 101 may be configured to output a voice 1240 corresponding to the meaning that the additional object 1221 intends to express through the electronic device 101 or the external device 301. For example, the processor 120 may use a voice application (such as Bixby TM) outputs the voice “You might have purchased a similar item.”, indicating that object 1210 is similar to the previously purchased item.

[0132] The electronic device 101 according to various embodiments may include a communication module 190, a display 210, and at least one processor 120 operatively connected to the communication module 190 and the display 210. The at least one processor 120 may be configured to receive an augmented reality image from at least one external device 301 performing wireless communication with the communication module 190 via the communication module 190, display an execution screen of an application associated with the augmented reality image on the display 210, determine whether an object 511 associated with input information input to the electronic device 101 is included in the augmented reality image, and display an additional object 521 on the object 511 based at least in part on the input information.

[0133] In an embodiment, the at least one processor 120 may be configured to generate input information based at least in part on information directly set by a user or information according to a situation outside the electronic device 101 .

[0134] In an embodiment, the at least one processor 120 may be configured to set the priority of the object 511 based at least in part on information directly set by the user or a situation outside the electronic device 101 , and identify the object 511 whose priority is high.

[0135] In an embodiment, the at least one processor 120 may be configured to determine whether the object 511 is meaningful to the user based at least in part on information directly set by the user or a situation outside the electronic device 101 , and identify a meaningful object.

[0136] In an embodiment, the at least one processor 120 may be configured to generate an additional object 521 at least partially overlapping the object 511 and allow the additional object 521 to have a facial shape representing an emotion.

[0137] In an embodiment, the at least one processor 120 may be configured to allow the additional object 521 to express an emotion as any one of a plurality of emotion stages including between a positive emotion and a negative emotion based at least in part on the input information.

[0138] In an embodiment, the at least one processor 120 may be configured to change the emotion expressed by the additional object 521 or the shape of the additional object 521 according to the type of the object 511 or the change of the state of the object 511 .

[0139] In an embodiment, the at least one processor 120 may be configured to generate a shape of the additional object 521 based at least in part on a shape of the object 511 or a length ratio of the object 511 .

[0140] The electronic device 101 according to various embodiments may include a communication module 190, a display 210, a memory 130, and at least one processor 120 operatively connected to the communication module 190, the display 210, and the memory 130. The at least one processor 120 may be configured to receive an augmented reality image from at least one external device 301 performing wireless communication with the communication module 190 via the communication module 190, display an execution screen of an application associated with the augmented reality image on the display 210, determine whether an object 511 associated with stored information stored in the memory 130 is included in the augmented reality image, and display an additional object 521 on the object 511 based at least in part on the stored information.

[0141] In an embodiment, at least one processor 120 may be configured to generate storage information based at least in part on usage pattern data in which the usage patterns of users are accumulated.

[0142] In an embodiment, at least one processor 120 may be configured to set priorities of objects 511 based at least in part on usage pattern data in which usage patterns of users are accumulated, and identify objects 511 whose priorities are high.

[0143] In an embodiment, at least one processor 120 may be configured to determine whether the object 511 is meaningful to the user based at least in part on usage pattern data that accumulates the user's usage pattern, and identify the meaningful object 511 .

[0144] According to various embodiments, the driving method of the electronic device 101 may include: receiving an augmented reality image from at least one external device 301 that performs wireless communication with the electronic device 101; displaying a running screen of an application associated with the augmented reality image on the display 210; determining whether an object 511 associated with at least one of first information input by a user and second information stored in a memory is included in the augmented reality image; and performing at least one of the following items on the object 511: generating a first additional object based on the first information, and generating a second additional object based on the second information.

[0145] In an embodiment, the driving method may further include generating first information based at least in part on information directly set by the user or information according to a situation outside the electronic device 101, or generating second information based at least in part on usage pattern data accumulating the user's usage pattern.

[0146] In an embodiment, the driving method may further include setting priorities of the objects 511 based at least in part on information directly set by a user or a situation outside the electronic device 101 , and identifying the objects 511 whose priorities are high.

[0147] In an embodiment, the driving method may further include setting priorities of the objects 511 based at least in part on usage pattern data accumulating usage patterns of users, and identifying the objects 511 whose priorities are high.

[0148] In an embodiment, the driving method may further include determining whether the object 511 is meaningful to the user based at least in part on information directly set by the user, a situation outside the electronic device 101 , or usage pattern data that accumulates the user's usage pattern, and identifying the meaningful object 511 .

[0149] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to one embodiment of the present disclosure, the electronic device is not limited to the above-mentioned electronic device.

[0150] It should be understood that the various embodiments of the present disclosure and the terms used herein are not intended to limit the technical features set forth herein to specific embodiments, but include various modifications, equivalents or alternative forms of the corresponding embodiments. With respect to the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It should be understood that, unless otherwise clearly indicated by the relevant context, the singular noun corresponding to a certain element may include one or more things. As used herein, phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C" may include any one or all possible combinations of elements listed together in one of the corresponding phrases in these phrases. As used herein, terms such as "first" and "second", or "first" and "second" may be used to simply distinguish a corresponding component from another component, and do not limit the component in other aspects (e.g., importance or order). It should be understood that if an element (e.g., a first element) is referred to as being “coupled,” “connected,” “coupled to,” or “connected to” another element (e.g., a second element) with or without the term “operably” or “communicatively,” it means that the element can be coupled to the other element directly (e.g., wired), wirelessly, or via a third element.

[0151] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms such as "logic", "logic block", "component", or "circuit". A module may be a single integral component or its smallest unit or part suitable for performing one or more functions. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0152] The various embodiments described herein may be implemented as software (e.g., program 140) including one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). For example, a processor (e.g., processor 120) of a machine (e.g., electronic device 101) may call at least one of the one or more instructions stored in the storage medium and execute the instruction with or without one or more other components under the control of the processor. In this way, the machine can be operated to perform at least one function according to at least one instruction called. One or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" merely means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between situations where data is semi-permanently stored in a storage medium and situations where data is temporarily stored in a storage medium.

[0153] According to an embodiment, the method according to various embodiments of the present disclosure may be included in a computer program product and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed via an application store (e.g., PlayStore). TM ) online distribution (e.g., download or upload), or directly between two user devices (e.g., smartphones). If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or a relay server.

[0154] According to various embodiments, each component (e.g., module or program) of the above-mentioned components may include a single entity or multiple entities. According to various embodiments, one or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform one or more functions in the same or similar manner as one or more functions of each component in the multiple components before integration in a corresponding one of the multiple components. According to various embodiments, the operations performed by a module, program or another component may be performed sequentially, in parallel, repeatedly or tentatively, or one or more operations in the operation may be performed or omitted in a different order, or one or more other operations may be added.

Claims

1. An electronic device, comprising: Communication module; at least one processor operatively connected to the communication module and the display, Wherein, the at least one processor is configured to: identifying at least one object included in a first image obtained by at least one external device, the external device establishing a communication channel with the electronic device via the communication module; If the identified at least one object is associated with input information input to the electronic device, generating a first additional object on the identified at least one object based at least in part on the input information; sending first information associated with the first additional object to the at least one external device so that the first additional object is displayed by the at least one external device; determining whether a change of the identified at least one object occurs based on second image data obtained by the at least one external device when the first additional object is displayed through the at least one external device; In response to determining that a change of the identified at least one object has occurred, generating a second additional object based at least in part on the input information and the extent of the change of the identified at least one object; and transmitting second information associated with the second additional object to the at least one external device so that the first additional object displayed by the at least one external device is changed based on the second additional object, Wherein, the first additional object and the second additional object at least partially overlap with the identified at least one object.

2. The electronic device according to claim 1, in, The input information includes at least one of information based at least in part on a direct setting by a user or information according to a situation outside the electronic device.

3. The electronic device according to claim 1, further comprising a display, in, The at least one processor is configured to: When at least one of the first additional object and the second additional object is displayed through the at least one external device, a guide associated with at least one of the first additional object and the second additional object is provided through the display.

4. The electronic device according to claim 1, wherein: The at least one processor is configured to: If the at least one identified object is out of the second image data, generating a third additional object for identifying whether the at least one identified object is missed; and Third information associated with the third additional object is transmitted to the at least one external device so that the third additional object is displayed through the at least one external device.

5. The electronic device according to claim 1, wherein: The at least one processor is configured to: The first additional object and the second additional object are allowed to have facial shapes representing emotions.

6. The electronic device according to claim 5, wherein: The at least one processor is configured to: The first additional object and the second additional object are allowed to express the emotion as any one of a plurality of emotion stages included between a positive emotion and a negative emotion.

7. The electronic device according to claim 1, wherein: The at least one processor is configured to: The shapes of the first additional object and the second additional object are generated based at least in part on a shape of the identified at least one object or a ratio of lengths of the identified at least one object.

8. An electronic device comprising: Communication module; Memory; and at least one processor operatively connected to the communication module, the display and the memory, Wherein, the at least one processor is configured to: identifying at least one object included in a first image obtained by at least one external device, the external device establishing a communication channel with the electronic device via the communication module; If the identified at least one object is associated with stored information stored in the memory, generating a first additional object on the identified at least one object based at least in part on the stored information; sending first information associated with the first additional object to the at least one external device so that the first additional object is displayed by the at least one external device; determining whether a change of the identified at least one object occurs based on second image data obtained by the at least one external device when the first additional object is displayed through the at least one external device; In response to determining that a change to the identified at least one object has occurred, generating a second additional object based at least in part on the stored information and an extent of the change to the identified at least one object; and transmitting second information associated with the second additional object to the at least one external device so that the first additional object displayed by the at least one external device is changed based on the second additional object, Wherein, the first additional object and the second additional object at least partially overlap with the identified at least one object.

9. The electronic device according to claim 8, wherein: The at least one processor is configured to: The stored information is generated based at least in part on usage pattern data in which usage patterns of users are accumulated.

10. The electronic device according to claim 8, wherein: The at least one processor is configured to: When at least one of the first additional object and the second additional object is displayed through the at least one external device, a guide associated with at least one of the first additional object and the second additional object is provided through the display.

11. The electronic device according to claim 8, wherein: The at least one processor is configured to: If the at least one identified object is out of the second image data, generating a third additional object for identifying whether the at least one identified object is missed; and Third information associated with the third additional object is transmitted to the at least one external device so that the third additional object is displayed through the at least one external device.

12. The electronic device according to claim 8, wherein: The at least one processor is configured to: The first additional object and the second additional object are allowed to have facial shapes representing emotions.

13. The electronic device according to claim 12, wherein: The at least one processor is configured to: The first additional object and the second additional object are allowed to express the emotion as any one of a plurality of emotion stages included between a positive emotion and a negative emotion.

Images