Electronic device and control method therefor

Abstract

An electronic device is disclosed. The electronic device comprises: a communication interface; and a display including a plurality of haptic elements and, if information corresponding to a preset motion event occurred in a first image output from an external display device is received through the communication interface on the basis of the motion event, identifies at least one area corresponding to the motion event among a plurality of areas included in the display and controls the display such that a second image corresponding to the received motion event is displayed on the at least one area and that a haptic element corresponding to the at least one area among the plurality of haptic elements provides haptic feedback.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more particularly, to a method for providing an image and haptic feedback related to an event to a user by considering an event occurring within an image output from an external device.

[0002] Advances in electronic technology have led to the development and proliferation of various types of electronic devices. To help users effectively enjoy content, electronic devices (e.g., haptic devices) exist to assist users. Haptic devices are devices that provide tactile information to users by contacting their skin or using body parts while using electronic devices such as computers or TVs. Therefore, there is a pressing need for technological development in electronic devices that provide users with information about content or easily transmit feedback.

[0003] An electronic device according to one embodiment of the present disclosure includes a communication interface, a display having a plurality of haptic elements, and an external display device. When information corresponding to the motion event is received through the communication interface based on a preset motion event occurring within a first image output from the display, the electronic device can identify at least one area corresponding to the motion event among a plurality of areas included in the display.

[0004] The one or more processors can control the display to display a second image corresponding to the received motion event in the at least one area and to cause a haptic element corresponding to the at least one area among the plurality of haptic elements to provide haptic feedback.

[0005] A method for controlling an electronic device according to an embodiment of the present disclosure may include, when information corresponding to a preset motion event occurring within a first image output from an external display device is received, a step of identifying at least one area corresponding to the motion event among a plurality of areas included in a display having a plurality of haptic elements.

[0006] The above control method may include a step of displaying a second image corresponding to the received motion event in the at least one area.

[0007] The above control method may include a step of providing haptic feedback by a haptic element corresponding to at least one area among the plurality of haptic elements.

[0008] In one embodiment of the present disclosure, a non-transitory computer-readable recording medium storing computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, the operation may include a step of identifying at least one area corresponding to the motion event among a plurality of areas included in a display having a plurality of haptic elements, when information corresponding to the motion event is received based on a preset motion event occurring in a first image output from an external display device.

[0009] The above operation may include a step of displaying a second image corresponding to the received motion event in the at least one area.

[0010] The above operation may include a step of providing haptic feedback by a haptic element corresponding to at least one area among the plurality of haptic elements.

[0011] FIG. 1 is a drawing schematically illustrating a control method of an electronic device according to one embodiment.

[0012] Figure 2 is a block diagram showing the configuration of an electronic device according to one embodiment.

[0013] Figure 3 is a flowchart for explaining a control method of an electronic device according to one embodiment.

[0014] FIG. 4 is a drawing for explaining a method of displaying a second image according to one embodiment.

[0015] FIG. 5 is a drawing for explaining a method for displaying an image with changed colors according to one embodiment.

[0016] FIG. 6 is a diagram illustrating a method for providing haptic feedback according to one embodiment.

[0017] FIG. 7 is a drawing for explaining a method of dividing a display into multiple areas according to one embodiment.

[0018] FIG. 8 is a drawing for explaining a method of displaying a first UI according to one embodiment.

[0019] FIG. 9 is a drawing for explaining a method of displaying a second UI corresponding to a remote control according to one embodiment.

[0020] FIG. 10 is a flowchart illustrating a method for displaying a second UI corresponding to a remote control according to one embodiment.

[0021] FIGS. 11A and 11B are drawings for explaining an image display method and a haptic feedback providing method according to one embodiment.

[0022] FIG. 12a and FIG. 12b are drawings for explaining an image display method and a haptic feedback providing method according to one embodiment.

[0023] Figure 13 is a block diagram showing a detailed configuration of an electronic device according to one embodiment.

[0024] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

[0025] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail.

[0026] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0027] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0028] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0029] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0030] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0031] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0032] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor (not shown), excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0033] FIG. 1 is a drawing schematically illustrating a control method of an electronic device according to one embodiment.

[0034] According to FIG. 1, according to one embodiment, when a motion event occurs within an image (1) output from an external display device, the electronic device (10) may provide an image (1) and haptic feedback corresponding to the generated motion event. According to one example, when a movement event of a car occurs within an image (1) output from an external display device, the electronic device (10) may display an image including an object (12) corresponding to the movement event of the car through the display (11).

[0035] In one example, the electronic device (10) may include a plurality of haptic elements for providing haptic feedback. In one example, the electronic device (10) may include at least one haptic element corresponding to each of a plurality of areas of the display (11). The electronic device (10) may provide haptic feedback through a haptic element corresponding to an area of ​​the display (11) corresponding to a location where an object (12) is displayed.

[0036] Hereinafter, various embodiments will be described in which an electronic device (10) provides an image corresponding to a motion event to a user when a motion event occurs within an external display device, and at the same time provides haptic feedback that enables the user to effectively recognize the motion event, thereby enabling the user to easily recognize the motion event.

[0037] Figure 2 is a block diagram showing the configuration of an electronic device according to one embodiment.

[0038] According to FIG. 2, the electronic device (100) may include a communication interface (110), a display (120), and one or more processors (130).

[0039] The electronic device (100) may be implemented as a haptic device for providing haptic feedback, according to one embodiment. A haptic device refers to a device that transmits tactile information to a user by contacting the user's skin or utilizing a body part while the user is using an electronic device, such as a computer or TV. According to one example, the electronic device (100) may be a mouse-shaped device, but is not limited thereto.

[0040] The external display device may be implemented as a variety of devices equipped with a display, such as a smart TV, tablet, monitor, smartphone, desktop computer, laptop computer, etc., according to one embodiment. The external display device according to one embodiment of the present disclosure is not limited to the devices described above, and the external display device may be implemented as an external display device having the functions of two or more of the devices described above.

[0041] Meanwhile, the electronic device (100) can communicate with an external display device in various ways. According to one embodiment, a communication module for communication with the external device may be implemented in the same manner. For example, the electronic device (100) can communicate with the external device using a Bluetooth module.

[0042] The communication interface (110) can input and output various types of data. For example, the communication interface (110) can transmit and receive various types of data to and from an external device (e.g., a source device), an external storage medium (e.g., a USB memory), an external server (e.g., a web hard drive) through a communication method such as AP-based Wi-Fi (Wireless LAN network), Bluetooth, Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), optical, coaxial, etc.

[0043] For example, the communication interface (110) may include a BLE (Bluetooth Low Energy) module. BLE refers to Bluetooth technology that enables transmission and reception of low-power, low-capacity data in the 2.4 GHz frequency band with a range of about 10 m. However, the present invention is not limited thereto, and the communication interface (110) may also include a Wi-Fi communication module. That is, the communication interface (110) may include at least one of a BLE (Bluetooth Low Energy) module or a Wi-Fi communication module.

[0044] The display (120) may be implemented as a display including a self-luminous element or a display including a non-luminous element and a backlight. For example, it may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc. The display (120) may also include a driving circuit, a backlight unit, etc., which may be implemented in a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. Meanwhile, the display (120) may be implemented as a touch screen combined with a touch sensor, a flexible display, a rollable display, a 3D display, a display in which a plurality of display modules are physically connected, etc. The processor (130) can control the display (120) to output an output image obtained according to the various embodiments described above. Here, the output image may be a high-resolution image of 4K or 8K or higher. The output image may also be a game image according to one embodiment.

[0045] According to one embodiment, the display (120) may include a plurality of haptic elements. The haptic elements may be implemented as motors for providing haptic feedback (e.g., vibration feedback) to a user, but are not limited thereto. According to one example, the display (120) may include a preset number of haptic elements. For example, the display (120) may include a preset number of haptic elements corresponding to a preset number of sub-regions of the display, but is not limited thereto, and it is to be understood that the display may include a number of haptic elements different from the number of sub-regions corresponding to the display.

[0046] One or more processors (130) (hereinafter, “processors”) are electrically connected to the communication interface (110) and the display (120) to control the overall operation of the electronic device (100). The processor (130) may be composed of one or more processors. Specifically, the processor (130) may perform operations of the electronic device (100) according to various embodiments of the present disclosure by executing at least one instruction stored in a memory.

[0047] According to one embodiment, the processor (130) may be implemented as a digital signal processor (DSP), a microprocessor, a graphics processing unit (GPU), an artificial intelligence (AI) processor, a neural processing unit (NPU), or a time controller (TCON) for processing a digital image signal. However, the present invention is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), or an ARM processor, or may be defined by the relevant terminology. In addition, the processor (130) may be implemented as a system on chip (SoC) or large scale integration (LSI) having a built-in processing algorithm, or may be implemented in the form of an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA).

[0048] According to one embodiment, the display (120) may be implemented as an Electrophoretic Display (EPD) film, and the electronic device (100) may include a backplane circuit including a plurality of elements for dividing the display (120) into a plurality of areas.

[0049] According to one embodiment, when a motion event occurs within an image output from an external display device, the processor (130) may receive information corresponding to the occurred motion event.

[0050] For example, a motion event may be an event in which a dynamic object moves within an image, or an event in which a pixel changes rapidly (e.g., an explosion event), but is not limited thereto. For example, information corresponding to a motion event may include, but is not limited to, at least one of pixel information or position information of an image corresponding to the motion event among images output from an external display device. For example, the processor (130) may receive information corresponding to a motion event through the communication interface (110) based on a preset motion event that occurs within a first image output from the external display device. For example, the first image may be a dynamic image of a different type including a game image.

[0051] According to one embodiment, the processor (130) can identify at least one area within the display (120) based on information corresponding to a motion event.

[0052] According to one example, when information corresponding to a motion event is received through the communication interface (110) based on a preset motion event occurring within a first image output from an external display device, the processor (130) may identify at least one area corresponding to the motion event among a plurality of areas included in the display (120). According to one example, the display (120) may be divided into a preset number of areas, and location information (or coordinate information) corresponding to each of the plurality of areas may be stored in a memory.

[0053] According to one example, the processor (130) may identify at least one area of ​​the display (120) corresponding to an area where a motion event occurs within an external display device, based on location information corresponding to the received motion event. For example, when an event in which a dynamic object is identified at a first location within a first image occurs, the processor (130) may identify at least one area of ​​the display (120) corresponding to the first location within the first image, based on location information included in the information corresponding to the received motion event.

[0054] In one embodiment, the processor (130) may display an image corresponding to a motion event and provide haptic feedback.

[0055] In one example, the processor (130) may control the display (120) to display a second image corresponding to a motion event received in at least one area and to cause a haptic element corresponding to at least one area among the plurality of haptic elements to provide haptic feedback. The haptic feedback may be, for example, vibration feedback output through the haptic element, but is not limited thereto. In one example, the second image may be an image of a dynamic object that occurred in the first image. For example, if an event in which a car moves in the first image occurs, the second image may be an image of the car present in the first image. However, the present invention is not limited thereto.

[0056] According to the example described above, when a motion event occurs within an external display device, the electronic device (100) can provide the user with an image corresponding to the motion event and simultaneously provide haptic feedback that enables the user to effectively recognize the motion event. Accordingly, the user can easily recognize the motion event, thereby enhancing user satisfaction.

[0057] Figure 3 is a flowchart for explaining a control method of an electronic device according to one embodiment.

[0058] According to FIG. 3, according to one embodiment, a control method can identify whether information corresponding to a motion event is received based on a preset motion event occurring within a first image output from an external display device (S310). According to one example, assume that a motion event in which a car moves within the first image occurs. The processor (130) can receive information including location information of the car moving within the first image and image information of the car from the external display device via the communication interface (110).

[0059] Next, according to one embodiment, when information corresponding to a motion event is received (S310: Y), the control method may identify at least one area corresponding to the motion event among a plurality of areas included in a display (120) having a plurality of haptic elements (S320). According to one example, when information including location information of a vehicle moving within a first image and image information of the vehicle is received, the processor (130) may identify at least one area within the display (120) corresponding to a first location of the vehicle moving within the first image among the plurality of areas within the display (120) based on the received information.

[0060] Next, according to one embodiment, the control method may display a second image corresponding to the motion event received in at least one area (S330). According to one example, when at least one area is identified within the display (120), the processor (130) may control the display (120) to display an image of a car in the identified at least one area. When the car moves in real time, the processor (130) may control the display (120) to display a dynamic image of the car moving in real time.

[0061] Next, according to one embodiment, the control method may provide haptic feedback by a haptic element corresponding to at least one area among a plurality of haptic elements (S340). According to one example, the processor (130) may identify at least one haptic element corresponding to at least one identified area among the plurality of haptic elements, and control the identified at least one haptic element to provide haptic feedback (e.g., vibration feedback).

[0062] FIG. 4 is a drawing for explaining a method of displaying a second image according to one embodiment.

[0063] According to one embodiment, the processor (130) may display a second image based on location information and image information of a dynamic object within the first image. According to one example, a preset motion event may be an event in which a preset dynamic object is identified within the first image. The preset dynamic object may be, for example, a car or a person moving within the first image, but is not limited thereto, and may of course be a different type of object whose movement is identified within the first image. According to one example, information corresponding to the motion event may include location information of the dynamic object and image information of the dynamic object. The location information of the dynamic object may include real-time movement information of the dynamic object.

[0064] According to FIG. 4, a first image (40) may be displayed on the display of an external display device according to an example. The external display device may identify a motion event corresponding to the movement of a vehicle, which is a dynamic object, within the first image (40). When a motion event corresponding to the movement of the vehicle is identified, the external display device may transmit information corresponding to the identified motion event (information including vehicle location information and vehicle image information) to an electronic device (400).

[0065] According to one example, the processor (130) may receive information corresponding to the identified motion event from an external display device through the communication interface (110). The processor (130) may identify at least one area of ​​the display (410) corresponding to the location of the vehicle within the first image (40) based on the location information of the vehicle included in the information corresponding to the identified motion event. For example, when a motion event corresponding to movement of the vehicle is identified in the seventh area and the eighth area within the first image (40), the processor (130) may identify an area of ​​the display (410) corresponding to the seventh area and the eighth area within the first image (40) among the plurality of areas of the display (410).

[0066] According to an example, the processor (130) may display a second image (411) in at least one area of ​​the display (410) based on the received image information of the dynamic object. For example, the processor (130) may display an image of a car (411) in an area of ​​the display (410) corresponding to the seventh area and the eighth area in the first image (40) based on the received image information of the car.

[0067] FIG. 5 is a drawing for explaining a method for displaying an image with changed colors according to one embodiment.

[0068] According to FIG. 5, according to one embodiment, the processor (130) may identify a representative color corresponding to an object. According to one example, the representative color corresponding to an object may be a color value having the maximum proportion among the color values ​​corresponding to the object. According to one example, the processor (130) may identify a color having the maximum proportion among the plurality of pixels based on color information corresponding to each of the plurality of pixels included in the image information of the received object. For example, when a second image (511) is identified based on information corresponding to a received motion event, the processor (130) may identify the number of pixels having the first color as the maximum based on color information corresponding to each of the plurality of pixels included in the identified second image (511). The processor (130) may identify the first color having the maximum number of pixels as the representative color corresponding to the object.

[0069] In one embodiment, the processor (130) may display a second image including a dynamic object of the identified representative color. In one example, when the representative color is identified, the processor (130) may identify an image (512) of the dynamic object updated with the identified representative color and display it through the display (520).

[0070] According to one embodiment, the processor (130) may display an updated second image based on a user input for displaying a second image including a dynamic object of a representative color. According to one example, as shown in the left drawing of FIG. 5, assume a case where an image (511) of a dynamic object corresponding to a motion event is displayed through a display (510) included in an electronic device (501). When a user input for displaying the image (511) of the dynamic object in a representative color is received, the processor (130) may identify the representative color of the image (511) of the dynamic object based on image information of the image (511) of the dynamic object. When the representative color of the image (511) of the dynamic object is identified, the processor (130) may display an image (512) of the dynamic object of the representative color through a display (520) included in the electronic device (502), as shown in the right drawing of FIG. 5.

[0071] FIG. 6 is a diagram illustrating a method for providing haptic feedback according to one embodiment.

[0072] According to FIG. 6, according to one embodiment, the electronic device (600) may include a plurality of haptic elements provided at positions corresponding to each of a plurality of areas within the display (601). According to one example, when there are 15 areas within the display (601), at least one haptic element may be positioned at a position corresponding to each of the 15 areas. However, the present invention is not limited thereto.

[0073] According to one embodiment, the processor (130) may control at least one haptic element based on information corresponding to a motion event. According to one example, the information corresponding to the motion event may further include control information for at least one haptic element corresponding to at least one area among a plurality of haptic elements. The processor (130) may control at least one haptic element corresponding to at least one area based on the control information for the haptic element included in the information corresponding to the motion event.

[0074] For example, assume that a dynamic object is identified at a location corresponding to a seventh region and an eighth region in a first image. The processor (130) may display an image of the dynamic object in a seventh region and an eighth region in a display (601) corresponding to the seventh region and the eighth region in the first image based on information corresponding to the motion event. When control information for at least one haptic element corresponding to the seventh region and the eighth region included in the information corresponding to the motion event is received, the processor (130) may control at least one haptic element corresponding to the seventh region and the eighth region based on the control information to provide haptic feedback. Accordingly, the electronic device (600) may provide an image of the dynamic object to the user, and at the same time, provide haptic feedback in an area where the image of the dynamic object is displayed.

[0075] According to one embodiment, the processor (130) may control the haptic intensity of at least one haptic element corresponding to at least one area based on haptic intensity information of the haptic element. According to one example, the control information for the haptic element may further include haptic intensity information of the haptic element. According to one example, the haptic intensity information of the haptic element may be identified based on at least one of audio information corresponding to a motion event, information on a range of occurrence of the motion event within a first image, and information on a duration of the motion event. The processor (130) may receive the haptic intensity information of the haptic element from an external display device through the communication interface (110), including audio information corresponding to a motion event, information on a range of occurrence of the motion event within the first image, and information on a duration of the motion event. Alternatively, the processor (130) may also identify information on a duration of the motion event based on a displayed second image. For example, the processor (130) can identify the time from the start of displaying the second image on the display (601) to the end of displaying the second image as the duration of the motion event.

[0076] In one example, there may be a plurality of preset intensity levels (e.g., 10 intensity levels). For example, the haptic intensity information of the haptic element may identify a relatively high level of haptic intensity as the audio size corresponding to the motion event is louder, the occurrence range of the motion event in the first image is relatively wider, and the duration of the motion event is longer. In one example, the memory may pre-store weight values ​​corresponding to audio information corresponding to the motion event, information on the occurrence range of the motion event in the first image, and information on the duration of the motion event, respectively. The processor (130) may identify the haptic intensity information of the haptic element based on the weights stored in the memory, the audio information, the information on the occurrence range of the motion event, and the information on the duration of the motion event, and control the haptic intensity of at least one haptic element corresponding to at least one identified area based on the identified haptic intensity information. However, it is not limited thereto, and it is of course possible for the processor (130) to receive haptic intensity information from an external display device through a communication interface (110).

[0077] In one embodiment, the processor (130) may provide haptic feedback at preset time intervals. In one example, assume that a vehicle movement event lasts for 1 minute. When information about the vehicle movement event is received, the processor (130) may provide haptic feedback for a preset time period (e.g., 2 seconds). Subsequently, if the vehicle movement event continues even after the preset time period has elapsed, the processor (130) may provide haptic feedback for an additional preset time period (e.g., 2 seconds).

[0078] FIG. 7 is a drawing for explaining a method of dividing a display into multiple areas according to one embodiment.

[0079] According to FIG. 7, according to one embodiment, the control method may first determine whether a user input for dividing the display into multiple regions is identified (S710). In one example, the processor (130) may receive a user input for dividing the display into multiple regions through a user interface. For example, the processor (130) may receive a user input for dividing the display into 15 regions through the user interface. However, this is not limited thereto, and the number of regions into which the display is divided may of course vary.

[0080] Alternatively, for example, the processor (130) may receive user input for dividing the display into multiple regions from an external device, including an external display device, via the communication interface (110).

[0081] Next, according to one embodiment, when a user input is identified (S710: Y), the control method can identify at least one haptic element corresponding to each of a plurality of regions among a plurality of haptic elements based on the user input (S720). According to one example, it is assumed that the display (120) includes 30 haptic elements. When a user input for dividing the display (120) into 15 regions is received, the processor (130) can identify 15 haptic elements corresponding to the 15 regions of the display (120) among the 30 haptic elements. For example, when two haptic elements exist in an region corresponding to a first region among the plurality of regions of the display (120), the processor (130) can identify one haptic element among the two haptic elements. When an object exists in the first area, the processor (130) can control one haptic element corresponding to the first area to provide haptic feedback.

[0082] Alternatively, when a user input for dividing the display (120) into 15 regions is received, for example, the processor (130) may identify 30 haptic elements corresponding to the 15 regions of the display (120) among 30 haptic elements. For example, when two haptic elements exist in a region corresponding to a second region among the multiple regions of the display (120), the processor (130) may identify the two haptic elements. For example, when an object exists in the second region, the processor (130) may control two haptic elements corresponding to the second region to provide haptic feedback.

[0083] Next, according to one embodiment, the control method can control at least one haptic element identified based on the received control information (S730).

[0084] FIG. 8 is a drawing for explaining a method of displaying a first UI according to one embodiment.

[0085] According to FIG. 8, according to one embodiment, the control method may first determine whether a user input for the electronic device (100) to operate as a game controller is received (S810). According to one example, the electronic device (100) may further include a memory in which first UI (User Interface) information corresponding to the game controller is stored. According to one example, the first UI information corresponding to the game controller may be information on a UI for receiving a user input for controlling a game output from an external display device. According to one example, the processor (130) may receive a user input for the electronic device (100) to operate as a game controller through the user interface. Alternatively, the processor (130) may also receive a user input for the electronic device (100) to operate as a game controller from an external display device through the communication interface (110).

[0086] Next, according to one embodiment, when a user input for the electronic device (100) to operate as a game controller is received (S810: Y), the control method may display a first UI (User Interface) corresponding to the game controller based on first UI (User Interface) information corresponding to the game controller stored in the memory (S820). According to one example, the processor (130) may display the first UI corresponding to the game controller through the display (120) based on the first UI information stored in the memory.

[0087] Next, according to one embodiment, the control method can determine whether a user input is received through the first UI (S830).

[0088] Next, according to one embodiment, when a user input is received through the first UI (S830: Y), the control method may transmit location information on the display (120) corresponding to the user input to an external display device (S840). According to one example, when a first user input is received through the first UI, the processor (130) may identify location information of an area of ​​the display (120) where the first user input is received among a plurality of areas, and transmit the identified location information to the external display device through the communication interface (110). When the identified location information is received, the external display device may identify a user input (or user command) corresponding to the identified location information, and perform an operation based on the identified user input.

[0089] For example, assume that a first user input for selecting a third area of ​​a display (120) is received through a first UI. The processor (130) can transmit location information of the third area corresponding to the first user input to an external display device through the communication interface (110). The external display device that has received the location information of the third area can identify the user input corresponding to the location information of the third area as a command for ending a game, and perform an operation for ending the game being played based on the identified command.

[0090] Alternatively, according to one embodiment, when a user input is received through the first UI, the processor (130) may identify control information corresponding to the user input based on the received user input. According to one example, the electronic device (100) may further include a memory in which first UI (User Interface) information corresponding to the game controller and control information corresponding to the first UI are stored. According to one example, the first UI may include a plurality of UIs corresponding to each of a plurality of functions of the game controller, and the control information corresponding to the first UI may include control information for performing each of a plurality of functions corresponding to each of the plurality of UIs.

[0091] For example, when a user input is received through the first UI, the processor (130) may identify control information corresponding to the user input based on information stored in the memory. For example, when a user input for selecting a UI corresponding to a first function among a plurality of functions included in the first UI is received, the processor (130) may identify the first function based on the received user input, and identify control information for performing the first function based on information stored in the memory.

[0092] For example, the processor (130) may transmit the identified control information to an external display device via the communication interface (110). For example, the processor (130) may transmit control information for performing a first function to the external display device, and the external display device may perform the first function based on the received control information.

[0093] FIGS. 9 and 10 are drawings and flowcharts for explaining a method of displaying a second UI corresponding to a remote control according to one embodiment.

[0094] According to FIG. 10, first, according to an embodiment, a control method may determine whether a user input for operating an electronic device (100) as a remote control is received (S1010). According to an example, the electronic device (100) may further include a memory in which second UI (User Interface) information corresponding to the remote control is stored. According to an example, the second UI information corresponding to the remote control may be information about a UI for receiving a user input for controlling content output from an external display device.

[0095] For example, the processor (130) may receive user input for the electronic device (100) to operate as a remote control via a user interface. Alternatively, the processor (130) may receive user input for the electronic device (100) to operate as a remote control from an external display device via a communication interface (110).

[0096] Next, according to one embodiment, when a user input for operating an electronic device (100) as a remote control is received (S1010: Y), the control method can display a second UI (User Interface) corresponding to the remote control based on second UI (User Interface) information corresponding to the remote control stored in the memory (S1020).

[0097] For example, as illustrated in FIG. 9, the processor (130) may display a second UI corresponding to the remote control through the display based on the second UI information stored in the memory. The processor (130) may display the second UI (901) through the display included in the electronic device (900). For example, the second UI (901) may include multiple UIs corresponding to each of multiple functions of the remote control, and each of the multiple UIs may be displayed in a corresponding area among the multiple areas of the display. For example, if there are 15 areas of the display, 15 UIs corresponding to each of the multiple functions may be displayed in the corresponding areas of the display.

[0098] Returning to FIG. 10, according to one embodiment, the control method can determine whether user input is received through the second UI (S1030).

[0099] According to one embodiment, when a user input is received through a second UI (S1030: Y), the control method may transmit location information on a display corresponding to the user input to an external display device (S1040). According to one example, when a second user input is received through the second UI, the processor (130) may identify location information of an area of ​​the display (120) where the second user input is received among a plurality of areas, and transmit the identified location information to the external display device through the communication interface (110). When the identified location information is received, the external display device may identify a user input (or user command) corresponding to the identified location information, and perform an operation based on the identified user input.

[0100] For example, assume that a second user input for selecting an 11th area of ​​the display (120) is received through a second UI. The processor (130) can transmit location information of the 11th area corresponding to the second user input to an external display device through the communication interface (110). The external display device that has received the location information of the 11th area can identify the user input corresponding to the location information of the 11th area as a command for ending or starting a game, and perform an operation for ending or starting the game being played based on the identified command.

[0101] Alternatively, according to one embodiment, when a user input is received through the second UI, the processor (130) may identify control information corresponding to the user input based on the received user input. According to one example, the electronic device (100) may further include a memory in which second UI (User Interface) information corresponding to the remote control and control information corresponding to the second UI are stored. According to one example, the second UI may include a plurality of UIs corresponding to each of a plurality of functions of the remote control, and the control information corresponding to the second UI may include control information for performing each of a plurality of functions corresponding to each of the plurality of UIs.

[0102] For example, when a user input is received through the second UI, the processor (130) may identify control information corresponding to the user input based on information stored in the memory. For example, when a user input is received for selecting a UI corresponding to a second function among a plurality of functions included in the second UI, the processor (130) may identify the second function based on the received user input, and identify control information for performing the second function based on information stored in the memory.

[0103] For example, the processor (130) may transmit the identified control information to an external display device via the communication interface (110). For example, the processor (130) may transmit control information for performing a second function to the external display device, and the external display device may perform the second function based on the received control information.

[0104] FIGS. 11A and 11B are drawings for explaining an image display method and a haptic feedback providing method according to one embodiment.

[0105] According to FIG. 11A, a first image (1100) may be displayed on an external display device according to an embodiment. When a preset motion event occurs within the first image (1100), the processor (130) may receive information corresponding to the motion event via the communication interface (110). In one example, the preset motion event may include a bomb explosion event and a gunshot wound event. When a bomb explosion event and a gunshot wound event occur within the first image (1100), the processor (130) may receive information corresponding to each of the bomb explosion event and the gunshot wound event via the communication interface (110). For example, the processor (130) may receive location information and image information corresponding to the bomb explosion event, and location information and image information corresponding to the gunshot wound event.

[0106] According to FIG. 11b, in one embodiment, the processor (130) may control the display to display a second image corresponding to the received motion event and cause a haptic element corresponding to at least one area of ​​the plurality of haptic elements to provide haptic feedback.

[0107] For example, let's assume that a bomb explosion event occurs in the second and third regions within the first image (1100), and a gunshot event occurs in the ninth region within the first image (1100). When location information and image information corresponding to the bomb explosion event are respectively received, and location information and image information corresponding to the gunshot event are respectively received, the processor (130) can display a second image (1111), in which an image corresponding to the explosion event is displayed in the second and third regions of the display, and an image corresponding to the gunshot event is displayed in the ninth region of the display, through a display included in the electronic device (1110), as illustrated in the left drawing of FIG. 11B.

[0108] For example, if a bomb explosion event occurs in the second and third regions within the first image (1100), and a gunshot event occurs in the ninth region within the first image (1100), as illustrated in the right drawing of FIG. 11B, the processor (130) may control the display so that haptic elements corresponding to the second region (1121), the third region (1122), and the ninth region (1123) of the display provide haptic feedback. In this case, at least one haptic element (e.g., one or two) corresponding to each of the second region (1121), the third region (1122), and the ninth region (1123) may provide haptic feedback. For example, the haptic intensity corresponding to each of the second area (1121), the third area (1122), and the ninth area (1123) may vary depending on audio information corresponding to the motion event, information about the range of occurrence of the motion event within the first image (1100), and information about the duration of the motion event.

[0109] FIG. 12a and FIG. 12b are drawings for explaining an image display method and a haptic feedback providing method according to one embodiment.

[0110] According to FIG. 12A, a first image (1200) may be displayed on an external display device according to an embodiment. When a preset motion event occurs within the first image (1200), the processor (130) may receive information corresponding to the motion event via the communication interface (110). In one example, the preset motion event may include a monster appearance event. When a monster appearance event occurs within the first image (1200), the processor (130) may receive information corresponding to the monster appearance event via the communication interface (110). For example, the processor (130) may receive location information and image information corresponding to the monster appearance event.

[0111] According to FIG. 12b, in one embodiment, the processor (130) may control the display to display a second image corresponding to the received motion event and cause a haptic element corresponding to at least one area of ​​the plurality of haptic elements to provide haptic feedback.

[0112] For example, let's assume that a monster appearance event occurs in the fourth area (1223), the ninth area (1221), and the tenth area (1222) within the first image (1200). When location information and image information corresponding to the monster appearance event are each received, as illustrated in the left drawing of FIG. 12B, the processor (130) can display a second image (1211) in which an image corresponding to the monster appearance event is displayed in the fourth area (1223), the ninth area (1221), and the tenth area (1222) of the display through a display included in the electronic device (1210).

[0113] For example, when a monster appearance event occurs in the fourth area (1223), the ninth area (1221), and the tenth area (1222) within the first image (1200), as illustrated in the right drawing of FIG. 12B, the processor (130) may control the display so that the haptic elements corresponding to the fourth area (1223), the ninth area (1221), and the tenth area (1222) of the display provide haptic feedback. In this case, at least one haptic element (e.g., one or two) corresponding to each of the fourth area (1223), the ninth area (1221), and the tenth area (1222) may provide the haptic feedback. For example, the haptic intensity corresponding to each of the fourth area (1223), the ninth area (1221), and the tenth area (1222) may vary depending on audio information corresponding to the motion event, information about the range of occurrence of the motion event within the first image (1200), and information about the duration of the motion event.

[0114] In one example, haptic feedback may be provided via a haptic element while a second image corresponding to a motion event is displayed. In other words, the action of displaying the second image via the display and the action of providing haptic feedback may be performed simultaneously.

[0115] Figure 13 is a block diagram showing a detailed configuration of an electronic device according to one embodiment.

[0116] According to FIG. 13, the electronic device (100') may include a communication interface (110), a display (120), one or more processors (130), a memory (140), a microphone (150), a speaker (160), at least one sensor (170), and a user interface (180). Among the configurations illustrated in FIG. 13, a detailed description of configurations that overlap with those illustrated in FIG. 2 will be omitted.

[0117] The memory (140) can store data required for various embodiments. The memory (140) may be implemented in the form of a memory embedded in the electronic device (100) or may be implemented in the form of a memory that can be attached or detached from the electronic device (100) depending on the purpose of data storage. For example, data for operating the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for expanding the functions of the electronic device (100) may be stored in a memory that can be attached or detached from the electronic device (100).

[0118] Meanwhile, in the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD)). In addition, in the case of memory that can be detachably attached to the electronic device (100), it may be implemented as at least one of memory cards (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc. It can be implemented.

[0119] A microphone (150) may refer to a module that acquires sound and converts it into an electrical signal, and may be a condenser microphone, a ribbon microphone, a moving coil microphone, a piezoelectric element microphone, a carbon microphone, or a MEMS (Micro Electro Mechanical System) microphone. In addition, it may be implemented in an omnidirectional, bidirectional, unidirectional, subcardioid, supercardioid, or hypercardioid manner.

[0120] The speaker (160) may be composed of a tweeter for reproducing high-frequency sounds, a midrange for reproducing mid-frequency sounds, a woofer for reproducing low-frequency sounds, a subwoofer for reproducing ultra-low-frequency sounds, an enclosure for controlling resonance, and a crossover network for dividing the frequency of an electric signal input to the speaker into bands.

[0121] The speaker (160) can output an audio signal to the outside of the electronic device (100'). The speaker (160) can output multimedia playback, recording playback, various notification sounds, voice messages, etc. The electronic device (100') may include an audio output device such as the speaker (160), but may also include an output device such as an audio output terminal. In particular, the speaker (160) can provide acquired information, information processed and produced based on acquired information, response results to user voice, operation results, etc. in voice form.

[0122] At least one sensor (170, hereinafter referred to as a sensor) may include a plurality of sensors of various types. The sensor (170) may measure a physical quantity or detect an operating state of an electronic device (100') and convert the measured or detected information into an electrical signal. The sensor (170) may include a camera, and the camera may include a lens that focuses visible light or other optical signals reflected by an object and received onto an image sensor, and an image sensor that may detect visible light or other optical signals. Here, the image sensor may include a 2D pixel array divided into a plurality of pixels. Alternatively, at least one sensor (170) may include a temperature sensor or an infrared sensor.

[0123] The user interface (180) is a configuration for the electronic device (100') to interact with the user. For example, the user interface (180) may include, but is not limited to, at least one of a touch sensor, a motion sensor, a button, a jog dial, a switch, a microphone, or a speaker.

[0124] According to the above-described example, when a motion event occurs within an external display device, the electronic device (100') can provide the user with an image corresponding to the motion event and simultaneously provide haptic feedback that enables the user to effectively recognize the motion event. Accordingly, the user can easily recognize the motion event, thereby enhancing user satisfaction.

[0125] Meanwhile, the methods according to the various embodiments of the present disclosure described above may be implemented in the form of applications that can be installed on existing electronic devices. Alternatively, the methods according to the various embodiments of the present disclosure described above may be performed using a deep learning-based trained neural network (or a deep learned neural network), i.e., a learning network model. Furthermore, the methods according to the various embodiments of the present disclosure described above may be implemented only through a software upgrade or a hardware upgrade of an existing electronic device. Furthermore, the various embodiments of the present disclosure described above may also be performed through an embedded server provided in an electronic device or an external server of the electronic device.

[0126] Meanwhile, according to a temporary example of the present disclosure, the various embodiments described above can be implemented as software including commands stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The machine is a device that can call commands stored from the storage medium and operate according to the called commands, and may include a display device (e.g., display device (A)) according to the disclosed embodiments. When a command is executed by a processor, the processor can perform a function corresponding to the command directly or by using other components under the control of the processor. The command may include code provided or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0127] Furthermore, according to one embodiment, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily provided in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0128] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0129] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In electronic devices, communication interface; A display having multiple haptic elements; and When information corresponding to the motion event is received through the communication interface based on a preset motion event occurring within a first image output from an external display device, at least one area corresponding to the motion event among a plurality of areas included in the display is identified, An electronic device that controls the display to display a second image corresponding to the received motion event in at least one area and to cause a haptic element corresponding to the at least one area among the plurality of haptic elements to provide haptic feedback.

2. In paragraph 1, The above preset motion events are: An event in which a preset dynamic object in the first image is identified, Information corresponding to the above motion event is: Contains location information of the above dynamic object and image information of the above dynamic object, One or more of the above processors, Identifying at least one area corresponding to the motion event among the plurality of areas based on the location information of the dynamic object, An electronic device that displays the second image in at least one area based on image information of the dynamic object.

3. In paragraph 2, One or more of the above processors, Identifying a representative color corresponding to the object based on the received image information, An electronic device displaying the second image including the dynamic object of the identified representative color.

4. In paragraph 1, The above plurality of haptic elements are, It is provided at a location corresponding to each of the above multiple areas, Information corresponding to the above motion event is: Further comprising control information for a haptic element corresponding to at least one area among the plurality of haptic elements, One or more of the above processors, An electronic device that controls at least one haptic element corresponding to the at least one area based on control information for the haptic element included in information corresponding to the motion event.

5. In paragraph 4, The above control information is, It further includes haptic intensity information of the haptic element, One or more of the above processors, Controlling the haptic intensity of at least one haptic element corresponding to at least one area based on the haptic intensity information of the haptic element; The haptic intensity information of the above haptic element is, An electronic device identified based on at least one of audio information corresponding to the motion event, information on the occurrence range of the motion event within the first image, and information on the duration of the motion event.

6. In paragraph 4, One or more of the above processors, When a user input for dividing the display into the plurality of regions is identified, at least one haptic element corresponding to each of the plurality of regions among the plurality of haptic elements is identified based on the user input, An electronic device that controls at least one identified haptic element based on the received control information.

7. In paragraph 1, The above electronic device, It further includes a memory storing first UI (User Interface) information corresponding to the game controller; One or more of the above processors, When the electronic device receives a user input for operating as the game controller, the electronic device displays a first UI corresponding to the game controller through the display based on the information stored in the memory. An electronic device that, when a user input is received through the first UI, transmits location information on the display corresponding to the user input to the external display device through the communication interface.

8. In paragraph 1, The above electronic device, It further includes a memory in which first UI (User Interface) information corresponding to the game controller and control information corresponding to the first UI are stored; One or more of the above processors, When the electronic device receives a user input for operating as the game controller, the electronic device displays a first UI corresponding to the game controller through the display based on the information stored in the memory. When a user input is received through the first UI, control information corresponding to the user input is identified based on the information stored in the memory, An electronic device that transmits the identified control information to the external display device through the communication interface.

9. In paragraph 1, The above electronic device, It further includes a memory in which second UI (User Interface) information corresponding to the remote control is stored; One or more of the above processors, When the electronic device receives a user input for operating the remote control, the electronic device displays a second UI corresponding to the remote control through the display based on the information stored in the memory. An electronic device that, when a user input is received through the second UI, transmits location information on the display corresponding to the user input to the external display device through the communication interface.

10. In paragraph 1, The above electronic device, It further includes a memory in which second UI (User Interface) information corresponding to the remote control and control information corresponding to the second UI are stored; One or more of the above processors, When the electronic device receives a user input for operating the remote control, the electronic device displays a second UI corresponding to the remote control through the display based on the information stored in the memory. When a user input is received through the second UI, control information corresponding to the user input is identified based on the information stored in the memory, An electronic device that transmits the identified control information to the external display device through the communication interface.

11. In paragraph 1, The above display is, An electronic device implemented with an EPD (Electrophoretic Display) film.

12. In a method for controlling an electronic device, A step of identifying at least one area corresponding to the motion event among a plurality of areas included in a display having a plurality of haptic elements, when information corresponding to the motion event is received based on a preset motion event occurring within a first image output from an external display device; A step of displaying a second image corresponding to the received motion event in at least one area; and A control method, comprising: a step of providing haptic feedback by a haptic element corresponding to at least one area among the plurality of haptic elements; 13. In paragraph 12, The above preset motion events are: An event in which a preset dynamic object in the first image is identified, Information corresponding to the above motion event is: Contains location information of the above dynamic object and image information of the above dynamic object, The above identifying step is, Identifying at least one area corresponding to the motion event among the plurality of areas based on the location information of the dynamic object, The steps indicated above are: A control method for displaying the second image in at least one area based on image information of the dynamic object.

14. In paragraph 13, Further comprising a step of identifying a representative color corresponding to the object based on the received image information; The steps indicated above are: A control method for displaying the second image including the dynamic object of the identified representative color.

15. A non-transitory computer-readable recording medium storing computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, wherein the operation is: A step of identifying at least one area corresponding to the motion event among a plurality of areas included in a display having a plurality of haptic elements, when information corresponding to the motion event is received based on a preset motion event occurring within a first image output from an external display device; A step of displaying a second image corresponding to the received motion event in at least one area; and A computer-readable recording medium, comprising: a step of providing haptic feedback by a haptic element corresponding to at least one area among the plurality of haptic elements;

Images