Method of displaying screen and electronic device for performing same

Abstract

An embodiment provides a display method by an electronic device. The electronic device may comprise a foldable housing including a first housing including a first display area on at least one surface thereof, a second housing including a second display area on at least one surface thereof, and a hinge rotatably connected to the first housing and the second housing while being interposed therebetween such that the first display area and the second display area face each other. The electronic device may be configured to: provide an execution screen of an application to at least one of the first display area and the second display area in a state in which the foldable housing is partially folded; detect an output event with respect to a visual object; and display the visual object in a target area including at least a portion of an unoccupied area other than an occupied area of the execution screen.

Description

Screen display method and electronic device performing the same

[0001] One embodiment relates to a method for displaying information on a screen in a foldable electronic device.

[0002] Recently, active research and development has been underway on foldable electronic devices, whose housings can be folded or unfolded, centered around a hinge structure. Foldable electronic devices can expand the display area when unfolded, while reducing volume when folded, thereby enhancing user convenience.

[0003] In one embodiment, an electronic device includes a foldable housing including a first housing including a first display area on at least one surface, a second housing including a second display area on at least one surface, and a hinge rotatably connected between the first housing and the second housing so that the first display area and the second display area face each other, at least one processor, and a memory including one or more storage media storing instructions, wherein when the instructions are individually or collectively executed by the at least one processor, the electronic device may: when an execution screen of an application is provided on at least one of the first display area and the second display area in a partially folded state, and an output event for a visual object is detected, display the visual object in a target area including at least a portion of a non-occupied area other than an occupied area of ​​the execution screen, wherein the target area may correspond to a display classification set for the visual object among a plurality of display classifications for the occupied area or the non-occupied area.

[0004] In one embodiment, a display method performed by an electronic device is provided, wherein the electronic device includes a foldable housing including a first housing including a first display area on at least one surface, a second housing including a second display area on at least one surface, and a hinge rotatably connected between the first housing and the second housing so that the first display area and the second display area face each other, at least one processor, and a memory including one or more storage media storing instructions, wherein the method may include the steps of providing an execution screen of an application to at least one of the first display area and the second display area in a state where the foldable housing is partially folded, detecting an output event for a visual object, and displaying the visual object in a target area including at least a portion of a non-occupied area other than an occupied area of ​​the execution screen, wherein the target area may correspond to a display classification set for the visual object among a plurality of display classifications for the occupied area or the non-occupied area.

[0005] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0006] FIG. 2 is a drawing illustrating an unfolded state of an electronic device according to one embodiment.

[0007] FIG. 3 is a drawing illustrating a folded state of an electronic device according to one embodiment.

[0008] FIG. 4 is a perspective view showing an example of a fully unfolded state or a partially unfolded intermediate state of an electronic device according to one embodiment.

[0009] FIG. 5 is a block diagram of a display module according to one embodiment.

[0010] FIG. 6 is a drawing illustrating a display of an electronic device operating in a flex state according to one embodiment.

[0011] FIG. 7 is a flowchart of a screen display method performed by an electronic device according to one embodiment.

[0012] FIGS. 8 to 11 are drawings explaining a screen display method of an electronic device according to one embodiment.

[0013] Hereinafter, an embodiment of the present document may be described with reference to the attached drawings.

[0014] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0015] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0016] The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0017] The auxiliary processor (123) may control at least a part of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0018] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0019] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0020] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0021] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0022] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0023] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0024] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0025] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0026] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0027] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0028] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0029] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0030] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0031] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0032] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0033] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0034] In one embodiment, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0035] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0036] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service on its own, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0037] FIG. 2 is a diagram illustrating an unfolded state of an electronic device according to an embodiment. FIG. 3 is a diagram illustrating a folded state of an electronic device according to an embodiment. FIG. 4 is a perspective view illustrating an example of a fully unfolded state or a partially unfolded intermediate state of an electronic device according to an embodiment.

[0038] The electronic device (200) of FIGS. 2 to 4 may be a foldable or bendable electronic device, as an example of the electronic device (101) illustrated in FIG. 1.

[0039] The drawings below in FIG. 4 illustrate a spatial coordinate system defined by an X-axis, a Y-axis, and a Z-axis that are orthogonal to each other. Here, the X-axis may represent the width direction of the electronic device, the Y-axis may represent the length direction of the electronic device, and the Z-axis may represent the height (or thickness) direction of the electronic device. In the following description, the term "first direction" may mean a direction parallel to the Z-axis.

[0040] Referring to FIGS. 2 and 3, in one embodiment, the electronic device (200) may include a foldable housing (201), and a flexible or foldable display (250) (hereinafter, simply “display” 250) (e.g., the display device (160) of FIG. 1) disposed within a space formed by the foldable housing (201). According to one embodiment, a surface on which the display (250) is disposed (or a surface on which the display (250) is visible from the outside of the electronic device (200)) may be defined as a front surface of the electronic device (200). And, a surface opposite to the front surface may be defined as a rear surface of the electronic device (200). In addition, a surface surrounding the space between the front surface and the rear surface may be defined as a side surface of the electronic device (200).

[0041] According to one embodiment, the foldable housing (201) may include a first housing structure (210), a second housing structure (220) including a sensor area (222), a first rear cover (215), a second rear cover (225), and a hinge structure (230). Here, the hinge structure (230) may include a hinge cover that covers a foldable portion of the foldable housing (201). The foldable housing (201) of the electronic device (200) is not limited to the shape and combination shown in FIGS. 2 and 3, and may be implemented by a combination and / or combination of other shapes or parts. For example, in one embodiment, the first housing structure (210) and the first rear cover (215) may be formed integrally, and the second housing structure (220) and the second rear cover (225) may be formed integrally.

[0042] According to one embodiment, the first housing structure (210) is connected to the hinge structure (230) and may include a first side facing a first direction and a second side facing a second direction opposite to the first direction. The second housing structure (220) is connected to the hinge structure (230) and may include a third side facing a third direction and a fourth side facing a fourth direction opposite to the third direction. The second housing structure (220) may rotate about the hinge structure (230) with respect to the first housing structure (210). The electronic device (200) may be variable between a folded state and an unfolded state.

[0043] According to one embodiment, the electronic device (200) may have the first side facing the third side in a fully folded or closed state, and the third direction may be the same as the first direction in a fully unfolded or open state.

[0044] According to one embodiment, the first housing structure (210) and the second housing structure (220) are arranged on both sides with respect to the folding axis (A) as the center, and may have an overall symmetrical shape with respect to the folding axis A. As described below, the angle or distance between the first housing structure (210) and the second housing structure (220) may vary depending on whether the state of the electronic device (200) is an unfolded state, a folded state, or a partially open (or partially folded) intermediate state. According to one embodiment, unlike the first housing structure (210), the second housing structure (220) additionally includes a sensor area (222) in which various sensors are arranged, but may have a mutually symmetrical shape in other areas.

[0045] According to one embodiment, as illustrated in FIG. 2, the first housing structure (210) and the second housing structure (220) may together form a recess for accommodating the display (200). According to one embodiment, due to the sensor area (222), the recess may have two or more different widths in a direction perpendicular to the folding axis (A). According to one embodiment, the recess may have a first width (w1) between a first portion (210a) of the first housing structure (210) that is parallel to the folding axis (A) and a first portion (220a) formed at an edge of the sensor area (222) of the second housing structure (220), and the recess may have a second width (w2) formed by a second portion (210b) of the first housing structure (210) and a second portion (220b) of the second housing structure (220) that is parallel to the folding axis A and does not correspond to the sensor area (222). In this case, the second width (w2) may be formed longer than the first width (w1). According to one embodiment, the first portion (220a) and the second portion (220b) of the second housing structure (220) may have different distances from the folding axis A. The width of the recess is not limited to the illustrated example. In another embodiment, the recess may have multiple widths due to the shape of the sensor area (222) or the asymmetrical shape of the first housing structure (210) and the second housing structure (220). In one embodiment, the sensor area (222) may be formed to have a predetermined area adjacent to one corner of the second housing structure (220). However, the arrangement, shape, and size of the sensor area (222) are not limited to the illustrated example. For example, in one embodiment, the sensor area (222) may be provided in another corner of the second housing structure (220) or in any area between the upper corner and the lower corner.In one embodiment, components for performing various functions built into the electronic device (200) may be exposed on the front of the electronic device (200) through the sensor area (222) or through one or more openings provided in the sensor area (222). In one embodiment, the components may include various types of sensors. The sensors may include, for example, at least one of a front camera, a receiver, or a proximity sensor. According to one embodiment, the sensor area (222) in the second housing structure (220) may be omitted or formed in a position different from that shown in the drawing.

[0046] According to one embodiment, at least a portion of the first housing structure (210) and the second housing structure (220) may be formed of a metallic or non-metallic material having a rigidity of a size selected to support the display (250). At least a portion formed of the metallic material may provide a ground plane of the electronic device (200) and may be electrically connected to a ground line formed on a printed circuit board disposed inside the foldable housing (201).

[0047] According to one embodiment, the first rear cover (215) is disposed on one side of the folding axis (A) on the rear surface of the electronic device (200) and may have, for example, a substantially rectangular periphery, the periphery of which may be wrapped by the first housing structure (210). Similarly, the second rear cover (225) is disposed on the other side of the folding axis (A) on the rear surface of the electronic device (200) and the periphery of which may be wrapped by the second housing structure (220).

[0048] According to one embodiment, the first rear cover (215) and the second rear cover (225) may have substantially symmetrical shapes with respect to the folding axis (A). However, the first rear cover (215) and the second rear cover (225) do not necessarily have mutually symmetrical shapes, and in one embodiment, the electronic device (200) may include the first rear cover (215) and the second rear cover (225) of various shapes. In addition, in one embodiment, the first rear cover (215) may be formed integrally with the first housing structure (210), and the second rear cover (225) may be formed integrally with the second housing structure (220).

[0049] According to one embodiment, the first rear cover (215), the second rear cover (225), the first housing structure (210), and the second housing structure (220) may form a space in which various components of the electronic device (200) (e.g., a printed circuit board or a battery) may be placed. According to one embodiment, one or more components may be placed or visually exposed on the rear surface of the electronic device (200). For example, at least a portion of the sub-display may be visually exposed through the first rear area (216) of the first rear cover (215). In one embodiment, one or more components or sensors may be visually exposed through the second rear area (226) of the second rear cover (225). In one embodiment, the sensors may include a proximity sensor and / or a rear camera.

[0050] According to one embodiment, a front camera exposed on the front side of the electronic device (200) through one or more openings provided in the sensor area (222) or a rear camera exposed through the second rear area (226) of the second rear cover (225) may include one or more lenses, an image sensor, and / or an image signal processor. A flash may include, for example, a light emitting diode or a xenon lamp. In some embodiments, two or more lenses (infrared camera, wide-angle and telephoto lenses) and image sensors may be arranged on one side of the electronic device (200).

[0051] Referring to FIG. 3, the hinge cover may be configured to be disposed between the first housing structure (210) and the second housing structure (220) so as to cover an internal component (e.g., the hinge structure (230)). According to one embodiment, the hinge structure (230) may be covered by a portion of the first housing structure (310) and the second housing structure (320) or exposed to the outside depending on the state of the electronic device (200) (unfolded state, intermediate state, or folded state).

[0052] According to one embodiment, as illustrated in FIG. 2, when the electronic device (200) is in an unfolded state (e.g., a fully unfolded state), the hinge structure (230) may be covered by the first housing structure (210) and the second housing structure (220) and not exposed. As another example, as illustrated in FIG. 3, when the electronic device (200) is in a folded state (e.g., a fully folded state), the hinge structure (230) may be exposed to the outside between the first housing structure (210) and the second housing structure (220). As another example, when the first housing structure (210) and the second housing structure (220) are in an intermediate state where they are folded at a certain angle, the hinge structure (230) may be partially exposed to the outside between the first housing structure (210) and the second housing structure (220). However, in this case, the exposed area may be less than that in the completely folded state. In one embodiment, the hinge structure (230) may include a curved surface.

[0053] According to one embodiment, the display (250) may be placed on a space formed by the foldable housing (201). For example, the display (250) may be mounted on a recess formed by the foldable housing (201) and may be visible from the outside through the front of the electronic device (200). For example, the display (250) may constitute most of the front of the electronic device (200). Accordingly, the front of the electronic device (200) may include the display (250) and a portion of the first housing structure (210) adjacent to the display (250) and a portion of the second housing structure (220). And, the back of the electronic device (200) may include a first back cover (215), a portion of a first housing structure (210) adjacent to the first back cover (215), a second back cover (225), and a portion of a second housing structure (220) adjacent to the second back cover (225).

[0054] According to one embodiment, the display (250) may refer to a display in which at least a portion of the display can be transformed into a flat or curved surface. According to one embodiment, the display (250) may include a folding area (253), a first area (251) arranged on one side (e.g., the left side of the folding area (253) illustrated in FIG. 2) with respect to the folding area (253), and a second area (252) arranged on the other side (e.g., the right side of the folding area (253) illustrated in FIG. 2).

[0055] However, the division of the regions of the display (250) illustrated in FIG. 2 is exemplary, and the display (250) may be divided into a plurality of regions (for example, four or more or two) depending on the structure or function. For example, in the embodiment illustrated in FIG. 2, the regions of the display (200) may be divided by a folding region (203) extending parallel to the folding axis (A), but in one embodiment, the regions of the display (200) may be divided based on another folding axis (for example, a folding axis parallel to the width direction of the electronic device).

[0056] According to one embodiment of the present disclosure, the display (250) may be coupled with or disposed adjacent to a touch panel equipped with a touch detection circuit and a pressure sensor capable of measuring the intensity (pressure) of a touch. For example, the display (250) may be coupled with or disposed adjacent to a touch panel that detects an electromagnetic resonance (EMR) stylus pen, as an example of a touch panel.

[0057] According to one embodiment, the first region (251) and the second region (252) may have an overall symmetrical shape centered on the folding region (253). However, unlike the first region (251), the second region (252) may include a cut notch depending on the presence of the sensor region (222), but may have a shape symmetrical with respect to the first region (251) in other regions. In other words, the first region (251) and the second region (252) may include a portion having a symmetrical shape and a portion having an asymmetrical shape.

[0058] According to one embodiment, the edge thicknesses of the first region (251) and the second region (252) may be formed to be different from the edge thickness of the folding region (253). The edge thickness of the folding region (253) may be formed to be thinner than the thicknesses of the first region (251) and the second region (252). In terms of thickness, the first region (251) and the second region (252) may have an asymmetrical shape when the first region (251) and the second region (252) are viewed in cross-section. For example, the edge of the first region (251) may be formed to have a first radius of curvature, and the edge of the second region (252) may be formed to have a second radius of curvature that is different from the first radius of curvature. In one embodiment, the first region (251) and the second region (252) in terms of thickness may have a symmetrical shape when the first region (251) and the second region (252) are viewed in cross section. This will be described in detail below through embodiments disclosed in the drawings below, FIG. 10A.

[0059] Hereinafter, the operation of the first housing structure (210) and the second housing structure (220) and each area of ​​the display (250) according to the state of the electronic device (200) (e.g., folded state, unfolded state, or intermediate state) will be described.

[0060] According to one embodiment, when the electronic device (200) is in an unfolded state (e.g., FIG. 2), the first housing structure (210) and the second housing structure (220) may be arranged to form an angle of 180 degrees and face the same direction. The surface of the first region (251) and the surface of the second region (252) of the display (250) may form an angle of 180 degrees with each other and face the same direction (e.g., toward the front of the electronic device). The folding region (253) may form the same plane as the first region (251) and the second region (252).

[0061] According to one embodiment, when the electronic device (200) is in a folded state (e.g., FIG. 3), the first housing structure (210) and the second housing structure (220) may be arranged to face each other. The surface of the first region (251) and the surface of the second region (252) of the display (250) may form a narrow angle (e.g., between 0 and 10 degrees) with each other and may face each other. The folding region (253) may be formed as a curved surface having at least a portion of a predetermined curvature.

[0062] According to one embodiment, when the electronic device (200) is in an intermediate state, the first housing structure (210) and the second housing structure (220) may be arranged at a certain angle with respect to each other. The surface of the first region (251) and the surface of the second region (252) of the display (250) may form an angle that is larger than an angle in a folded state and smaller than an angle in an unfolded state. The folding region (253) may be formed as a curved surface having at least a certain curvature, and the curvature at this time may be smaller than that in a folded state.

[0063] FIG. 4(a) illustrates a fully unfolded state of the electronic device (200), and FIG. 4(b) may illustrate an intermediate state in which the electronic device (200) is partially unfolded. As described above, the electronic device (200) may be variable between a folded state and an unfolded state. According to one embodiment, the electronic device (200) may be folded in two ways: 'in-folding' in which the front surface of the electronic device (200) is folded to form an acute angle when viewed in the folding axis direction (e.g., the A-axis of FIG. 2), and 'out-folding' in which the front surface of the electronic device (200) is folded to form an obtuse angle. For example, in the electronic device (200), in a folded state in an in-folding manner, a first side of the first housing structure (210) may face a third side of the second housing structure (220), and in a fully unfolded state, the first side of the first housing structure (210) and the third side of the second housing structure (220) may face the same direction (e.g., a direction parallel to the Z-axis).

[0064] As another example, the electronic device (200) may be folded in an out-folding manner so that the second side of the first housing structure (210) faces the fourth side of the second housing structure (220).

[0065] Additionally, the electronic device (200) may include multiple hinge axes (e.g., two mutually parallel hinge axes including the A-axis of FIG. 2 and another axis parallel to the A-axis), although not shown in the drawing, in which case the electronic device (200) may be folded in a 'multi-folding' manner that combines the in-folding and out-folding methods.

[0066] The above-described in-folding type may refer to a state in which the display (250) is not exposed to the outside in a fully folded state. The above-described out-folding type may refer to a state in which the display (250) is exposed to the outside in a fully folded state. FIG. 4 (b) illustrates an intermediate state in which the electronic device (200) is partially unfolded during the in-folding process.

[0067] For convenience, the following description focuses on a state in which the electronic device (200) is folded in an in-folding manner, but it should be noted that these descriptions can also be applied to a state in which the electronic device (200) is folded in an out-folding manner.

[0068] According to one embodiment, a sensor module of the electronic device (200) (e.g., sensor module (176) of FIG. 1) can detect a folding angle of the electronic device (200). The sensor module according to one embodiment may be a hall sensor, an acceleration sensor, and / or a gyro sensor (e.g., a gyroscope), and the acceleration sensor and / or the gyro sensor can detect the folding angle of the electronic device (200). The sensor module according to one embodiment may be disposed within the hinge structure (230) to detect the number of gear rotations to detect the folding angle. The sensor module according to one embodiment may detect not only the folding angle but also the folding start point and the angle at which folding stops.

[0069] For example, the sensor module may be a Hall sensor. The Hall sensor may be a transducer means that generates an electrical signal (e.g., voltage) in response to a magnetic field. The Hall sensor may generate an electrical signal of relatively large strength when the magnetic field is strong, and may generate an electrical signal of relatively small strength when the magnetic field is weak. The Hall sensor may be located in the first housing structure (210), and an element that generates a magnetic field (e.g., a magnetic body) may be located in the second housing structure (220). Depending on the implementation, the Hall sensor may be located in the second housing structure (220), and the magnetic body may be located in the first housing structure (210). When the electronic device (200) changes from a fully unfolded state to a folded state, the Hall sensor may come close to the magnetic body, and may detect the folding state of the display (250) in response to the magnetic field of the magnetic body. In addition, the Hall sensor may detect a folding angle through the strength of the electrical signal generated in response to the magnetic field.

[0070] The Hall sensor is an example of a sensor module, and the sensor module is not limited to the Hall sensor. According to one embodiment, the folding angle of the electronic device (200) can be detected using a magnetic (geomagnetic) sensor and a Hall sensor. For example, the magnetic (geomagnetic) sensor and the Hall sensor can include a transmitter for generating a magnetic field of a specific frequency and a receiver for receiving the magnetic field generated by the transmitter, and can detect the folding angle based on a change in the state of the electronic device (200). For example, the magnetic (geomagnetic) sensor can measure the direction using a magnetic field and magnetic force lines, and the Hall sensor can detect the strength of the magnetic field to confirm a change in the state of the electronic device (200) and detect a folding angle according to the confirmed change in the state.

[0071] As described above, the sensor module detects the folding state of the display (250) and detects the folding angle. However, according to one embodiment, the processor can detect the folding state of the display (250) and calculate the folding angle based on the sensing result of the sensor module. For example, the processor can receive an electrical signal according to magnetic field detection from the Hall sensor. The processor can detect the folding state of the display (250) by receiving the electrical signal from the Hall sensor. In addition, the processor can calculate the folding angle of the display (250) based on the intensity of the electrical signal received from the Hall sensor.

[0072] According to one embodiment, the processor can determine the movement of the electronic device (200) by measuring the azimuth, pitch, and roll values ​​of the electronic device (200) through motion data acquired from the sensor module. The motion data according to one embodiment may include 3-axis motion data (x1, y1, z1) acquired from an acceleration sensor, or 9-axis motion data acquired additionally using a gyro sensor and a geomagnetic sensor.

[0073] FIG. 5 is a block diagram of a display module according to one embodiment.

[0074] Referring to FIG. 5, a display module (560) (e.g., the display module (160) of FIG. 1) may include a display (510) and a display driver IC (DDI) (530) for controlling the display (510). The DDI (530) may include an interface module (531), a memory (533) (e.g., a buffer memory), an image processing module (535), or a mapping module (537). The DDI (530) may receive, for example, image information including image data or an image control signal corresponding to a command for controlling the image data, from another component of an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (200) of FIGS. 2 to 4) through the interface module (531). For example, according to one embodiment, image information may be received from a processor (e.g., a main processor such as an application processor or an auxiliary processor such as a graphics processing unit that operates independently of the function of the main processor) (e.g., a processor (120) of FIG. 1). The DDI (530) may communicate with a touch circuit (550) or a sensor module (576) through the interface module (531). In addition, the DDI (530) may store at least a portion of the received image information in the memory (533), for example, in units of frames. The image processing module (535) may, for example, perform preprocessing or postprocessing (e.g., resolution, brightness, or size adjustment) on at least a portion of the image data based on at least a characteristic of the image data or a characteristic of the display (510). The mapping module (537) may generate a voltage value or a current value corresponding to the image data preprocessed or postprocessed through the image processing module (535). According to one embodiment, the generation of the voltage value or the current value may be performed by, for example, It may be performed at least in part based on the properties of the pixels of the display (510) (e.g., the arrangement of the pixels (RGB stripe or pentile structure), or the size of each sub-pixel).At least some pixels of the display (510) may be driven based at least in part on, for example, the voltage value or current value, so that visual information (e.g., text, an image, or an icon) corresponding to the image data may be displayed through the display (510).

[0075] According to one embodiment, the display module (560) may further include a touch circuit (550). The touch circuit (550) may include a touch sensor (551) and a touch sensor IC (553) for controlling the same. The touch sensor IC (553) may control the touch sensor (551) to detect, for example, a touch input or a hovering input for a specific location of the display (510). For example, the touch sensor IC (553) may detect the touch input or the hovering input by measuring a change in a signal (e.g., voltage, light amount, resistance, or charge amount) for a specific location of the display (510). The touch sensor IC (553) may provide information (e.g., location, area, pressure, or time) regarding the detected touch input or hovering input to the processor. In one embodiment, at least a portion of the touch circuit (550) (e.g., touch sensor IC (553)) may be included as part of the display driver IC (530), or as part of the display (510), or as part of another component (e.g., auxiliary processor (123) of FIG. 1) disposed external to the display module (560).

[0076] According to one embodiment, the display module (560) may further include at least one sensor (e.g., a fingerprint sensor, an iris sensor, a pressure sensor, or an illuminance sensor) of the sensor module (576), or a control circuit therefor. In this case, the at least one sensor or the control circuit therefor may be embedded in a part of the display module (560) (e.g., the display (510) or the DDI (530)) or a part of the touch circuit (550). For example, if the sensor module (576) embedded in the display module (560) includes a biometric sensor (e.g., a fingerprint sensor), the biometric sensor may obtain biometric information (e.g., a fingerprint image) associated with a touch input through a part of the display (510). As another example, if the sensor module (576) embedded in the display module (560) includes a pressure sensor, the pressure sensor may obtain pressure information associated with a touch input through a part or the entire area of ​​the display (510). According to one embodiment, the touch sensor (551) or sensor module (576) may be positioned between pixels of a pixel layer of the display (510), or above or below the pixel layer.

[0077] FIG. 6 is a drawing illustrating a display of an electronic device operating in a flex state according to one embodiment.

[0078] FIG. 6 is an example of a screen displayed through an electronic device (200) (e.g., a display of the electronic device (101) of FIG. 1 (e.g., a display module (160) of FIG. 1, a display (250) of FIGS. 2 to 4, a display (510) of FIG. 5). The electronic device (200) may include at least some of the components of the electronic device (101) described above with reference to FIG. 1. For example, the electronic device (200) may include at least one processor including a processing circuit (e.g., a processor (120) of FIG. 1), a memory including one or more storage media for storing instructions (e.g., a memory (130) of FIG. 1), and a display.

[0079] Referring to FIG. 6, the electronic device (200) may be in an intermediate state, i.e., a partially folded state, and the display may be operating in a flex state.

[0080] In the present disclosure, the "flex state" of the electronic device (200) may be defined as a state in which the first side of the display displays an execution screen of an application while simultaneously controlling the second side of the display to display at least one menu or control icon for controlling the execution screen. The term and expression "flex state" used in the present disclosure may be changed and replaced with terms such as flex mode. For example, the flex state of the display may be operated in a state in which the second housing structure (220) is placed on the bottom surface and the first housing structure (210) forms a specific angle with respect to the second housing structure (220). In this case, the fourth side of the second housing structure (220) may be placed so as to face the direction of gravity (G). The electronic device (200) can display all or at least a part of a screen related to an application being executed through a first display area (251) located in the first housing structure (210) when the second housing structure (220) is placed on the floor and the first housing structure (210) is at a specific angle from the second housing structure (220), and can display a control element (610) for controlling the screen through a second display area (252) located in the second housing structure (220). For example, the flex state of the display can be driven when the first housing structure (210) is placed on the floor and the second housing structure (220) is at a specific angle from the first housing structure (210). In this case, the second surface of the first housing structure (210) can be placed so as to face the direction of gravity (G).The electronic device (200) may display all or at least a part of a screen related to an application being executed through a second area (252) (hereinafter, referred to as the second display area (252)) of a display located in the second housing structure (220) when the first housing structure (210) is placed on the floor and the second housing structure (220) is at a specific angle from the first housing structure (210), and may display a control element (610) for controlling the screen through a first area (251) (hereinafter, referred to as the first display area (251)) of a display located in the first housing structure (210). According to one embodiment, the flex state may be driven in a state in which either the first housing structure (210) or the second housing structure (220) is placed on the floor or in a state in which one side of the first housing structure (210) or the second housing structure (220) is placed or maintained so as to face the direction of gravity (G). One side of the first housing structure (210) or the second housing structure (220) facing the direction of gravity (G) may include a state in which the normal of one side of the first housing structure (210) or the second housing structure (220) forms an angle within a critical angle with the direction of gravity.

[0081] The control element (610) may include at least one control icon (610a) for controlling the screen. For example, if the screen is an execution screen of an application, the control element (610) may include at least one button (610a) for controlling the execution screen. For example, if the screen is a media playback application, the control element (610) may include at least one button (610a) for controlling the playback of the media. For example, if the screen is a camera application, the control element (610) may include at least one button (610a) related to capturing an image.

[0082] According to one embodiment, the electronic device (200) may display all or at least a portion of a screen related to an application being executed through a first display area (251) located in the first housing structure (210) when the second housing structure (220) is placed on the floor and the first housing structure (210) is at a specific angle from the second housing structure (220), and may display another application related to the application being executed through a second display area (252) located in the second housing structure (220). For example, when a video call application is being executed through the first display area (251), a contact application may be displayed through the second display area (252).

[0083] The first display area (251) and / or the second display area (252) may have an occupied area. The occupied area may represent an area in the first display area (251) and / or the second display area (252) where a visual object (or a visual user interface (UI) element) is displayed.

[0084] The occupied area may represent at least a portion of the first display area (251) and the second display area (252) where a visual object in the form of a pop-up is displayed on the home screen or lock screen of the electronic device (200). For example, the occupied area may be an area where a visual object for a banner, a drop-down menu, or a notification is displayed. The occupied area may be an area where a visual object is temporarily or continuously displayed.

[0085] The occupied area may represent at least a portion of the first display area (251) and the second display area (252) on which visual objects in the form of widgets, such as a clock, weather, calendar, and gallery, are displayed on the home screen or lock screen of the electronic device (200). The occupied area may be an area on which visual objects, such as icons (e.g., application icons, functional icons such as favorites), buttons, text, sliders, and switches (e.g., toggle switches), are displayed.

[0086] The occupied area may represent at least a portion of the first display area (251) and the second display area (252) on which visual objects are displayed on the execution screen of the application. For example, the occupied area may correspond to the entire execution screen of the application on which visual objects are displayed. For example, the occupied area may correspond to at least a portion of the execution screen of the application on which content elements of the application are displayed. For example, the occupied area may correspond to at least a portion of the execution screen of the application on which control elements (610) are displayed.

[0087] The first display area (251) and / or the second display area (252) may have a non-occupied area. The non-occupied area may refer to an area in the first display area (251) and / or the second display area (252) where no visual object (or visual user interface (UI) element) is displayed.

[0088] The non-occupied area may represent at least a portion of the remaining area excluding at least a portion of the first display area (251) and the second display area (252) where a visual object in the form of a pop-up is displayed on the home screen or lock screen of the electronic device (200).

[0089] The non-occupied area may refer to at least a portion of the remaining area excluding at least a portion of the first display area (251) and the second display area (252) where visual objects in the form of widgets, such as a clock, weather, calendar, and gallery, are displayed on the home screen or lock screen of the electronic device (200). The non-occupied area may be an area where visual objects, such as icons, buttons, text, sliders, and switches, are not displayed.

[0090] The non-occupied area may represent at least a portion of the remaining area excluding at least a portion of the first display area (251) and the second display area (252) in which visual objects are displayed on the execution screen of the application. For example, the non-occupied area may correspond to the remaining area excluding the area in which the content elements of the application are displayed on the execution screen of the application. For example, the non-occupied area may correspond to the remaining area excluding the area in which the control elements (610) are displayed on the execution screen of the application. The non-occupied area may correspond to the area in which no visual objects are displayed on the execution screen of the application.

[0091] The non-occupied area may be an area where the first display area (251) and / or the second display area (252) is temporarily or non-temporarily disabled. The non-occupied area may be an area where the first display area (251) and / or the second display area (252) is displayed in black for a specified period of time or longer. The non-occupied area may be any area where content elements or control elements for user interaction are not substantially displayed.

[0092] FIG. 7 is a flowchart of a screen display method performed by an electronic device according to one embodiment.

[0093] The operations 710 to 730 below can be performed by an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (200) of FIGS. 2 to 4 and 6). Hereinafter, the operations 710 to 730 are performed in a state where the display of the electronic device (e.g., the display (250) of FIGS. 2 to 4 or the display (510) of FIG. 5) operates in a flex mode.

[0094] In operation 710, the electronic device may provide an execution screen of an application in at least one of a first display area (e.g., the first display area (251) of FIG. 6) and a second display area (e.g., the second display area (252) of FIG. 6) while the foldable housing is partially folded. Providing the execution screen may include, for example, displaying, presenting, representing, reproducing, rendering, disposing, and including the execution screen.

[0095] According to one embodiment, an electronic device may provide content elements of an application to a first display area. The content elements may represent graphical representations indicating content. The electronic device may, for example, adjust the size and / or shape of the content elements in full view mode, or crop a portion of the content elements in full view mode and output them to a first display area smaller than the entire area.

[0096] According to one embodiment, the electronic device may output, in a second display area, a control element (e.g., a control element (610) of FIG. 6) corresponding to one or more items of a lower level than the first level, belonging to a first level menu item mapped to a flex mode in an application. The control element is a graphical representation indicating a control item selectable by a user on the display, and the electronic device may execute an operation and / or function of an application assigned to the control item in response to the selection of the control item. The operation and / or function of the application assigned to the control item may include, for example, deforming, changing, and replacing content elements and / or control elements output on the display. For example, the operation and / or function of the application assigned to the control item may include executing various processes, software modules, and other applications. Replacing a control element may, for example, represent an action of erasing a control element of a certain level that was being output from a second display area and instead outputting a control element of a lower level to the second display area. For convenience of explanation, the following description assumes that the application's content elements are output to the first display area and the control elements are output to the second display area.

[0097] At operation 720, the electronic device can determine whether an output event exists for a visual object.

[0098] According to one embodiment, an electronic device may receive a notification generated by any application of the electronic device from the application, or a notification generated from an external electronic device connected to the electronic device via a network. By receiving a notification, the electronic device may detect an output event for a visual object corresponding to the notification.

[0099] In one embodiment, any application on an electronic device can generate a notification when a certain condition is met. For example, a condition may be the receipt of a new message in a messenger application or the receipt of a call in a phone application. For example, a condition may be the occurrence of an event designated to generate a notification in an application, such as the start of product delivery in a shopping application.

[0100] In one embodiment, the electronic device may detect an output event for a visual object corresponding to an advertisement of a running application. For example, the electronic device may detect an output event for a visual object corresponding to an advertisement video for watching a video of an entertainment application.

[0101] In operation 730, the electronic device may display a visual object in a target area that includes at least a portion of a non-occupied area other than the occupied area of ​​the execution screen of the application.

[0102] The occupied area may refer to an area where a visual object (or a visual user interface (UI) element) is displayed in the first display area and / or the second display area. The occupied area may refer to at least a portion of the first display area and the second display area where a visual object in the form of a pop-up is displayed on the home screen or lock screen of the electronic device. The occupied area may refer to at least a portion of the first display area and the second display area where a visual object in the form of a widget, such as a clock, weather, calendar, or gallery, is displayed on the home screen or lock screen of the electronic device. The occupied area may refer to at least a portion of the first display area and the second display area where a visual object is displayed on the execution screen of an application. For example, the occupied area may correspond to the entire execution screen of an application where a visual object is displayed. For example, the occupied area may correspond to at least a portion of the execution screen of an application where a content element of the application is displayed. For example, the occupied area may correspond to at least a portion of the execution screen of an application where a control element is displayed.

[0103] The non-occupied area may refer to an area in which no visual object (or visual user interface (UI) element) is displayed in the first display area and / or the second display area. The non-occupied area may refer to at least a portion of the remaining area excluding at least a portion of the first display area and the second display area where visual objects in the form of pop-ups are displayed on the home screen or the lock screen of the electronic device. The non-occupied area may refer to at least a portion of the remaining area excluding at least a portion of the first display area and the second display area where visual objects in the form of widgets, such as a clock, weather, calendar, and gallery, are displayed on the home screen or the lock screen of the electronic device. The non-occupied area may refer to at least a portion of the remaining area excluding at least a portion of the first display area and the second display area where visual objects are displayed on the execution screen of the application.

[0104] The target area may represent at least a portion of the first display area and the second display area corresponding to a display classification distinguished based on the priority or importance of the visual object. The target area may correspond to a display classification set for the visual object among a plurality of display classifications for the occupied area or the non-occupied area.

[0105] According to one embodiment, an electronic device can identify information about an occupied area of ​​an execution screen of an application. Identifying information about the occupied area may include, for example, detecting, obtaining, recognizing, determining, and sensing information about the occupied area. The electronic device can form at least one surface for rendering an image of an execution screen to be output to a display, and the at least one surface can be synthesized by a surface flinger to be output to the display. The electronic device can form views (e.g., text, images) on the surface according to a view hierarchy (e.g., a tree structure) and store information about the occupied area of ​​each view.

[0106] According to one embodiment, information about the occupied area may include coordinate information about the occupied area of ​​the application's execution screen. The coordinate information may include coordinate information of at least a portion of a first display area where content elements of the application are displayed and at least a portion of a second display area where control elements are displayed. The coordinate information may include start coordinate information and end coordinate information. Based on the information about the occupied area, the electronic device may display a visual object in a target area that includes at least a portion of the non-occupied area.

[0107] According to one embodiment, an electronic device may identify an occupied area and / or an unoccupied area based on a rectangular shape. For example, an occupied area may be identified as a minimum rectangular shape including an area where a visual object is displayed, or a combination of a plurality of rectangular shapes. A unoccupied area may be identified as a rectangular shape excluding an occupied area from a display area (e.g., a first display area, a second display area), or as a remaining area excluding a plurality of rectangular shapes corresponding to an occupied area. However, the identification of an occupied area and an unoccupied area is not limited to a specific shape, and for example, an electronic device may identify an occupied area and / or an unoccupied area by dividing the occupied area and / or the unoccupied area by a specific boundary.

[0108] According to one embodiment, an electronic device may identify an area that meets a predetermined criterion (e.g., an area that is greater than or equal to a predetermined threshold) as an occupied area / unoccupied area. For example, if an area where a visual object is not displayed is below a predetermined threshold, the electronic device may not identify that area as an unoccupied area. For example, if an area where a visual object is displayed is below a predetermined threshold, the electronic device may not identify that area as an occupied area but may include it in the unoccupied area.

[0109] According to one embodiment, an electronic device may display a visual object in a target area corresponding to a display category set for the visual object among a plurality of display categories. The plurality of display categories may be distinguished based on the priority or importance of the visual object. The plurality of display categories may be distinguished to provide visibility to a user according to the priority or importance of the visual object. Alternatively, the plurality of display categories may be distinguished to avoid disturbing the user's concentration on a main task running in a first display area or a second display area. For example, an occupied area of ​​the first display area may be more appropriate for displaying a visual object having a higher priority or importance than an unoccupied area of ​​the first display area, the first display area may be more appropriate for displaying a visual object having a higher priority or importance than the second display area, and the upper half of the second display area may be more appropriate for displaying a visual object having a higher priority or importance than the lower half of the second display area. For example, a visual object closer to the center of the first / second display area may be more appropriate for displaying a visual object having a higher priority or importance.

[0110] According to one embodiment, the electronic device may store, for each visual object, a preset display classification according to the type of the visual object.

[0111] According to one embodiment, the display classification set for a visual object can be changed by the user.

[0112] In one embodiment, each display category may be integrated so that visual objects belonging to (or corresponding to) that display category are displayed in the same target area. For example, all visual objects belonging to the first category and the second category may be displayed in the target area corresponding to the first category. For example, all visual objects belonging to the first category and the second category may be displayed in the target area corresponding to the second category.

[0113] In one embodiment, the target area where each visual object is displayed may be segmented based on the type of application and / or notification associated with the visual object belonging to a display category. For example, among notifications from applications associated with visual objects belonging to a third category, notifications configured by the user to receive may be displayed in a target area that provides greater visibility than advertising notifications.

[0114] According to one embodiment, an electronic device may display a visual object in a target area that includes at least a portion of a non-occupied area of ​​an execution screen of an application provided in at least one of a first display area and a second display area. A method for displaying a visual object in a target area according to a display classification of the visual object (or set for the visual object) is described with reference to FIGS. 8 to 11.

[0115] FIGS. 8 to 11 are drawings explaining a screen display method of an electronic device according to one embodiment.

[0116] FIGS. 8 to 11 are examples of screens displayed through a display (e.g., a display module (160) of FIG. 1, a display (250) of FIGS. 2 to 4, a display (510) of FIG. 5) of an electronic device (e.g., an electronic device (101) of FIG. 1 or an electronic device (200) of FIGS. 2 to 4 and 6), respectively. The electronic device may include at least some of the components of the electronic device (101) or the electronic device (200) described above. For example, the electronic device may include at least one processor including a processing circuit (e.g., a processor (120) of FIG. 1 or a processor (210) of FIGS. 2 to 4), a memory including one or more storage media for storing instructions (e.g., a memory (130) of FIG. 1), and a display.

[0117] A first display area (820) of a display of an electronic device (e.g., the first display area (251) of FIGS. 2, 4, and 6) and / or a second display area (810) (e.g., the second display area (252) of FIGS. 2, 4, and 6) may have an occupied area. The occupied area may represent an area in the first display area (820) and / or the second display area (810) where a visual object (or a visual user interface (UI) element) is displayed.

[0118] When an application's execution screen is provided on an electronic device, the occupied area may represent at least a portion of the first display area (820) and the second display area (810) where visual objects are displayed on the application's execution screen.

[0119] In one embodiment, the occupied area may correspond to the entire screen area where the running screen will be provided on the display.

[0120] According to one embodiment, when information related to the area in which the execution screen is actually provided (e.g., the size of a streaming video, a screen ratio changed in response to a flex mode, etc.) can be obtained from an application that provides a execution screen on an electronic device, the occupied area may represent an area excluding the actual non-occupied area (or, margin area).

[0121] According to one embodiment, when the display classification of the visual object is the first classification, the electronic device may display the visual object in a target area that is at least a portion of an upper half area or a lower half area of ​​the first display area (820). The upper half area of ​​the first display area (820) may represent an area that is far from the second display area (810) when the first display area (820) is divided in half by a line parallel to the hinge axis. The lower half area of ​​the first display area (820) may represent an area that is close to the second display area (810) when the first display area (820) is divided in half by a line parallel to the hinge axis. For example, when the visual object is for an alarm, the electronic device may display a visual object for the alarm in the target area regardless of whether the target area is an occupied or unoccupied area. For example, a visual object for an alarm may be displayed at the top of the upper half and the bottom of the lower half of the first display area (820). For example, a visual object for an alarm may be displayed at the center of the first display area (820) according to a user setting, and this is not limited to the present disclosure. Depending on the size of the occupied area of ​​the first display area (820), a visual object for an alarm may be displayed in an unoccupied area or may be displayed by partially or completely overlapping the occupied area. A visual object belonging to the first category may be displayed in a target area that may provide greater visibility than a second category (e.g., personal message) described below and less visibility than a sixth category (e.g., emergency alert). For example, a visual object may be displayed with different visibility or importance by varying the size and / or shape of the target area depending on the display category. The size of the target area of ​​the first category may be smaller than the sizes of the target areas of other display categories.

[0122] According to an embodiment, referring to FIG. 8, when an output event of a visual object (841) is detected, the electronic device can check information about an occupied area (831, 832) of an application's execution screen. If the display classification of the visual object (841) is the second classification, the electronic device can display the visual object in a target area that is at least a part of a non-occupied area of ​​the first display area (820) (e.g., the first display area (251) of FIG. 6). For example, if the visual object (841) is for a notification that does not require continuous playback or output, the display classification may be the second classification. For example, if the visual object (841) is for a push notification of a personal message, the display classification may be the second classification. The target area of ​​the second classification may be in a horizontally long shape. The user of the electronic device can set a setting to allow or reject push notifications for the message application. Users can allow or deny notifications for different message types for a messaging application, or allow or deny notifications for the messaging application itself on their electronic devices. An application (i.e., the application's server) sends a push notification for a message to the electronic device, and if the application and its corresponding message are allowed to receive push notifications, the electronic device can display a visual object (841) for the message push notification. For example, the electronic device can obtain information from the messenger application indicating whether a new message received is a private message or a group message.

[0123] In one embodiment, referring to FIG. 9, when the display classification of the visual object (842) is the third classification, the electronic device may display the visual object (842) in a target area that is at least a portion of a non-occupied area in the upper half area of ​​the second display area (810) (e.g., the second display area (252) of FIG. 6). The upper half area of ​​the second display area (810) may represent an area close to the first display area (810) when the second display area (810) is divided in half by a line parallel to the hinge axis. For example, when the visual object (842) is for a push notification of an application such as finance, shopping, or entertainment, the display classification may be the third classification. If the visual object (842) is a notification that has been set and accepted by the user according to his / her needs (e.g., a deposit / withdrawal notification in a financial application, a restock notification in a shopping application, a new upload notification in a subscribed channel in an entertainment application, etc.), the display classification of the visual object (842) may be the third classification. On the other hand, any visual object for an advertising notification (e.g., a product advertisement notification in a financial application, a sale advertisement notification in a shopping application, a popular video advertisement notification in an entertainment application, etc.) may be displayed in a target area of ​​another display classification that may provide lower visibility depending on importance or user settings. The display classification according to the type or nature of the application and / or notification is not limited to the present disclosure.

[0124] According to one embodiment, the types of applications and / or notifications of applications associated with any visual object belonging to (or corresponding to) the second and third categories may be preset. A user of the electronic device may change the types of applications and / or notifications of applications belonging to the second and third categories. The types of applications and / or notifications of applications belonging to the second and third categories may vary based on the environment or status of the electronic device.

[0125] According to one embodiment, when the display classification of the visual object (842) is the fourth classification, the electronic device may display the visual object (842) in a target area that is at least a portion of an unoccupied area in the lower half of the second display area (810). The lower half of the second display area (810) may represent an area that is far from the first display area (810) when the second display area (810) is divided in half by a line parallel to the hinge axis. For example, when the visual object (842) is for a group message, the display classification may be the fourth classification. The display classifications of the aforementioned personal messages and group messages may be set under the premise that personal messages may have a higher priority or importance than messages in a group conversation of three or more people. The display classifications of personal messages and group messages may be the same or different depending on the type of message application and / or the user's settings. For example, the electronic device may display a visual object (842) in a target area at the bottom of the occupied area (833) of the second display area (810) if the occupied area (833) is sufficiently small.

[0126] In one embodiment, referring to FIG. 10, when the display classification of the visual object (843) is the fifth classification, the electronic device may display the visual object (843) in a target area, which is at least a portion of the non-occupied area excluding the area where any visual object can be displayed for the third and fourth classifications of the second display area (810). For example, the target area may be limited to the inside of the non-occupied area. For example, when the visual object (843) is for an advertisement, the display classification may be the fifth classification. For example, while the electronic device provides a playback screen of a streaming video on the first display area (820), the electronic device may separately display an advertisement for video viewing on the second display area (810). Therefore, the user may not be interrupted from watching the video.

[0127] According to one embodiment, if the non-occupied area for displaying the visual objects (841 to 843) is insufficient, regardless of the display classification, the electronic device may display the visual objects (841 to 843) in a target area corresponding to another display classification. For example, if the non-occupied area of ​​the first display area (820) for displaying the visual object (841) is insufficient, the electronic device may display the visual object (841) in a target area that is at least a portion of the non-occupied area of ​​the second display area (810).

[0128] Depending on the state of the electronic device or the running application, if a plurality of different non-occupied areas appear in the first display area (820) and / or the second display area (810), a specific non-occupied area among the plurality of non-occupied areas may be determined (or designated) as a target area. For example, if a plurality of different non-occupied areas appear in the first display area (820) and / or the second display area (810), the target area may be determined (or designated) based on the area sizes of the plurality of non-occupied areas. The target area may be determined based on the non-occupied area having the largest area among the plurality of non-occupied areas.

[0129] According to one embodiment, if there is not enough non-occupied area to display the visual objects (841-843), the electronic device may display the visual objects (841-843) in a target area that includes a portion of the occupied area. For example, if there is not enough non-occupied area of ​​the second display area (820) to display the visual objects (842), the electronic device may display the visual objects (842) in a target area that is part of the upper half of the occupied area (833) and the upper half of the non-occupied area.

[0130] According to one embodiment, the electronic device may display visual objects (841 to 843) in a target area including a non-occupied area and at least a portion of an occupied area. For example, while a user setting or a visual object (841 to 843) is displayed in a non-occupied area of ​​the first display area (820) or the second display area (810), a user input (e.g., a touch input for enlarging or moving the visual object (841 to 843)) received may cause the visual object (841 to 843) to be displayed in a target area including an occupied area of ​​the first display area (820) or the second display area (810).

[0131] In one embodiment, referring to FIG. 11, when the display classification of the visual object (844) is the sixth classification, the electronic device may display the visual object (844) in a target area that is at least a part of the occupied area or the non-occupied area of ​​the first display area (820) by overlapping the execution screen of the application. For example, when the visual object (844) is for an emergency alert, the display classification may be the sixth classification. For example, when the display classification of the visual object (844) is the sixth classification, the electronic device may display the visual object (844) in the target area regardless of whether it is the occupied area or the non-occupied area of ​​the first display area (820).

[0132] According to one embodiment, the electronic device may display any visual object in any area of ​​the first display area (820) or the second display area (810) based on the priority or importance of any visual object, regardless of the display classification. The electronic device may vary the target area in which any visual object is displayed based on user settings.

[0133] In one embodiment, an electronic device (101; 200) comprises: a foldable housing (201) comprising a first housing (210) including a first display area (251; 820) on at least one surface, a second housing (220) including a second display area (252; 810) on at least one surface, and a hinge (230) rotatably connected between the first housing (210) and the second housing (220) such that the first display area (251; 820) and the second display area (252; 810) face each other; at least one processor (120); And a memory (130) including one or more storage media for storing instructions, and when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101; 200) causes: while the execution screen of the application is provided in at least one of the first display area (251; 820) and the second display area (252; 810) in a partially folded state, if an output event for a visual object is detected, to display a visual object in a target area including at least a part of a non-occupied area other than an occupied area of ​​the execution screen, and the target area may correspond to a display classification set for the visual object among a plurality of display classifications for the occupied area or the non-occupied area.

[0134] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101; 200) may be caused to: display a visual object in a target area by checking information about the occupied area of ​​the execution screen.

[0135] According to one embodiment, information about the occupied area may include coordinate information about the occupied area of ​​the execution screen.

[0136] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101; 200) may: cause the visual object to be displayed in a target area that is at least a portion of an upper half area or a lower half area of ​​a first display area (251; 820), if the display classification of the visual object is a first classification.

[0137] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101; 200) may: cause the visual object to be displayed in a target area that is at least a portion of a non-occupied area of ​​the first display area (251; 820), if the display classification of the visual object is a second classification.

[0138] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101; 200) may: cause the visual object to be displayed in a target area that is at least a portion of a non-occupied area in the upper half of the second display area (252; 810), if the display classification of the visual object is a third classification.

[0139] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101; 200) may: cause the visual object to be displayed in a target area that is at least a portion of a non-occupied area in the lower half of the second display area (252; 810), if the display classification of the visual object is a fourth classification.

[0140] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101; 200) may cause: when the display classification of the visual object is the fifth classification, to display the visual object in a target area that is at least a part of an unoccupied area excluding an area in which any visual object can be displayed for the third classification and the fourth classification of the second display area (252; 810).

[0141] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120), the electronic device (101; 200) may cause: when the display classification of the visual object is the sixth classification, the visual object to be displayed in a target area that is at least a part of the occupied area or the non-occupied area of ​​the first display area (251; 820) to overlap with the execution screen.

[0142] According to one embodiment, the display classification set for a visual object may be changeable by the user.

[0143] In one embodiment, a display method performed by an electronic device (101; 200) comprises: a foldable housing (201) comprising a first housing (210) including a first display area (251; 820) on at least one surface; a second housing (220) including a second display area (252; 810) on at least one surface; and a hinge (230) rotatably connected between the first housing (210) and the second housing (220) such that the first display area (251; 820) and the second display area (252; 810) face each other; at least one processor (120); And a memory (130) including one or more storage media for storing instructions, and a step of providing an execution screen of an application to at least one of a first display area (251; 820) and a second display area (252; 810) in a state where the foldable housing (201) is partially folded, a step of detecting an output event for a visual object, and a step of displaying a visual object in a target area including at least a part of a non-occupied area other than an occupied area of ​​the execution screen, wherein the target area may correspond to a display classification set for the visual object among a plurality of display classifications for the occupied area or the non-occupied area.

[0144] According to one embodiment, the step of displaying a visual object in a target area that is at least a part of a non-occupied area of ​​the execution screen other than the occupied area may include the step of displaying a visual object in the target area by checking information about the occupied area of ​​the execution screen.

[0145] According to one embodiment, information about the occupied area may include coordinate information about the occupied area of ​​the execution screen.

[0146] According to one embodiment, the display method performed by the electronic device (101; 200) may further include a step of displaying the visual object in a target area that is at least a part of an upper half area or a lower half area of ​​a first display area (251; 820), when the display classification of the visual object is the first classification.

[0147] According to one embodiment, the step of displaying a visual object in a target area that includes at least a portion of a non-occupied area of ​​the execution screen, which is not an occupied area, may include the step of displaying the visual object in a target area that is at least a portion of the non-occupied area of ​​the first display area (251; 820), when the display classification of the visual object is a second classification.

[0148] According to one embodiment, the step of displaying a visual object in a target area that includes at least a portion of a non-occupied area other than an occupied area of ​​the execution screen may include, when the display classification of the visual object is a third classification, displaying the visual object in a target area that is at least a portion of a non-occupied area in an upper half area of ​​the second display area (252; 810).

[0149] According to one embodiment, the step of displaying a visual object in a target area that includes at least a portion of a non-occupied area other than an occupied area of ​​the execution screen may include, when the display classification of the visual object is a fourth classification, displaying the visual object in a target area that is at least a portion of a non-occupied area in a lower half area of ​​the second display area (252; 810).

[0150] According to one embodiment, the step of displaying a visual object in a target area that includes at least a portion of a non-occupied area other than an occupied area of ​​the execution screen may include, when the display classification of the visual object is the fifth classification, displaying the visual object in a target area that is at least a portion of a non-occupied area excluding an area in which any visual object can be displayed for the third classification and the fourth classification of the second display area (252; 810).

[0151] According to one embodiment, the display method performed by the electronic device (101; 200) may further include a step of displaying the visual object in a target area that is at least a part of the occupied area or the non-occupied area of ​​the first display area (251; 820) by overlapping the execution screen, when the display classification of the visual object is the sixth classification.

[0152] According to one embodiment, the display classification set for a visual object may be changeable by a user.

[0153] According to one embodiment, a computer-readable recording medium can store a program for executing a display method performed by an electronic device (101; 200) in combination with a hard disk drive.

[0154] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0155] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0156] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0157] The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0158] Although the embodiments have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the described embodiments. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0159] Therefore, other implementations, embodiments and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In an electronic device (101; 200), A foldable housing (201) comprising a first housing (210) including a first display area (251; 820) on at least one surface, a second housing (220) including a second display area (252; 810) on at least one surface, and a hinge (230) rotatably connected between the first housing (210) and the second housing (220) such that the first display area (251; 820) and the second display area (252; 810) face each other; at least one processor (120); and A memory (130) comprising one or more storage media storing instructions. Including, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 200) causes: While the above foldable housing (201) is partially folded and the execution screen of the application is provided on at least one of the first display area (251; 820) and the second display area (252; 810), if an output event for a visual object is detected, the visual object is displayed on a target area that includes at least a part of a non-occupied area other than the occupied area of ​​the execution screen. The target area corresponds to a display classification set for the visual object among a plurality of display classifications for the occupied area or the non-occupied area. Electronic devices (101; 200).

2. In paragraph 1, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 200) causes: By checking the information about the occupied area of ​​the above execution screen, the visual object is displayed in the target area. Electronic devices (101; 200).

3. In either of paragraphs 1 and 2, The above information on the above occupied area is, Containing coordinate information for the occupied area of ​​the above execution screen, Electronic devices (101; 200).

4. In any one of paragraphs 1 to 3, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 200) causes: If the above display classification of the above visual object is the first classification, To display the visual object in a target area, which is at least a part of the upper half area or the lower half area of ​​the first display area (251;820). Electronic devices (101; 200).

5. In any one of paragraphs 1 to 4, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 200) causes: If the above display classification of the above visual object is the second classification, To display the visual object in a target area that is at least a part of the non-occupied area of ​​the first display area (251; 820). Electronic devices (101; 200).

6. In any one of paragraphs 1 to 5, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 200) causes: If the above display classification of the above visual object is the third classification, To display the visual object in a target area that is at least a part of a non-occupied area in the upper half of the second display area (252; 810). Electronic devices (101; 200).

7. In any one of paragraphs 1 to 6, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 200) causes: If the above display classification of the above visual object is the fourth classification, To display the above visual object in a target area that is at least a part of a non-occupied area in the lower half of the second display area (252; 810). Electronic devices (101; 200).

8. In any one of paragraphs 1 to 7, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 200) causes: If the above display classification of the above visual object is the fifth classification, To cause the above visual object to be displayed in a target area that is at least a part of the non-occupied area excluding the area where any visual object can be displayed for the third and fourth categories of the second display area (252; 810). Electronic devices (101; 200).

9. In any one of paragraphs 1 to 8, When the above instructions are individually or collectively executed by the at least one processor (120), the electronic device (101; 200) causes: If the above display classification of the above visual object is the 6th classification, To display the visual object so as to overlap the execution screen in a target area that is at least a part of the occupied area or non-occupied area of ​​the first display area (251; 820). Electronic devices (101; 200).

10. In any one of paragraphs 1 to 9, The display classification set for the above visual object can be changed by the user. Electronic devices (101; 200).

11. In a display method performed by an electronic device (101; 200), The above electronic device (101; 200) A foldable housing (201) comprising a first housing (210) including a first display area (251; 820) on at least one surface, a second housing (220) including a second display area (252; 810) on at least one surface, and a hinge (230) rotatably connected between the first housing (210) and the second housing (220) such that the first display area (251; 820) and the second display area (252; 810) face each other; at least one processor (120); and A memory (130) comprising one or more storage media storing instructions. Including, A step of providing an application execution screen to at least one of the first display area (251; 820) and the second display area (252; 810) while the foldable housing (201) is partially folded; A step for detecting an output event for a visual object; and A step of displaying the visual object in a target area that includes at least a part of a non-occupied area other than the occupied area of ​​the above execution screen. Including, The target area corresponds to a display classification set for the visual object among a plurality of display classifications for the occupied area or the non-occupied area. How to display.

12. In paragraph 11, The step of displaying the visual object in the target area, which is at least a part of the non-occupied area other than the occupied area of ​​the above execution screen, A step of displaying the visual object in the target area by checking information about the occupied area of ​​the above execution screen. Including, How to display.

13. In either of paragraphs 11 and 12, The above information on the above occupied area is, Containing coordinate information for the occupied area of ​​the above execution screen, How to display.

14. In any one of paragraphs 11 to 13, A step of displaying the visual object in a target area that is at least a part of an upper half area or a lower half area of ​​the first display area (251;820) when the display classification of the visual object is a first classification. Including more, How to display.

15. A computer-readable recording medium storing a program for executing the method of claim 11 in combination with hardware.

Images