Electronic device and control method thereof

Abstract

This electronic device comprises a memory for storing instructions, a display, and at least one processor including processing circuitry, wherein the instructions, when executed individually or collectively by the at least one processor, identify a first partial area in an image, the first partial area having a size corresponding to a window; display the first partial area in the image on the window through the display; identify an expansion direction of the window on the basis of a user input to the window; identify, on the basis of the expansion direction, an area of the image that is newly displayed outside the first partial area; identify a target object included in the newly displayed area of the image; and provide at least one of haptic feedback, sound feedback, and visual feedback on the basis of the target object.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device and a control method thereof that provide feedback based on a user input that changes a displayed image area.

[0002] Electronic devices can display images. The displayed image may be limited to a specific area. Electronic devices may not display the entire image, but rather only a specific area of ​​the image.

[0003] Electronic devices can change the area displayed based on user input. As the area displayed changes, the objects visible to the user can change. This is because the objects displayed change as the area in which the image is displayed changes.

[0004] An electronic device may provide feedback to a user regarding the display of an image. The electronic device may provide feedback to a user regarding changes in an image area.

[0005] The present disclosure is designed to improve the above-described problem, and an object of the present disclosure is to provide an electronic device and a control method thereof that, when a user input for changing a window displaying a portion of an image is received, provides feedback corresponding to a target object identified in an area displayed in a changed window.

[0006] According to one embodiment, an electronic device includes at least one processor including a memory storing instructions, a display, and processing circuitry, wherein the instructions, when individually or collectively executed by the at least one processor, identify a first partial region of an image having a size corresponding to a window among an area of ​​the image, display the first partial region of the image in the window through the display, identify an expansion direction of the window based on a user input for the window, identify a newly displayed region outside the first partial region in the image based on the expansion direction, identify a target object included in the newly displayed region of the image, and provide at least one of haptic feedback, sound feedback, or visual feedback based on the target object.

[0007] The above instructions, when individually or collectively executed by the at least one processor, may receive the user input for expanding the window while the first partial region is displayed in the window of the first size.

[0008] The instructions, when individually or collectively executed by the at least one processor, may, when the user input is received, identify the expansion direction of the window based on the user input, identify a second partial area of ​​the image to be displayed in the window of a second size expanded based on the user input, identify the target object in the newly displayed area excluding the first partial area in the second partial area, and control the display to display the second partial area in the window of the second size.

[0009] The instructions, when individually or collectively executed by the at least one processor, may identify at least one object for which feedback is to be provided based on the image, determine at least one feedback corresponding to the at least one object, identify the target object in the newly displayed area within the second partial area among the at least one object, and provide feedback corresponding to the target object among the determined at least one feedback.

[0010] The instructions, when individually or collectively executed by the at least one processor, identify an anchor region of the at least one object within the image, and when it is confirmed that the anchor region of the object is included in the newly displayed region within the second partial region, identify the object corresponding to the confirmed anchor region as the target object.

[0011] When the instructions are individually or collectively executed by the at least one processor, if a plurality of target objects are identified in the newly displayed area, the instructions identify a first overlapping area in which a first anchor area of ​​the first target object and the newly displayed area overlap, identify a second overlapping area in which a second anchor area of ​​the second target object and the newly displayed area overlap, and compare the first overlapping area and the second overlapping area to identify a final target object.

[0012] The instructions, when individually or collectively executed by the at least one processor, may obtain a difference value between an area of ​​the first overlapping area and an area of ​​the second overlapping area, and if the difference value is greater than or equal to a threshold value, identify an object in a larger area among the first overlapping area and the second overlapping area as the final target object, and provide at least one of the haptic feedback, the sound feedback, or the visual feedback based on the final target object.

[0013] When the above instructions are individually or collectively executed by the at least one processor, the final target object can be identified based on a preset priority if the difference value is less than the threshold value.

[0014] The instructions, when individually or collectively executed by the at least one processor, may obtain the image to be displayed in the window, identify a depth map corresponding to the image, and determine the at least one feedback corresponding to the at least one object based on the image and the depth map.

[0015] The instructions, when individually or collectively executed by the at least one processor, may, upon receiving a user input to minimize the window, identify a currently provided or most recently provided feedback, provide the feedback by gradually reducing the intensity of the feedback, and control the display to display a preset default screen.

[0016] According to one embodiment, a control method of an electronic device includes the steps of: identifying a first partial area of ​​an image having a size corresponding to a window among areas of the image; displaying the first partial area of ​​the image in the window; identifying an expansion direction of the window based on a user input for the window; identifying a newly displayed area outside the first partial area in the image based on the expansion direction; identifying a target object included in the newly displayed area of ​​the image; and providing at least one of haptic feedback, sound feedback, or visual feedback based on the target object.

[0017] The above control method can receive the user input for expanding the window while the first partial area is displayed in the window of the first size.

[0018] The step of identifying the expansion direction may include, when the user input is received, identifying the expansion direction of the window based on the user input, and the control method may include the step of identifying a second partial area of ​​the image to be displayed in the window of a second size expanded based on the user input, the step of identifying the target object in the newly displayed area excluding the first partial area in the second partial area, and the step of displaying the second partial area in the window of the second size.

[0019] The control method includes a step of identifying at least one object to which feedback is to be provided based on the image and a step of determining at least one feedback corresponding to the at least one object, wherein the step of identifying the target object identifies the target object in the newly displayed area within the second partial area among the at least one object, and the step of providing may provide feedback corresponding to the target object among the determined at least one feedback.

[0020] The control method includes a step of identifying an anchor region of the at least one object within the image, and the step of identifying the target object includes, when it is confirmed that the anchor region of the object is included in the newly displayed region within the second partial region, identifying the object corresponding to the confirmed anchor region as the target object.

[0021] The control method may include, when a plurality of target objects are confirmed in the newly displayed area, a step of identifying a first overlapping area in which a first anchor area of ​​a first target object and the newly displayed area overlap, a step of identifying a second overlapping area in which a second anchor area of ​​a second target object and the newly displayed area overlap, and a step of comparing the first overlapping area and the second overlapping area to identify a final target object.

[0022] The control method includes a step of obtaining a difference value between an area of ​​the first overlapping area and an area of ​​the second overlapping area, and a step of identifying an object in a larger area among the first overlapping area and the second overlapping area as the final target object if the difference value is greater than or equal to a threshold value, and the step of providing may provide at least one of the haptic feedback, the sound feedback, or the visual feedback based on the final target object.

[0023] The above control method may include a step of identifying the final target object based on a preset priority if the difference value is less than the threshold value.

[0024] The above control method includes a step of obtaining the image to be displayed in the window and a step of identifying a depth map corresponding to the image, and the step of determining the feedback can determine the at least one feedback corresponding to the at least one object based on the image and the depth map.

[0025] The above control method may include, when a user input for minimizing the window is received, the steps of identifying feedback currently being provided or most recently provided, providing the feedback by gradually reducing the intensity of the feedback, and displaying a preset default screen.

[0026] FIG. 1 is a drawing for explaining an electronic device according to one embodiment.

[0027] FIG. 2 is a block diagram illustrating an electronic device according to one embodiment.

[0028] FIG. 3 is a block diagram illustrating a specific configuration of the electronic device of FIG. 2, according to one embodiment.

[0029] FIG. 4 is a diagram illustrating an operation of providing feedback based on a window, according to one embodiment.

[0030] FIG. 5 is a drawing for explaining an operation of displaying a changed area according to one embodiment.

[0031] FIG. 6 is a diagram illustrating an operation of providing feedback corresponding to a target object according to one embodiment.

[0032] FIG. 7 is a diagram illustrating an operation of identifying a target object in a changed second area according to one embodiment.

[0033] FIG. 8 is a drawing for explaining an operation of displaying a first area according to one embodiment.

[0034] FIG. 9 is a drawing for explaining an operation of displaying a second area according to one embodiment.

[0035] FIG. 10 is a diagram illustrating an operation of providing feedback using an anchor area of ​​an object, according to one embodiment.

[0036] FIG. 11 is a drawing for explaining an overlapping area according to one embodiment.

[0037] FIG. 12 is a drawing for explaining an embodiment of identifying a plurality of target objects according to one embodiment.

[0038] FIG. 13 is a drawing for explaining an embodiment of identifying a plurality of target objects according to one embodiment.

[0039] FIG. 14 is a diagram for explaining an operation of providing feedback using a depth map according to one embodiment.

[0040] FIG. 15 is a drawing for explaining a depth map according to one embodiment.

[0041] FIG. 16 is a diagram illustrating an operation for identifying different depths according to one embodiment.

[0042] FIG. 17 is a drawing for explaining an operation of changing a window two or more times according to one embodiment.

[0043] FIG. 18 is a drawing for explaining an operation of changing a window two or more times according to one embodiment.

[0044] FIG. 19 is a diagram illustrating an operation of providing feedback based on spatial characteristics, according to one embodiment.

[0045] FIG. 20 is a drawing for explaining a depth map according to one embodiment.

[0046] FIG. 21 is a drawing for explaining an operation of dividing space according to one embodiment.

[0047] FIG. 22 is a diagram illustrating a plurality of devices that output feedback according to one embodiment.

[0048] FIG. 23 is a drawing for explaining a window minimization command according to one embodiment.

[0049] FIG. 24 is a drawing for explaining a window maximization command according to one embodiment.

[0050] FIG. 25 is a drawing for explaining commands for minimizing and maximizing a window, according to one embodiment.

[0051] FIG. 26 is a drawing for explaining an operation in which a window expands in all directions according to one embodiment.

[0052] FIG. 27 is a drawing for explaining an operation of enlarging an image while the window is fixed, according to one embodiment.

[0053] FIG. 28 is a diagram for explaining settings related to feedback according to one embodiment.

[0054] FIG. 29 is a diagram for explaining a haptic feedback setting according to one embodiment.

[0055] FIG. 30 is a drawing for explaining sound feedback settings according to one embodiment.

[0056] FIG. 31 is a diagram for explaining a visual feedback setting according to one embodiment.

[0057] FIG. 32 is a drawing for explaining feedback detail settings according to one embodiment.

[0058] FIG. 33 is a diagram illustrating a notification UI for indicating provision of feedback, according to one embodiment.

[0059] FIG. 34 is a diagram illustrating a notification UI for indicating provision of feedback, according to one embodiment.

[0060] FIG. 35 is a diagram illustrating a notification UI for indicating provision of feedback, according to one embodiment.

[0061] FIG. 36 is a diagram illustrating a notification UI for indicating provision of feedback, according to one embodiment.

[0062] FIG. 37 is a drawing for explaining slide variable settings according to one embodiment.

[0063] FIG. 38 is a diagram for explaining feedback according to the start and end times of a change in a window, according to one embodiment.

[0064] FIG. 39 is a drawing for explaining a slide operation according to one embodiment.

[0065] FIG. 40 is a diagram for explaining a feedback table according to one embodiment.

[0066] FIG. 41 is a drawing for explaining feedback corresponding to the expansion direction of a window, according to one embodiment.

[0067] FIG. 42 is a diagram illustrating feedback related to an XR device, according to one embodiment.

[0068] FIG. 43 is a drawing for explaining a method of controlling an electronic device according to one embodiment.

[0069] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

[0070] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0071] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0072] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0073] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0074] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0075] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0076] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor, excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0077] In this specification, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0078] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0079] FIG. 1 is a drawing for explaining an electronic device according to one embodiment.

[0080] Referring to FIG. 1, an electronic device (100) can display an image. The electronic device (100) can display the image through a window (10). The image may be an image selected by a user or an image automatically selected based on a preset control command. The image may be described as content.

[0081] The electronic device (100) may include a display. The window (10) may be displayed on only a portion of the entire screen that the display of the electronic device (100) can display. The electronic device (100) may display multiple images simultaneously. The electronic device (100) may display a first image on the entire screen. While displaying the first image, the electronic device (100) may display a second image through the window (10).

[0082] A window (10) may be a UI for providing an image provided by an application selected by a user. The window (10) may be described as a display UI, a target UI, an output UI, or an application UI.

[0083] The electronic device (100) can execute an application and display an image by dividing (or dividing) the entire screen of the display and using a window (10) corresponding to the application. The electronic device (100) can display an image provided by the application through the window (10).

[0084] For example, the electronic device (100) can display an image including an object (a) through a window (10).

[0085] The electronic device (100) can provide feedback through multiple modalities. Multimodality can refer to the use of at least one of vibration, audio, and images. The electronic device (100) can provide two or more types of feedback, including haptic feedback, sound feedback, and visual feedback.

[0086] A window (10) may include a window UI. The window (10) may include a border. The window UI may include at least one UI item. The border or at least one UI item may not be temporarily displayed.

[0087] A Windows UI may include common UI functions or common UI items of system applications managed by the system of an electronic device.

[0088] For example, a window (10) may represent an image display window. The window (10) may be displayed on the desktop or home screen of a system that displays images.

[0089] For example, Windows (10) may be hidden and not visible on the desktop or home screen.

[0090] The electronic device (100) can provide a window UI corresponding to an application. The window UI corresponding to each application may be composed of different functions or different types of UI items depending on the type of application.

[0091] The electronic device (100) can display an image provided by an application based on the size of the window UI. For example, the window (10) can be displayed on the entire area of ​​the display (140) of the electronic device (100). For example, the window (10) can be displayed on a portion of the display (140) of the electronic device (100).

[0092] The electronic device (100) can change the size of the window (10) based on the window border (including corners or vertices). The electronic device (100) can receive a user input for changing the size (expanding or reducing) of the window (10) through the window border. The window UI can include at least one of a maximize button, a minimize button, and a close button. The electronic device (100) can receive a user input for selecting each button.

[0093] The electronic device (100) may be configured with a slider / foldable type form factor. When the form factor of the electronic device (100) changes, the size of the display area of ​​the display (140) on which the image is displayed may change. When the size of the display area changes, the size of the window (10) may also change. When the form factor changes, the electronic device (100) may identify the direction of change in the display area based on the interaction. The electronic device (100) may change the size of the window (10) based on the identified direction of change in the display area.

[0094] FIG. 2 is a block diagram illustrating an electronic device according to one embodiment.

[0095] An electronic device (100) may include at least one processor (110) including a memory (120) for storing instructions, a display (140), and processing circuitry.

[0096] At least one processor (110) can identify a first portion of an image having a size corresponding to a window among the areas of the image. At least one processor (110) can control a display (140) to display an image (or screen) corresponding to an application through the window.

[0097] At least one processor (110) can display a first partial region of an image in a window via the display (140). The image may be an image provided by an application selected by the user. At least one processor (110) can display only a first partial region of the entire region of the image in the window. The first partial region may be described as a first region.

[0098] At least one processor (110) can determine the direction of expansion of a window based on user input for the window. At least one processor (110) can receive user input for changing the position or size of the window. At least one processor (110) can determine (or identify) the direction of change of the window based on the user input.

[0099] For example, at least one processor (110) can determine the direction in which the window moves.

[0100] For example, at least one processor (110) can determine the extension direction of the window.

[0101] For example, at least one processor (110) can determine the direction of reduction of the window.

[0102] At least one processor (110) can identify a newly displayed area outside of a first partial area in the image based on the direction of expansion of the window. When a user input for expanding the window is received, at least one processor (110) can identify a newly displayed area within the entire area of ​​the image provided by the application.

[0103] At least one processor (110) can identify a target object included in a newly displayed area of ​​an image. At least one processor (110) can identify a newly displayed area. At least one processor (110) can identify a target object among the newly displayed area. The target object may be a preset object. The target object may be changed according to the user's settings.

[0104] Once a target object is identified, at least one processor (110) can provide at least one of haptic feedback, sound feedback, or visual feedback based on the target object.

[0105] A target object may be an object that provides feedback. The target object may represent a specific object that provides feedback to the user. An image may contain multiple objects.

[0106] For example, at least one processor (110) may identify at least one object among a plurality of objects included in an image for which feedback can be provided. This is because feedback may not be provided if the object is not a preset object.

[0107] For example, feedback may be provided for all objects. In such an embodiment, the plurality of objects included in the image and at least one object for which feedback can be provided may be the same.

[0108] At least one processor (110) can identify feedback corresponding to a target object. At least one processor (110) can store a feedback table in memory (120). The feedback table can include information mapping feedback corresponding to objects.

[0109] For example, the feedback table may include a first feedback corresponding to a first object and a second feedback corresponding to a second object. If the target object is identified as the first object, at least one processor (110) may provide the first feedback.

[0110] At least one processor (110) can determine feedback corresponding to at least one object. The feedback provided for each object may be different.

[0111] For example, when a tree object is identified, tree-related feedback (e.g., first-degree haptics, the sound of leaves rustling, visual effects of the tree swaying, etc.) may be provided.

[0112] For example, when a car object is identified, car-related feedback (second-generation haptics, engine sounds, visual effects of the car's wheels spinning, etc.) can be provided.

[0113] For example, when a cloud object is identified, cloud-related feedback (such as haptics of the third century, wind sounds, or visual effects of clouds moving) can be provided.

[0114] For example, if a lake object is identified, lake-related feedback (e.g., haptic vibration at a preset frequency, water sounds, ripple visual effects, etc.) may be provided.

[0115] At least one processor (110) may store a feedback table. The feedback table may include information indicating feedback for each object. The feedback for each object included in the feedback may include predetermined information. At least one processor (110) may determine feedback corresponding to at least one object based on the stored feedback table.

[0116] The feedback table can include various information in addition to object-specific feedback. This information is described in FIGS. 40 and 41. FIG. 40 illustrates a feedback table indicating feedback related to at least one of the device, placement status, whether the device is in a standby state, and a connected device. FIG. 41 illustrates a feedback table indicating feedback corresponding to the direction of a window change.

[0117] At least one processor (110) can determine feedback corresponding to at least one object based on a feedback table. For example, assume that at least one object includes a first object, a second object, a third object, and a fourth object. At least one processor (110) can determine a first feedback corresponding to the first object, a second feedback corresponding to the second object, a third feedback corresponding to the third object, and a fourth feedback corresponding to the fourth object. If the target object is identified as the second object, at least one processor (110) can provide a second feedback corresponding to the second object.

[0118] The feedback may include at least one of haptic feedback, sound feedback, or visual feedback.

[0119] Feedback can represent various forms of output provided to the user. Feedback can include effects output in response to preset events. Effects can be described as 3D effects, AI effects, or sensory effects. Feedback can also be described as response actions, output actions, and more.

[0120] Haptic feedback can be feedback that outputs vibrations. Haptic feedback can be described as tactile feedback, vibration feedback, physical feedback, tactile signals, or vibration effects.

[0121] Sound feedback can be feedback that outputs sound. Sound feedback can be described as acoustic response, auditory feedback, sound feedback, audio feedback, or voice notification feedback.

[0122] Visual feedback can be feedback that produces a visual effect. Visual feedback can be described as visual feedback, screen feedback, graphical feedback, or graphical rendering feedback.

[0123] The electronic device (100) may include a vibration output unit, a speaker, or a display (140). At least one processor (110) may provide haptic feedback through the vibration output unit. At least one processor (110) may provide sound feedback through the speaker. At least one processor (110) may provide visual feedback through the display (140).

[0124] At least one processor (110) may perform an operation of simply outputting an image or outputting audio, which may differ from an operation of outputting feedback. Displaying an image or outputting audio based on user input may be a basic operation for providing content, etc. Feedback may represent an operation provided based on additional user input in addition to the basic operation. Feedback may represent a response operation provided by the electronic device (100) based on user input.

[0125] At least one processor (110) may receive a user input to expand a window while displaying a first partial region in a first-sized window. The first-sized window may be described as a first window or a first-sized window UI.

[0126] When a user input is received, at least one processor (110) can determine the direction of expansion of the window based on the user input. At least one processor (110) can identify a second partial area of ​​the image to be displayed in a second-sized window expanded based on the user input. The second partial area may include the first partial area that is currently displayed and the newly displayed area. The second partial area may be described as a second area. An operation of displaying the second area is described in FIG. 9.

[0127] At least one processor (110) can identify a target object in a newly displayed area excluding the first portion area in the second portion area. The area excluding the first portion area in the second portion area can be described as a third portion area. The newly displayed area can be described as a third portion area, a third area, or a target area.

[0128] At least one processor (110) can control the display (140) to display a second portion of the screen in a second-sized window. The second-sized window can be described as a second window or a second-sized window UI.

[0129] At least one processor (110) can identify at least one object for which to provide feedback based on an image provided by the application.

[0130] At least one processor (110) may determine at least one feedback corresponding to at least one identified object. A feedback mapping table may be used in the feedback determination operation.

[0131] At least one processor (110) can identify a target object in a newly displayed area within a second partial area of ​​at least one object. At least one processor (110) can provide feedback corresponding to the target object from among the determined at least one feedback. The at least one feedback can be multiple feedbacks. At least one processor (110) can identify feedback corresponding to the target object from among the multiple feedbacks. At least one processor (110) can provide the identified feedback.

[0132] At least one processor (110) can identify an anchor region of at least one object within an image. When it is confirmed that an anchor region of an object is included in a newly displayed region within a second partial region, at least one processor (110) can identify an object corresponding to the confirmed anchor region as a target object.

[0133] When multiple target objects are identified in the newly displayed area, at least one processor (110) can identify a first overlapping area where the first anchor area of ​​the first target object and the newly displayed area overlap.

[0134] At least one processor (110) can identify a second overlapping region where the second anchor region of the second target object and the newly displayed region overlap. At least one processor (110) can compare the first overlapping region and the second overlapping region to identify the final target object.

[0135] The anchor area may be an area representing an object. A description thereof is provided in FIGS. 10 and 11.

[0136] At least one processor (110) can obtain a difference value between the area of ​​the first overlapping area and the area of ​​the second overlapping area. If the difference value is greater than or equal to a threshold value, at least one processor (110) can identify an object in a larger area among the first overlapping area and the second overlapping area as a final target object. At least one processor (110) can provide at least one of haptic feedback, sound feedback, or visual feedback based on the final target object.

[0137] If the difference value is less than the threshold value, at least one processor (110) can identify the final target object based on a preset priority.

[0138] The operation of identifying multiple target objects and determining one final target object is described in FIGS. 12 and 13.

[0139] At least one processor (110) can obtain an image to be displayed in a window. At least one processor (110) can identify a depth map corresponding to the image. At least one processor (110) can determine at least one feedback corresponding to at least one object based on the image and the depth map.

[0140] A depth map may include information reflecting spatial characteristics. The spatial characteristics may include at least one of distance, height, area, region, or depth. At least one processor (110) may identify spatial characteristics included in an image based on the depth map. The spatial characteristics may be described as spatial features. At least one processor (110) may obtain (or generate) a depth map reflecting the spatial characteristics.

[0141] At least one processor (110) can provide feedback using a depth map. Once a target object is identified, at least one processor (110) can extract spatial features from the depth map based on the location of the target object. At least one processor (110) can provide feedback corresponding to the extracted spatial features.

[0142] Descriptions related to the depth map are described in FIGS. 14, 15, 16, 19, 20, and 21.

[0143] As an example, an operation of receiving multiple user inputs to change a window is described in FIGS. 17 and 18.

[0144] As an example, an operation of providing feedback through multiple sub-devices is described in FIG. 22.

[0145] When user input to minimize a window is received, at least one processor (110) can identify the currently provided or most recently provided feedback.

[0146] At least one processor (110) can provide feedback by gradually reducing the intensity of the feedback. At least one processor (110) can control the display (140) to display a preset default screen.

[0147] Descriptions related to the minimize and maximize commands of the first window are described in FIGS. 23, 24, and 25.

[0148] As an example, an embodiment in which the expansion of the first window is uniform in the entire direction is described in FIG. 26.

[0149] As an example, an embodiment in which the displayed area is changed (e.g., enlarged) while the size of the first window is fixed is described in FIG. 27.

[0150] Detailed settings related to feedback are described in FIGS. 28, 29, 30, 31, and 32. In FIGS. 28 through 32, the screen is depicted as being displayed on a smartphone-type device. Depending on the embodiment, the screen may be displayed on various types of devices, such as tablets, rather than smartphones.

[0151] The UI for indicating that feedback is provided is described in FIGS. 33, 34, 35, and 36.

[0152] A description of a display (140) capable of sliding operation is described in FIG. 37 and FIG. 39.

[0153] An embodiment of determining feedback by considering the start and end points of a continuous operation that changes the first window is described in FIG. 38.

[0154] An embodiment in which feedback is provided in relation to an XR (Extended Reality) device is described in FIG. 42.

[0155] According to one embodiment, a window of a first size may be described as a first window, and a window of a second size may be described as a second window. At least one processor (110) may receive an image to be displayed through the first window, identify an expansion direction of the first window based on a user input related to the first window, identify a target object included in a newly displayed area (a third partial area, a target area) in the image based on the expansion direction, and provide at least one of haptic feedback, sound feedback, or visual feedback based on the target object.

[0156] The first window may be a window (10) described in another drawing. At least one processor (110) can identify an area (the first area) to be displayed through the first window among the entire area of ​​the image. At least one processor (110) can display the first area through the first window. The operation of displaying the first area is described in FIG. 8.

[0157] At least one processor (110) can receive user input related to the first window. At least one processor (110) can receive user input for changing the first window.

[0158] The user input may include at least one of an input for expanding the size of the first window, an input for resizing the first window, and an input for moving the first window.

[0159] The first window may not display the entire area of ​​the image. At least one processor (110) may change the area displayed through the window based on user input. If the size of the first window changes, the area displayed in the image may change.

[0160] When a user input is received, at least one processor (110) can identify a direction in which the first window changes based on the user input. At least one processor (110) can identify a direction in which the first window changes size based on the user input.

[0161] For example, if the user input is to expand the first window, at least one processor (110) can identify the expansion direction of the first window.

[0162] For example, if the user input is to reduce the first window, at least one processor (110) can identify the reduction direction of the first window.

[0163] For example, if the user input is to move the first window, at least one processor (110) can identify the direction of movement of the first window.

[0164] For example, at least one processor (110) may receive a user input for expanding a first window. The at least one processor (110) may identify an expansion direction based on the user input. The at least one processor (110) may receive a user input for expanding the first window while displaying the first region. The at least one processor (110) may identify a region (third region) to be newly displayed based on the expansion direction among the entire region of the image. The at least one processor (110) may identify the third region based on the first region currently being displayed and the expansion direction.

[0165] At least one processor (110) can identify a second region based on the extended window. At least one processor (110) can identify a third region, excluding the first region, in the second region.

[0166] At least one processor (110) can identify a target object for a newly displayed area (third area).

[0167] At least one processor (110) can identify a first region to be displayed through a first window among the entire region of the image. At least one processor (110) can receive a user input for expanding the first window while displaying the first region through the first window.

[0168] At least one processor (110) may, when a user input is received, identify an expansion direction of a first window based on the user input. At least one processor (110) may identify a second area to be displayed through a second window expanded based on the user input. At least one processor (110) may identify a target object in the second area. At least one processor (110) may control the display (140) to display the second area through the second window.

[0169] At least one processor (110) can identify at least one object for which feedback is to be provided based on the image. A target object can be identified in a second region among the at least one object, and feedback corresponding to the target object can be provided among the determined feedback.

[0170] At least one processor (110) can identify an anchor region of at least one object, and when the anchor region of the object is identified in a second region, the object corresponding to the identified anchor region in the second region can be identified as a target object.

[0171] At least one processor (110) can identify a first overlapping region where a first anchor region and a second region of a first target object overlap, when multiple target objects are identified. At least one processor (110) can identify a second overlapping region where a second anchor region and a second region of a second target object overlap. At least one processor (110) can compare the first overlapping region and the second overlapping region to identify a final target object.

[0172] At least one processor (110) can obtain a difference value between the area of ​​the first overlapping area (first area) and the area of ​​the second overlapping area (second area). If the difference value is greater than or equal to a threshold value, an object in a larger area between the first overlapping area and the second overlapping area can be identified as the final target object. The area can be described as a size or area, etc. At least one processor (110) can identify an object in a larger area between the first area and the second area as the final target object.

[0173] At least one processor (110) can provide at least one of haptic feedback, sound feedback or visual feedback based on the final target object.

[0174] At least one processor (110) can identify the final target object based on a preset priority if the difference value is less than a threshold value.

[0175] At least one processor (110) can identify a depth map corresponding to an image received through a first window. At least one processor (110) can determine feedback corresponding to at least one object based on the image and the depth map.

[0176] At least one processor (110) can identify the currently provided feedback upon receiving a user input to minimize the first window. The at least one processor (110) can provide the feedback by gradually reducing the intensity of the feedback. The at least one processor (110) can control the display (140) to display a preset default screen.

[0177] FIG. 3 is a block diagram illustrating a specific configuration of the electronic device of FIG. 2, according to one embodiment.

[0178] FIG. 3 is a block diagram of an exemplary electronic device (100) capable of performing the operations described in this document.

[0179] Referring to FIG. 3, the electronic device (100) may be one of various forms of electronic devices, such as a notebook (190), smartphones having various form factors (e.g., bar-type smartphones, foldable-type smartphones, or slideable (or rollable) type smartphones), tablets, cellular phones, and other similar computing devices. The components, their relationships, and their functions illustrated in FIG. 1 are merely exemplary and do not limit the implementations described or claimed in this document. The electronic device (100) may be referred to as a mobile device, a user device, a multi-function device, a portable device, or a server.

[0180] The electronic device (100) may include components including at least one processor (110) (hereinafter referred to as processor (110)), at least one memory (120) (hereinafter referred to as memory (120)), at least one display (140) (hereinafter referred to as display (140)), at least one image sensor (150) (hereinafter referred to as image sensor (150)), at least one communication circuit (160) (hereinafter referred to as communication circuit (160)), and / or at least one sensor (170) (hereinafter referred to as sensor (170)). The above components are merely exemplary. For example, the electronic device (100) may include other components (e.g., power management integrated circuitry (PMIC), audio processing circuitry, an antenna, a rechargeable battery, or an input / output interface). For example, some components may be omitted from the electronic device (100). For example, some components may be integrated into one component.

[0181] The processor (110) may be implemented as one or more IC (integrated circuit (or circuitry)) chips and may perform various data processing. The processor (110) may include at least one electrical circuit and may individually or collectively perform distributed processing of instructions (or programs, data, etc.) stored in the memory (120). The processor (110) may include a processor assembly including one or more processing circuits. The processor (110) may include any processing circuit operative to control the performance and operations of one or more components (e.g., the memory (120), the display (140), the image sensor (150), the communication circuit (160), and / or the sensor (170)) of the electronic device (100). For example, the processor (110) (e.g., the application processor (AP)) may be implemented as a system on chip (SoC) (e.g., a single chip or a chipset). For example, the processor (110) may be implemented with multiple cores (or at least one core circuit), multiple chips, or multiple chipsets. For example, the processor (110) may include one or more processing circuits. For example, the processor (110) may include one or more processing circuits configured to individually and / or collectively perform various functions of the present disclosure. As a non-limiting example, at least a portion of the processor (110) may be included in a first chip of the electronic device (100), and at least another portion of the processor (110) may be included in a second chip of the electronic device (100) that is different from the first chip of the electronic device (100).

[0182] For example, the processor (110) may include a central processing unit (CPU) (111), a graphics processing unit (GPU) (112), a neural processing unit (NPU) (113), an image signal processor (ISP) (114), a display controller (115), a memory controller (116), a storage controller (117), a communication processor (CP) (118), and / or a sensor interface (119). These components of the processor (110) are merely exemplary. For example, the processor (110) may further include other components. For example, some components of the processor (110) may be omitted from the processor (110). For example, some components of the processor (110) may be included as separate components of the electronic device (100) outside the processor (110). For example, some components of the processor (110) (e.g., memory controller (116)) may be included within other components (e.g., at least a portion of memory (120), an interface (e.g., available for connection to at least one component of the electronic device (100)), a display (140) and / or an image sensor (150)).

[0183] The processor (110) may cause other components of the electronic device (100) to perform various operations by executing instructions stored in the memory (120). The CPU (111) (or central processing circuit) may be configured to control components of the processor (110) based on the execution of instructions stored in the memory (120) (e.g., volatile memory (121) and / or non-volatile memory (122)). The GPU (112) (or graphics processing circuit) may be configured to execute parallel operations (e.g., rendering). The NPU (113) (or neural processing circuit, or artificial intelligence (AI) chip) may be configured to execute operations for an artificial intelligence model (e.g., convolution computation). The ISP (114) (or image signal processing circuit) may be configured to process a raw image acquired through the image sensor (150) into a format suitable for a component within the electronic device (100) or a component of the processor (110). The display controller (115) (or display control circuit, or display processing unit (DPU)) may be configured to process an image acquired from the CPU (111), the GPU (112), the ISP (114), or the memory (120) (e.g., the volatile memory (121)) into a format suitable for the display (140). The memory controller (116) (or memory control circuit) may be configured to control reading data from the volatile memory (121) and writing data to the volatile memory (121). The storage controller (117) (or storage control circuit) may be configured to control reading data from the nonvolatile memory (122) and writing data to the nonvolatile memory (122).The CP (118) (communication processing circuit) may be configured to process data acquired from a component of the processor (110) into a format suitable for transmission to another electronic device via the communication circuit (160), or to process data acquired from another electronic device via the communication circuit (160) into a format suitable for processing by the component of the processor (110). For example, the communication circuit (160) may include one or more communication circuits. The sensor interface (119) (or sensing data processing circuit, sensor hub) may be configured to process data about the state of the electronic device (100) and / or the state of the surroundings of the electronic device (100), acquired via the sensor (170), into a format suitable for the component of the processor (110).

[0184] The memory (120) may include one or more storage media (or one or more storage devices). For example, the memory (120) may include a memory assembly including one or more storage media. For example, the one or more storage media may include permanent memory (e.g., non-volatile memory (122)) such as a hard drive, flash memory, read-only memory (ROM), semi-permanent memory (e.g., volatile memory (121)) such as random access memory (RAM), any other suitable type of storage (or storage assembly), or any combination thereof. The memory (120) may include cache memory, which is one or more different types of memory used to temporarily store data for a function or feature of the electronic device (100). As a non-limiting example, the cache memory may be included within the processor (110). The memory (120) may be fixedly embedded within the electronic device (100) or incorporated into one or more suitable types of components (e.g., a subscriber identity module (SIM) card and / or a secure digital (SD) card) that may be repeatedly inserted into and removed from the electronic device (100).

[0185] For example, the memory (120) may store one or more software applications, such as an operating system (or system) software application, a firmware software application, a driver software application, a plug-in (e.g., add-in, add-on, and / or applet) software application, and / or any other suitable software applications. For example, the one or more software applications may include instructions executable by the processor (110). For example, the memory (120) may store instructions callable by an application programming interface (API). For example, the memory (120) may store instructions within a library.

[0186] FIG. 4 is a diagram illustrating an operation of providing feedback based on a window, according to one embodiment.

[0187] Referring to FIG. 4, the electronic device (100) can analyze an image to be displayed on a window (10) (S410). The electronic device (100) can acquire an image to be displayed through the window (10). The electronic device (100) can analyze the acquired image. The electronic device (100) can determine feedback corresponding to the image. The electronic device (100) can determine the feedback based on the image analysis results.

[0188] The electronic device (100) can receive a user input for changing a window (10) (S420). The user input can include at least one of an input for changing the size of the window (10) and an input for changing the position of the window (10).

[0189] The electronic device (100) can provide feedback based on the changed window (20) (S430). The electronic device (100) can display an image through the changed window (20). The electronic device (100) can provide feedback related to the changed window (20) while displaying an image through the changed window (20).

[0190] For example, the electronic device (100) can obtain information related to the content of an executed application. The information related to the content may include at least one of metadata, image data, and audio data. The electronic device (100) can identify the category of the content (or image) based on the information related to the content. The electronic device (100) can determine feedback based on the category. When a user input for changing a window (10) is received, the electronic device (100) can provide feedback corresponding to the category through the changed window (20).

[0191] FIG. 5 is a drawing for explaining an operation of displaying a changed area according to one embodiment.

[0192] Referring to FIG. 5, the electronic device (100) can obtain an image to be displayed on the window (10) (S505). The electronic device (100) can obtain an image to be displayed on the window (10) based on a user command or a preset command.

[0193] The electronic device (100) can identify a first region (1) to be displayed on a window (10) among the entire region of the image (S520). The entire region included in the image may not be displayed through the window (10). The embodiment of FIG. 5 may represent an operation of displaying only a portion of the entire region of the image.

[0194] The electronic device (100) can display a first area (1) through a window (10) (S525). The first area (1) may be a portion of the entire area of ​​the image. The first area (1) may be an area currently being displayed through the window (10). The first area (1) may be described as an image corresponding to the first area.

[0195] The electronic device (100) can determine whether a user input for changing the window (10) is received (S530). For example, the electronic device (100) can receive a user input for expanding the window (10).

[0196] When a user input for changing a window (10) is received (S530-Y), the electronic device (100) can identify a second area (2) to be displayed in the changed window (20) among the entire area of ​​the image based on the user input (S535). For example, the second area (2) may be an area larger than the first area (1). The second area (2) may be an area expanded from the first area (1) based on the user input. The second area (2) may be described as an image corresponding to the second area.

[0197] The electronic device (100) can display the second area (2) through the changed window (20) (S555). The electronic device (100) can identify the changed window (20) based on a user input. The electronic device (100) can identify the second area (2) based on the user input. The electronic device (100) can display the second area (2) in the changed window (20).

[0198] FIG. 6 is a diagram illustrating an operation of providing feedback corresponding to a target object according to one embodiment.

[0199] Steps S605 and S630 of Fig. 6 may correspond to steps S505 and S530 of Fig. 5. Duplicate explanation is omitted.

[0200] The electronic device (100) can identify at least one object included in an image (S610).

[0201] At least one object may represent an object capable of providing feedback. Multiple objects may be identified in the image. The electronic device (100) may select objects capable of providing feedback from among the identified objects. This is because feedback may not be provided for all objects.

[0202] For example, all objects and objects providing feedback may be represented separately. All identified objects may be described as general objects. Objects capable of providing feedback may be described as candidate objects or feedback objects. At least one object indicated in step S610 may represent an object for which feedback can be provided.

[0203] The electronic device (100) can identify objects for which feedback can be provided based on pre-stored library information. The pre-stored library information may include information related to objects for which feedback can be provided. The library information is described in the UI (3230) of FIG. 32. The library information may be pre-generated information. The library information may be updated based on pre-set events (or cycles).

[0204] The electronic device (100) can determine feedback corresponding to at least one object (S615).

[0205] The electronic device (100) can determine whether a user input for changing the window (10) has been received (S630).

[0206] When a user input for changing the window (10) is received (S630-Y), the electronic device (100) can identify the direction in which the size of the window (10) is changed (S640).

[0207] The electronic device (100) can identify a target object corresponding to the identified direction among at least one object (S645).

[0208] The electronic device (100) can provide feedback corresponding to the target object (S650). The electronic device (100) can provide feedback corresponding to the target object among at least one of the feedbacks determined in step S615.

[0209] For example, the electronic device (100) can identify multiple objects in an image. The electronic device (100) can analyze each of the multiple objects. The electronic device (100) can determine feedback corresponding to object characteristics based on the analysis results. The electronic device (100) can determine feedback corresponding to each of the multiple objects. The electronic device (100) can determine multiple feedbacks related to the multiple objects. The electronic device (100) can obtain a feedback mapping table in which objects and feedbacks are mapped. When a target object is identified, the electronic device (100) can identify feedback (or target feedback) corresponding to the target object based on the feedback mapping table. The electronic device (100) can provide feedback corresponding to the target object.

[0210] FIG. 7 is a diagram illustrating an operation of identifying a target object in a changed second area according to one embodiment.

[0211] Steps S705, S720, S725, S730, S735, and S755 of FIG. 7 may correspond to steps S505, S520, S525, S530, S535, and S555 of FIG. 5. Steps S705, S710, S715, S730, S740, S745, and S750 of FIG. 7 may correspond to steps S605, S610, S615, S630, S640, S645, and S650 of FIG. 6. Duplicate explanations are omitted.

[0212] An electronic device (100) can obtain an image to be displayed on a window (10) (S705). The electronic device (100) can identify at least one object included in the image (S710). The electronic device (100) can determine feedback corresponding to the at least one object (S715). The electronic device (100) can identify a first area (1) to be displayed on the window (10) among the entire area of ​​the image (S720). The electronic device (100) can display the first area (1) through the window (10) (S725).

[0213] The electronic device (100) can determine whether a user input for changing the window (10) is received (S730). If a user input for changing the window (10) is received (S730-Y), the electronic device (100) can identify a second area (2) to be displayed in the changed window (20) among the entire area of ​​the image based on the user input (S735).

[0214] The electronic device (100) can identify the direction in which the size of the window (10) changes (S740). The electronic device (100) can identify a target object corresponding to the identified direction among the objects included in the second area (2) (S745). The electronic device (100) can provide feedback corresponding to the target object (S750). The electronic device (100) can display the second area (2) through the changed window (20) (S755).

[0215] FIG. 8 is a drawing for explaining an operation of displaying a first area according to one embodiment.

[0216] Referring to FIG. 8, an electronic device (100) can obtain an image (800). The image (800) can include a plurality of objects (a, b, c, d). The electronic device (100) can analyze the image (800) to identify the plurality of objects (a, b, c, d).

[0217] The electronic device (100) can identify a first area (1) to be displayed through a window (10) among the entire area of ​​the image (800). The first area (1) can include an object (a).

[0218] The electronic device (100) can control the display (140) to display a first area (1) including an object (a) through a window (10).

[0219] FIG. 9 is a drawing for explaining an operation of displaying a second area according to one embodiment.

[0220] Referring to FIG. 9, the electronic device (100) can obtain an image (900). The image (900) may correspond to the image (800) of FIG. 8. The electronic device (100) can analyze the image (900) to identify multiple objects (a, b, c, d).

[0221] An electronic device (100) can receive a user input for changing a window (10). The electronic device (100) can identify a window (20) based on the user input. The electronic device (100) can identify a second area (2) to be displayed in the changed window (20). The second area (2) can include an object (a) and an object (b).

[0222] The electronic device (100) can control the display (140) to display a second area (2) including an object (a) and an object (b) through a changed window (20).

[0223] The electronic device (100) can identify a second area (2) based on a user input, and identify a new object (b) other than a previously identified object (a) in the second area (2). When the new object (b) is identified, the electronic device (100) can identify the identified object (b) as a target object. The electronic device (100) can provide feedback corresponding to the identified target object.

[0224] FIG. 10 is a diagram illustrating an operation of providing feedback using an anchor area of ​​an object, according to one embodiment.

[0225] Steps S1005, S1010, S1030, S1035, S1050, and S1055 of FIG. 10 may correspond to steps S705, S710, S730, S735, S750, and S755 of FIG. 7. Duplicate explanations are omitted.

[0226] An electronic device (100) can obtain an image to be displayed on a window (10) (S1005). The electronic device (100) can identify at least one object included in the image (S1010).

[0227] When at least one object included in the image is identified, the electronic device (100) can identify an anchor area (or anchor box) of the at least one object (S1011).

[0228] The electronic device (100) can determine whether to receive a user input for changing the window (10) (S1030). Based on the user input, the electronic device (100) can identify a second area (2) to be displayed in the changed window (20) among the entire area of ​​the image (S1035).

[0229] The electronic device (100) can determine whether the anchor region of the target object is identified in the second region (2) (S1046). If the anchor region of the target object is not identified in the second region (2) (S1046-N), the electronic device (100) can repeat steps S1030, S1035, and S1046.

[0230] When the anchor area of ​​the target object is identified in the second area (2) (S1046-Y), the electronic device (100) can provide feedback corresponding to the target object (S1050). The electronic device (100) can display the second area (2) through the changed window (20) (S1055).

[0231] For example, the electronic device (100) can expand a window (10). The electronic device (100) can identify the location of the border of the expanded window (10). If the location of the border of the expanded window (10) is included in the anchor area of ​​the target object, the electronic device (100) can provide feedback corresponding to the target object.

[0232] FIG. 11 is a drawing for explaining an overlapping area according to one embodiment.

[0233] Referring to FIG. 11, an electronic device (100) can obtain an image (1100). The image (1100) may correspond to the image (900) of FIG. 9. The image (1100) may include a plurality of objects (a, b, c, d). The electronic device (100) can identify anchor areas (1101, 1102, 1103, 1104) corresponding to each of the plurality of objects.

[0234] The electronic device (100) can identify whether the changed window (20) overlaps the anchor area. The electronic device (100) can identify the overlapping area (1110) of the area where the changed window (20) is displayed and the anchor area (1102) of the object (b).

[0235] Once the overlapping area (1110) is identified, the electronic device (100) can identify the object (b) as a target object. The electronic device (100) can provide feedback corresponding to the target object.

[0236] For example, if the overlapping area is greater than a threshold value, the electronic device (100) can determine an object corresponding to the overlapping area as a target object.

[0237] For example, the electronic device (100) may receive metadata and image data of an initially executed image. The electronic device (100) may determine a category of the image based on the metadata and image data. The electronic device (100) may identify at least one object based on the category. The electronic device (100) may identify an anchor area of ​​at least one object. The electronic device (100) may compare the anchor area with a corner (or edge) constituting the window (10).

[0238] For example, multiple anchor areas or anchor boxes may exist corresponding to a single object. Multiple anchor areas or anchor boxes may exist corresponding to a first object. The electronic device (100) may identify an anchor area that contacts a corner of the window (10) among the multiple anchor areas. The electronic device (100) may determine a target object based on the identified anchor area.

[0239] FIG. 12 is a drawing for explaining an embodiment of identifying a plurality of target objects according to one embodiment.

[0240] Referring to FIG. 12, the electronic device (100) can determine whether to identify multiple target objects (S1205).

[0241] When multiple target objects are identified (S1205-Y), the electronic device (100) can identify a first anchor area of ​​a first target object and a second anchor area of ​​a second target object (S1210).

[0242] The electronic device (100) can identify a first overlapping area of ​​the first anchor area and the second area (2) (S1215).

[0243] The electronic device (100) can identify a second overlapping area of ​​the second anchor area and the second area (2) (S1220).

[0244] The electronic device (100) can obtain a difference value in area (or size) between the first overlapping area and the second overlapping area (S1225). The difference value can be described as a difference area value or a difference size value.

[0245] The electronic device (100) can identify whether the difference value is greater than or equal to a threshold value (S1230).

[0246] If the difference value is greater than or equal to the threshold value (S1230-Y), the electronic device (100) can determine one final target object based on the size of the overlapping area (S1235).

[0247] The electronic device (100) can identify an overlapping area having a larger area among the first overlapping area and the second overlapping area. The electronic device (100) can determine an object having an overlapping area having a larger area as the final target object.

[0248] For example, if the first overlapping area is larger than the second overlapping area, the electronic device (100) may determine the first target object as the final target object.

[0249] For example, if the second overlapping area is larger than the first overlapping area, the electronic device (100) may determine the second target object as the final target object.

[0250] If the difference value is less than the threshold value (S1230-N), the electronic device (100) can determine one final target object based on the preset priority (S1240).

[0251] The preset priority may be a predetermined priority based on the characteristics of the object (or the category of the object). If the difference between the overlapping areas is less than a threshold value, the electronic device (100) may compare the first characteristic of the first target object with the second characteristic of the second target object to determine the priorities. The electronic device (100) may store the priorities among multiple characteristics in advance. The electronic device (100) may determine the final target object based on the priorities among the multiple characteristics stored in advance.

[0252] The electronic device (100) can provide feedback corresponding to the final target object (S1245).

[0253] For example, if the difference value is greater than or equal to the threshold value according to step S1230, the electronic device (100) may perform an operation of comparing the sizes of the overlapping areas to determine one final target object. The electronic device (100) may not compare the sizes of the overlapping areas for objects with preset priorities. The electronic device (100) may directly determine the object with the preset priorities as the final target object.

[0254] FIG. 13 is a drawing for explaining an embodiment of identifying a plurality of target objects according to one embodiment.

[0255] Referring to FIG. 13, an electronic device (100) can obtain an image (1300). The image (1300) may correspond to the image (900) of FIG. 9. The image (1300) may include a plurality of objects (a, b, c, d). The electronic device (100) can identify anchor areas (1301, 1302, 1303, 1304) corresponding to each of the plurality of objects.

[0256] The electronic device (100) can identify a first overlapping area (1310) of the first anchor area (1302) and the second area (2). The electronic device (100) can identify a second overlapping area (1320) of the second anchor area (1304) and the second area (2).

[0257] The electronic device (100) can obtain a difference value in area (or size) between the first overlapping area (1310) and the second overlapping area (1320) (S1225). The difference value can be described as a difference area value or a difference size value.

[0258] The electronic device (100) can identify whether the difference value is greater than or equal to a threshold value. If the difference value is greater than or equal to the threshold value, the electronic device (100) can determine one final target object based on the size of the overlapping area.

[0259] The electronic device (100) can identify an overlapping area having a larger area among the first overlapping area (1310) and the second overlapping area (1320). The electronic device (100) can determine an object having an overlapping area having a larger area as the final target object.

[0260] For example, if the first overlapping area (1310) is larger than the second overlapping area (1320), the electronic device (100) can determine the first target object (b) as the final target object.

[0261] For example, if the second overlapping area (1320) is larger than the first overlapping area (1310), the electronic device (100) can determine the second target object (d) as the final target object.

[0262] FIG. 14 is a diagram for explaining an operation of providing feedback using a depth map according to one embodiment.

[0263] S1405 of Fig. 14 may correspond to S605 of Fig. 6. Duplicate explanation is omitted.

[0264] The electronic device (100) can obtain a depth map based on an image (S1406). The depth map can represent a map representing three-dimensional depth. The depth map can be described as a depth image or depth information.

[0265] The electronic device (100) can obtain the highest depth and the lowest depth from the depth map. The electronic device (100) can obtain the difference value (or depth difference) between the highest depth and the lowest depth (S1415).

[0266] The electronic device (100) can identify whether the difference value is greater than or equal to a threshold value (S1420). If the difference value is greater than or equal to the threshold value (S1420-Y), the electronic device (100) can provide feedback based on the depth map (S1450).

[0267] For example, the electronic device (100) can identify a target object. Once the target object is identified, the electronic device (100) can identify a depth corresponding to the target object based on a depth map. The electronic device (100) can provide feedback corresponding to the target object based on the depth. The electronic device (100) can identify the location of the target object in an image. The electronic device (100) can identify a depth corresponding to the location of the target object in a depth map. The image and the depth map can use the same coordinate information.

[0268] For example, the electronic device (100) can display a first region (1) through a window (10). The electronic device (100) can obtain a first average depth corresponding to the first region (1) based on a depth map. The electronic device (100) can provide feedback using the first average depth corresponding to the first region (1).

[0269] For example, the electronic device (100) can display the second area (2) through a modified window (20). The electronic device (100) can obtain a second average depth corresponding to the second area (2) based on the depth map. The electronic device (100) can provide feedback using the second average depth corresponding to the second area (2).

[0270] FIG. 15 is a drawing for explaining a depth map according to one embodiment.

[0271] Referring to FIG. 15, an electronic device (100) may receive an image (1510). The image (1510) may include an object representing information related to three-dimensional depth.

[0272] The electronic device (100) can obtain a depth map (1520) based on an image (1510). The depth map (1520) can include information indicating a three-dimensional depth of an object included in the image (1510).

[0273] FIG. 16 is a diagram illustrating an operation for identifying different depths according to one embodiment.

[0274] Referring to FIG. 16, the electronic device (100) can identify a first area (1) to be displayed in a window (10) in an image (1610). The electronic device (100) can identify a depth (1 m) corresponding to the first area (1) based on a depth map. The electronic device (100) can identify a depth (1 m) of a target object (building) included in the first area (1) based on the depth map.

[0275] The electronic device (100) can identify a second area (2) to be displayed in a changed window (20) in the image (1620). The electronic device (100) can identify a depth (30 m) corresponding to the second area (2) based on a depth map. The electronic device (100) can identify a depth (30 m) of a target object (building) included in the second area (2) based on the depth map.

[0276] For example, the electronic device (100) may provide sound feedback corresponding to a building object at a first intensity while displaying a first area (1) of an image (1610). When displaying a second area (2) according to a user input, the electronic device (100) may provide sound feedback corresponding to the building object at a second intensity. The first intensity may be different from the second intensity. The electronic device (100) may change the intensity of the feedback as the depth value increases. Increasing or decreasing the intensity of the feedback may vary depending on the object and the feedback corresponding to the object.

[0277] When the sound feedback corresponding to the building object includes a car sound, the electronic device (100) can provide a smaller intensity of the sound feedback as the depth value (height value) increases.

[0278] When the sound feedback corresponding to the building object includes wind noise, the electronic device (100) can provide a greater intensity of the sound feedback as the depth value (height value) increases.

[0279] FIG. 17 is a drawing for explaining an operation of changing a window two or more times according to one embodiment.

[0280] Steps S1730, S1735, S1745, S1750, and S1755 of FIG. 17 may correspond to steps S730, S735, S745, S750, and S755 of FIG. 7. Duplicate explanations are omitted.

[0281] The electronic device (100) can determine whether to receive a first user input for changing the window (10) while displaying the first area (1) through the window (10) (S1730).

[0282] When a first user input for changing a window (10) is received (S1730-Y), the electronic device (100) can identify a second area to be displayed in the changed window (20) among the entire area of ​​the image based on the first user input (S1735). The electronic device (100) can identify the direction in which the size of the window (10) changes based on the user input.

[0283] The electronic device (100) can identify a first target object corresponding to the identified direction among the objects included in the second area (2) (S1745).

[0284] The electronic device (100) can provide feedback corresponding to the first target object (S1750).

[0285] The electronic device (100) can display the second area (2) through the changed window (20) (S1755).

[0286] The electronic device (100) displaying the second area (2) can determine whether to receive a second user input for changing the window (10) (S1760).

[0287] When a second user input for changing a window (10) is received (S1760-Y), the electronic device (100) can identify a third area (3) to be displayed in the changed window (30) among the entire area of ​​the image based on the second user input (S1765). The electronic device (100) can identify the direction in which the size of the changed window (20) changes based on the user input.

[0288] The electronic device (100) can identify a second target object corresponding to the identified direction among the objects included in the third area (3) (S1770).

[0289] The electronic device (100) can provide feedback corresponding to the second target object (S1775).

[0290] The electronic device (100) can display the third area (3) through the modified window (30) (S1780).

[0291] FIG. 18 is a drawing for explaining an operation of changing a window two or more times according to one embodiment.

[0292] Referring to FIG. 18, an electronic device (100) may obtain an image (1800). The image (1800) may include a plurality of objects (a, b, c). The electronic device (100) may determine feedback corresponding to each of the plurality of objects (a, b, c).

[0293] The electronic device (100) can identify a first area (1) to be displayed on a window (10). The first area (1) can include an object (a). The electronic device (100) can display the first area (1) including the object (a) through the window (10). The electronic device (100) can provide feedback corresponding to the object (a).

[0294] An electronic device (100) displaying a first area (1) can receive a first user input that changes a window (10).

[0295] The electronic device (100) can identify a second area (2) to be displayed in a changed window (20). The second area (2) can include an object (a) and an object (b). The electronic device (100) can display the second area (2) including the object (a) and the object (b) through the changed window (20). The electronic device (100) can identify the newly identified object (b) as a first target object. The electronic device (100) can provide feedback corresponding to the first target object (b). When providing feedback corresponding to the first target object (b), the electronic device (100) can gradually reduce (fade out) the intensity of the feedback corresponding to the object (a).

[0296] The electronic device (100) displaying the second area (2) can receive a second user input that changes the changed window (20).

[0297] The electronic device (100) can identify a third area (3) to be displayed in a changed, re-modified window (30). The third area (3) can include an object (a), an object (b), and an object (c). The electronic device (100) can display the third area (3) including the object (a), the object (b), and the object (c) through the changed, re-modified window (30). The electronic device (100) can identify the newly identified object (c) as a second target object. The electronic device (100) can provide feedback corresponding to the second target object (c). When providing feedback corresponding to the second target object (c), the intensity of the feedback corresponding to the first target object (b) can be gradually reduced (faded out).

[0298] FIG. 19 is a diagram illustrating an operation of providing feedback based on spatial characteristics, according to one embodiment.

[0299] Steps S1905, S1910, S1915, S1930, S1940, S1945, and S1950 of FIG. 19 may correspond to steps S605, S610, S615, S630, S640, S645, and S650 of FIG. 6. Steps S1905 and S1906 of FIG. 19 may correspond to steps S1405 and S1406 of FIG. 14. Duplicate explanations are omitted.

[0300] The electronic device (100) can obtain an image to be displayed on a window (10) (S1905). The electronic device (100) can obtain a depth map based on the image (S1906).

[0301] The electronic device (100) can identify at least one object including spatial characteristics based on a depth map (S1910). The electronic device (100) can determine feedback corresponding to the at least one object (S1915).

[0302] The electronic device (100) can identify whether a user input for changing a window (10) has been received (S1930). If a user input for changing a window (10) has been received (S1930-Y), the electronic device (100) can identify a direction in which the size of the window (10) changes (S1940). The electronic device (100) can identify a target object corresponding to the identified direction among at least one object (S1945).

[0303] The electronic device (100) can provide feedback corresponding to a target object based on spatial characteristics (S1950).

[0304] FIG. 20 is a drawing for explaining a depth map according to one embodiment.

[0305] Referring to FIG. 20, an electronic device (100) may receive an image (2010). The image (2010) may include an object representing information related to three-dimensional depth.

[0306] The electronic device (100) can obtain a depth map (2020) based on the image (2010). The depth map (2020) can include information indicating a three-dimensional depth of an object included in the image (2010).

[0307] The electronic device (100) can acquire spatial characteristics through a depth map (2020) and determine at least one of the sound effect of the feedback, the intensity of the feedback, and the virtual phase of the feedback based on the spatial characteristics. The electronic device (100) can provide feedback corresponding to the target object based on at least one of the sound effect of the feedback, the intensity of the feedback, and the virtual phase of the feedback.

[0308] FIG. 21 is a drawing for explaining an operation of dividing space according to one embodiment.

[0309] Referring to FIG. 21, the electronic device (100) can identify a first area (1) to be displayed in a window (10) in an image (2110). The electronic device (100) can identify a depth (2 m) corresponding to the first area (1) based on a depth map. The electronic device (100) can identify a depth (2 m) of an object (2101) included in the first area (1) based on the depth map. The electronic device (100) can identify a spatial characteristic of the object (2101) as a cave based on the depth (2 m) of the object (2101).

[0310] The electronic device (100) can identify a second area (2) to be displayed in a changed window (20) in the image (2120). The electronic device (100) can identify a depth (50 m) corresponding to the second area (2) based on a depth map. The electronic device (100) can identify a depth (50 m) of an object (2102) included in the second area (2) based on the depth map. The electronic device (100) can identify a spatial characteristic of the object (2102) as a coastline based on the depth (50 m) of the object (2102).

[0311] The electronic device (100) can receive user input for changing the window (10) in the identified space. The electronic device (100) can reflect keywords corresponding to the identified space as weights for feedback generation. The electronic device (100) can acquire spatial features corresponding to the identified space based on a depth map. The spatial features can include at least one of whether the space is open and the width of the space. The electronic device (100) can provide feedback based on the spatial features.

[0312] The electronic device (100) can provide feedback based on at least one of distance information of the space identified based on the depth map, spatial characteristics of the identified space, or phase of the identified space.

[0313] The electronic device (100) can identify a target object (identified space) as a cave object. The electronic device (100) can provide sound feedback including a resonating sound effect corresponding to the cave object.

[0314] The electronic device (100) can identify the target object (identified space) as a coastal object. The electronic device (100) may not provide sound feedback including a ringing sound effect.

[0315] FIG. 22 is a diagram illustrating a plurality of devices that output feedback according to one embodiment.

[0316] Referring to FIG. 22, the electronic device (100) may include a plurality of sub-devices (100-1, 100-2). Each sub-device (100-1, 100-2) may provide feedback.

[0317] Referring to FIG. 22, the electronic device (100) can obtain an image (2200). The electronic device (100) can identify at least one object (2201, 2202, 2203) based on the image (2200). The electronic device (100) can identify a first area (1) to be displayed in a window (10). The electronic device (100) can identify a second area (2) to be displayed in a changed window (20) based on a user input.

[0318] In image (2200), it is assumed that object (2201) is located in the center, object (2202) is located to the right of object (2201), and object (2203) is located to the left of object (2202).

[0319] An electronic device (100) can determine the positions of multiple sub-devices (100-1, 100-2). It is assumed that the first sub-device (100-1) is a left-side device and the second sub-device (100-2) is a right-side device.

[0320] The electronic device (100) can identify a target object. The electronic device (100) can identify the direction in which the size of the window (10) changes. Based on the identified direction of the target object, the electronic device (100) can provide feedback using a device corresponding to the identified direction among a plurality of sub-devices.

[0321] Referring to embodiment (2210), the electronic device (100) can identify a second area (2) to be displayed in a changed window (20). The second area (2) can include an object (2202). The electronic device (100) can identify the object (2202) as a target object. The electronic device (100) can identify the direction (right) in which the size of the window (10) changes. The electronic device (100) can provide feedback corresponding to the target object (2202) using the second sub-device (100-2) corresponding to the identified direction (right).

[0322] Referring to embodiment (2220), the electronic device (100) can identify a second area (2) to be displayed in a changed window (20). The second area (2) can include an object (2203). The electronic device (100) can identify the object (2203) as a target object. The electronic device (100) can identify the direction (left) in which the size of the window (10) changes. The electronic device (100) can provide feedback corresponding to the target object (2203) using the first sub-device (100-1) corresponding to the identified direction (left).

[0323] For example, the electronic device (100) can generate a depth map (or depth estimation map) from an image. If the depth difference value included in the depth map is greater than or equal to a threshold value, the electronic device (100) can provide feedback based on the depth map. When a user input for changing the window (10) is received, the electronic device (100) can identify the altitude difference value or depth difference value of the target object based on the depth map. The electronic device (100) can provide feedback corresponding to the target object based on the altitude difference value or depth difference value.

[0324] For example, the electronic device (100) can identify the direction of change of the window (10). The electronic device (100) can identify a virtual phase. The virtual phase can represent a phase defined in three-dimensional space. The electronic device (100) can provide feedback by considering the direction of change of the window (10) among the virtual phases.

[0325] For example, when the window (10) is changed to the left direction, the electronic device (100) can provide feedback through the first sub-device (100-1) corresponding to the left direction.

[0326] For example, when the window (10) is changed to the right direction, the electronic device (100) can provide feedback through the second sub-device (100-2) corresponding to the right direction.

[0327] FIG. 23 is a drawing for explaining a window minimization command according to one embodiment.

[0328] Steps S2305, S2320, S2325, and S2330 of FIG. 23 may correspond to steps S505, S520, S525, and S530 of FIG. 5. Duplicate explanations are omitted.

[0329] The electronic device (100) can obtain an image to be displayed on a window (10) (S2305). The electronic device (100) can identify a first area (1) to be displayed on the window (10) among the entire area of ​​the image (S2320).

[0330] The electronic device (100) can identify whether a user input for minimizing the window (10) is received (S2330). If a user input for minimizing the window (10) is received, the electronic device (100) can temporarily not display the screen without terminating the application provided through the window (10). The user input for minimizing the window (10) can be described as a minimize command.

[0331] When a user input to minimize a window (10) is received (S2330-Y), the electronic device (100) can identify the currently provided (or output) feedback (S2335). The currently provided feedback may include at least one of haptic feedback, sound feedback, and visual feedback.

[0332] The electronic device (100) can provide feedback by gradually reducing the intensity of the identified feedback (currently provided feedback) (S2340). The action of gradually reducing the intensity of the feedback may ultimately indicate that the feedback is not provided at all.

[0333] The electronic device (100) may display a default screen corresponding to the minimize operation (S2345). The minimize operation may include an operation of controlling the electronic device (100) so as not to display the window (10) on the display (140) of the electronic device (100), but not to terminate an application that provides an image through the window (10). The minimize operation may be described as a minimize command.

[0334] FIG. 24 is a drawing for explaining a window maximization command according to one embodiment.

[0335] Steps S2405, S2420, S2425, and S2430 of FIG. 24 may correspond to steps S505, S520, S525, and S530 of FIG. 5. Duplicate explanations are omitted.

[0336] The electronic device (100) can obtain an image to be displayed on a window (10) (S2405). The electronic device (100) can identify a first area (1) to be displayed on the window (10) among the entire area of ​​the image (S2420).

[0337] The electronic device (100) can identify whether a user input for maximizing the window (10) has been received (S2430). When a user input for maximizing the window (10) has been received, the electronic device (100) can perform an operation to maximize the size of the window (10) that was displayed only on a portion of the display (140). The electronic device (100) can display the size of the window (10) on the entire screen of the display (140). The size of the window (10) cannot be larger than the size of the display (140).

[0338] When a user input for maximizing a window (10) is received (S2430-Y), the electronic device (100) can identify a second area to be displayed in the window to be maximized among the entire area of ​​the image based on the user input (S2431). The second area for maximizing the window (10) may be different from the entire area of ​​the image. The area to be maximized may refer to the entire area of ​​the display (140) and may not refer to the entire area of ​​the image.

[0339] If the resolution of the image matches the resolution of the display (140), the entire area of ​​the image and the second area (2) for maximization may be the same.

[0340] However, if the resolution of the image is different from the resolution of the display (140), the entire area of ​​the image and the second area (2) for maximization may be different.

[0341] The electronic device (100) can identify the currently provided (or output) feedback (S2435). The currently provided feedback may include at least one of haptic feedback, sound feedback, and visual feedback.

[0342] The electronic device (100) can provide feedback by gradually increasing the intensity of the identified feedback (currently provided feedback) (S2340). The operation of gradually increasing the intensity of the feedback may consequently include an operation of providing the intensity of the feedback as a threshold.

[0343] The electronic device (100) can display the second area (2) through a maximized window (S2445).

[0344] FIG. 25 is a drawing for explaining commands for minimizing and maximizing a window, according to one embodiment.

[0345] Referring to the embodiment (2510) of FIG. 25, the electronic device (100) can display a first area (1) through a window (10).

[0346] Referring to embodiment (2520) of FIG. 25, the electronic device (100) can perform an operation of minimizing the window (10). The electronic device (100) can receive a user input for minimizing the window (10). By minimizing the window (10), the electronic device (100) can no longer display the window (10) on the display (140). The electronic device (100) can display the default screen.

[0347] Referring to the embodiment (2530) of FIG. 25, the electronic device (100) can perform an operation of maximizing the window (10). The electronic device (100) can receive a user input for maximizing the window (10). By maximizing the window (10), the electronic device (100) can maximize the window (10) to the size of the display (140). The electronic device (100) can display the second area (2) through the maximized window.

[0348] FIG. 26 is a drawing for explaining an operation in which a window expands in all directions according to one embodiment.

[0349] Referring to FIG. 26, the electronic device (100) can obtain an image (2610). The image (2610) can include a first object (2601) and a second object (2602).

[0350] For example, the first object (2601) may be a mountain range object. The second object (2602) may be a cave object.

[0351] For example, the electronic device (100) can identify an object using spatial characteristics. The electronic device (100) can identify an area representing a space corresponding to a cave as a second object (2602). The electronic device (100) can identify an area representing a space corresponding to the outside of the cave as a first object (2601).

[0352] The electronic device (100) can identify a first area (1) to be displayed in a window (10) among the entire area of ​​the image (2610). The first area (1) can include a first object (2601).

[0353] The electronic device (100) can receive a user input for changing a window (10). It is assumed that the window (10) is changed to a window (20) based on the user input. The user input may be an input for expanding the window (10) in all directions. For example, the user input may be an input for expanding the window (10) in all directions, such as up, down, left, right, and diagonally.

[0354] The electronic device (100) can identify a second area (2) to be displayed in the changed window (20). The second area (2) can include a first object (2601) and a second object (2602). The electronic device (100) can identify a second object (2602), which is a new object, in the second area (2). The electronic device (100) can identify the second object (2602) as a target object. The electronic device (100) can provide feedback corresponding to the second object (2602), which is the target object.

[0355] For example, the electronic device (100) may receive a user input for maximizing the size of a window (10) in a first region (1). When the user input for maximizing the size of the window (10) is received, the electronic device (100) may identify a second region (2). The electronic device (100) may change the focusing target from a first object (2601) identified in the first region (1) to a second object (2602) identified in the second region (2). The electronic device (100) may change the focusing target by identifying the distance value of each object based on a depth map.

[0356] For example, the electronic device (100) may receive a user input for minimizing the size of a window (10) in a first area (1). When the user input for minimizing the size of the window (10) is received, the electronic device (100) may gradually reduce the feedback provided while displaying the first area (1).

[0357] For example, if there is no object focused on in the first region (1), the electronic device (100) can identify an object located in the central region as a target object. The electronic device (100) can provide feedback corresponding to the object for the central region.

[0358] FIG. 27 is a drawing for explaining an operation of enlarging an image while the window is fixed, according to one embodiment.

[0359] Referring to an embodiment (2710) of FIG. 27, the electronic device (100) can display a first area (1) through a window (10). The first area (1) can include a first object (2701) and a second object (2702). As an example, the first object (2701) can be a plurality of ducks. The second object (2702) can be a ship.

[0360] Referring to embodiment (2720) of FIG. 27, the electronic device (100) can display an image that enlarges a portion of the first region (1) through a window (10). The enlarged image may include only the first object (2701). The enlarged image may not include the second object (2702).

[0361] The electronic device (100) can receive a user input for enlarging the first area (1). The user input may be an input that does not change the size of the window (10). The electronic device (100) can output an image of the first area (1) enlarged while keeping the size of the window (10) fixed.

[0362] FIG. 28 is a diagram for explaining settings related to feedback according to one embodiment.

[0363] The screen (2800) of FIG. 28 may be a screen related to settings that provide feedback. The electronic device (100) may control the display (140) to display the screen (2800).

[0364] The screen (2800) may include at least one of a UI (2810) for determining provision of haptic feedback, a UI (2820) for determining provision of sound feedback, or a UI (2830) for determining provision of visual feedback.

[0365] The screen (2800) allows the user to decide whether to provide (or output) certain feedback.

[0366] FIG. 29 is a diagram for explaining a haptic feedback setting according to one embodiment.

[0367] The screen (2900) of FIG. 29 may be a screen related to settings that provide haptic feedback. The electronic device (100) may control the display (140) to display the screen (2900).

[0368] The screen (2900) may include at least one of a UI (2910) for determining whether to provide haptic feedback, a UI (2920) for determining haptic intensity according to a change in a window, a UI (2930) for indicating haptic intensity according to the size of the window, and a UI (2940) for indicating haptic intensity according to a variable speed.

[0369] The UI (2910) may be a UI for determining whether to provide haptic feedback. The electronic device (100) may receive a user input for providing haptic feedback or a user input for not providing haptic feedback through the UI (2910).

[0370] The UI (2920) may include at least one of a UI for adjusting the intensity of haptics provided when the size of the window is expanded or a UI for adjusting the intensity of haptics provided when the size of the window is reduced.

[0371] The UI (2930) may include a graph indicating the intensity of haptic sensations according to the size of the window. Through the UI (2930), a user can easily and intuitively perceive the relationship between the window size and the intensity of haptic sensations. The graph included in the UI (2930) may be changed according to user input. When a user input for adjusting the graph included in the UI (2930) is received, the electronic device (100) may change the intensity of haptic sensations according to the size of the window.

[0372] For example, when the haptic intensity corresponding to the expansion of the window through the UI (2920) increases, the maximum value of the y-axis of the graph included in the UI (2930) may increase.

[0373] The UI (2940) may include a graph representing the intensity of haptics according to the variable speed of the window. The variable speed may represent the speed at which the size of the window changes.

[0374] FIG. 30 is a drawing for explaining sound feedback settings according to one embodiment.

[0375] The screen (3000) of FIG. 30 may be a screen related to settings for providing sound feedback. The electronic device (100) may control the display (140) to display the screen (3000).

[0376] The screen (3000) may include at least one of a UI (3010) for determining whether to provide sound feedback, a UI (3020) for determining sound intensity according to a change in the window, a UI (3030) for indicating sound intensity according to the size of the window, and a UI (3040) for indicating sound intensity according to a variable speed.

[0377] The UI (3010) may be a UI for determining whether to provide sound feedback. The electronic device (100) may receive a user input for providing sound feedback or a user input for not providing sound feedback through the UI (3010).

[0378] The UI (3020) may include at least one of a UI for adjusting the intensity of a sound provided when the size of the window is expanded or a UI for adjusting the intensity of a sound provided when the size of the window is reduced.

[0379] The UI (3030) may include a graph indicating sound intensity according to the size of the window. Through the UI (3030), a user can easily and intuitively perceive the relationship between the window size and sound intensity. The graph included in the UI (3030) may be changed according to user input. When a user input for adjusting the graph included in the UI (3030) is received, the electronic device (100) may change the sound intensity according to the size of the window.

[0380] For example, when the sound intensity corresponding to the expansion of the window through the UI (3020) increases, the maximum value of the y-axis of the graph included in the UI (3030) may increase.

[0381] The UI (3040) may include a graph representing the sound intensity according to the variable speed of the window. The variable speed may represent the speed at which the window changes size.

[0382] FIG. 31 is a diagram for explaining a visual feedback setting according to one embodiment.

[0383] The screen (3100) of FIG. 31 may be a screen related to settings that provide visual feedback. The electronic device (100) may control the display (140) to display the screen (3100).

[0384] The screen (3100) may include at least one of a UI (3110) for determining whether to provide visual feedback, a UI (3120) for determining an animation size according to a change in the window, a UI (3130) for indicating an animation size according to the size of the window, and a UI (3140) for indicating an animation size according to a variable speed.

[0385] The UI (3110) may be a UI for determining whether to provide visual feedback. The electronic device (100) may receive a user input for providing visual feedback or a user input for not providing visual feedback through the UI (3110).

[0386] The UI (3120) may include at least one of a UI for adjusting the size of an animation provided when the size of the window is expanded or a UI for adjusting the size of an animation provided when the size of the window is reduced.

[0387] The UI (3130) may include a graph indicating the size of the animation according to the size of the window. Through the UI (3130), the user can easily and intuitively perceive the relationship between the size of the window and the size of the animation. The graph included in the UI (3130) may be changed according to user input. When a user input for adjusting the graph included in the UI (3130) is received, the electronic device (100) may change the size of the animation according to the size of the window.

[0388] For example, when the animation size corresponding to the expansion of the window through the UI (3120) increases, the maximum value of the y-axis of the graph included in the UI (3130) may increase.

[0389] The UI (3140) may include a graph representing the size of the animation according to the variable speed of the window. The variable speed may represent the rate at which the size of the window changes.

[0390] FIG. 32 is a drawing for explaining feedback detail settings according to one embodiment.

[0391] The screen (3200) of FIG. 32 may include at least one UI for detailed settings related to feedback generation. The screen (3200) may include at least one of a UI (3210) for describing feedback detailed settings, a UI (3220) for determining whether to generate feedback in an idle state, or a UI (3230) for representing a feedback library. The electronic device (100) may control the display (140) to display the screen (3200).

[0392] The UI (3210) may include text describing the detailed settings provided on the screen (3200). The UI (3210) may include information indicating that a user can directly set detailed settings related to generating feedback through the screen (3200).

[0393] The UI (3220) may be a UI for determining whether to pre-generate feedback in an idle state. The idle state may indicate a state in which an image is not displayed through a window. The idle state may indicate a state in which an image is acquired by the electronic device (100) but not output through the display (140). The idle state may be described as a pre-prepared state or a standby state. The state may be described as a mode.

[0394] The UI (3230) may include a library indicating whether feedback is set for each object. The UI (3230) may also include a UI for determining whether feedback is provided for multiple objects. Through the UI (3230), a user can determine whether to provide feedback for each individual object.

[0395] For example, the UI (3230) may include at least one of a UI indicating whether to generate feedback corresponding to a sea object, a UI indicating whether to generate feedback corresponding to an art object, a UI indicating whether to generate feedback corresponding to a wind object, and a UI indicating whether to generate feedback corresponding to a lightning object.

[0396] FIG. 33 is a diagram illustrating a notification UI for indicating provision of feedback, according to one embodiment.

[0397] Referring to FIG. 33, the electronic device (100) may display a UI (3310) indicating that feedback is provided. The electronic device (100) may control the display (140) to display the UI (3310) through a window (10).

[0398] The UI (3310) may be a UI for notifying that feedback is provided. The UI (3310) may include information indicating that a function for providing feedback can be performed. The UI (3310) may include at least one of text information for notifying that feedback is provided and image information indicating that feedback is provided.

[0399] FIG. 34 is a diagram illustrating a notification UI for indicating provision of feedback, according to one embodiment.

[0400] Referring to FIG. 34, the electronic device (100) may display at least one UI (3410, 3420) indicating that an image is being analyzed. The electronic device (100) may control the display (140) to display at least one UI (3410, 3420) through a window (10).

[0401] The UI (3410) may include text information indicating that the image is being analyzed to provide feedback. The UI (3420) may include image information indicating that the image is being analyzed to provide feedback. The electronic device (100) may display at least one of the UI (3410) or the UI (3420) via a window (10).

[0402] FIG. 35 is a diagram illustrating a notification UI for indicating provision of feedback, according to one embodiment.

[0403] Referring to FIG. 35, the electronic device (100) may display a UI (3510) indicating that feedback is provided. The electronic device (100) may control the display (140) to display the UI (3510) through a window (10).

[0404] The electronic device (100) can display the UI (3510) at a preset location among the entire area for displaying the window (10). The preset location can be changed according to the user's settings.

[0405] The UI (3510) may be a UI for notifying that feedback is provided. The UI (3510) may include image information indicating that a function for providing feedback can be performed. The image information may include at least one of a sign, an icon, and a symbol.

[0406] When the UI (3510) is displayed, the user can easily recognize that feedback can be provided on the displayed image.

[0407] FIG. 36 is a diagram illustrating a notification UI for indicating provision of feedback, according to one embodiment.

[0408] Referring to embodiment (3610) of FIG. 36, the electronic device (100) may display at least one UI (3611, 3612, 3613, 3614) to indicate that feedback is provided.

[0409] The electronic device (100) can display a UI (3611) through a window (10) indicating that feedback corresponding to the object can be provided.

[0410] The electronic device (100) may display a UI (3612, 3613, 3614) through the window (10) indicating that feedback may be provided in the direction in which the size of the window (10) changes. The image described in the embodiment (3610) may correspond to the image (800) of FIG. 8.

[0411] The UI (3612) may indicate that feedback may be provided when the window (10) is expanded to the right.

[0412] The UI (3613) may indicate that feedback may be provided when the window (10) is expanded to the left.

[0413] The UI (3614) may indicate that feedback may be provided when the window (10) is expanded upwards.

[0414] The electronic device (100) can display the UI (3612, 3613, 3614) at a position corresponding to the change direction of the window (10).

[0415] Referring to the embodiment (3620) of FIG. 36, the electronic device (100) may display a UI (3621) indicating an expanding direction. The UI (3621) may indicate that a user input for expanding the window (10) to the right has been received. The UI (3621) may indicate that feedback is to be provided or is currently being provided based on the received user input.

[0416] FIG. 37 is a drawing for explaining slide variable settings according to one embodiment.

[0417] Referring to FIG. 37, the electronic device (100) can control the display (140) to display a screen (3700) for setting feedback according to the slide variation. The electronic device (100) can include a display (140) that can slide. The size of the display (140) can change depending on the slide. When the size of the display (140) changes, the size of the window displayed on the display (140) can also change.

[0418] User input for changing the size of a window may include input for changing the slide display (140). The user input may include an operation for sliding the display (140) itself.

[0419] The screen (3700) may include a graph indicating the intensity of feedback provided in response to a sliding operation. The intensity of feedback provided in response to a sliding operation may be applied to haptic feedback, sound feedback, and visual feedback.

[0420] Figure 37 may illustrate a settings screen for haptic feedback. For example, when the display (140) expands according to a sliding operation, the feedback intensity may gradually increase.

[0421] For example, when the display (140) is expanded by a threshold ratio (75%) based on the maximum expansion (100%), the electronic device (100) can provide maximum feedback intensity. When the display (140) is expanded by more than the threshold ratio (75%), the electronic device (100) can gradually reduce the feedback intensity.

[0422] FIG. 38 is a diagram for explaining feedback according to the start and end times of a change in a window, according to one embodiment.

[0423] Steps S3825, S3845, and S3855 of FIG. 38 may correspond to steps S725, S745, and S755 of FIG. 7. Duplicate explanations are omitted.

[0424] The electronic device (100) can display a first area (1) through a window (10) (S3825). The electronic device (100) can determine whether a first event that initiates a change in the window (10) is identified (S3830). The first event may include an event in which a user input that initiates a change in the size of the window (10) is received.

[0425] When the first event is identified (S3830-Y), the electronic device (100) can identify a target object corresponding to the identification direction that changes the window (10) (S3845). The electronic device (100) can identify an object corresponding to the identification direction among at least one object identified in the image as the target object.

[0426] The electronic device (100) can provide a first feedback corresponding to the target object (S3850). The first feedback can represent feedback of a first intensity.

[0427] After providing the first feedback, the electronic device (100) may determine whether a second event that terminates the change of the window (10) is identified (S3851). The second event may include an event in which a user input that terminates the change of the size of the window (10) is received.

[0428] When a second event is identified (S3851-Y), the electronic device (100) may provide second feedback corresponding to the target object (S3852). The second feedback may represent feedback of a second intensity. For example, the second intensity may be greater than the first intensity.

[0429] The electronic device (100) can display the second area (2) through the changed window (20) (S3855).

[0430] FIG. 39 is a drawing for explaining a slide operation according to one embodiment.

[0431] An embodiment (3910) of FIG. 39 may represent a user input that initiates a sliding motion. The electronic device (100) may include a display (140) capable of sliding. When a user input that initiates sliding the display (140) is received, the electronic device (100) may identify a target object corresponding to the sliding direction. The electronic device (100) may provide a first feedback corresponding to the target object with a first intensity.

[0432] Embodiment 39 (3920) of FIG. 39 may represent a user input terminating a sliding motion. When a user input terminating sliding of the display (140) is received, the electronic device (100) may provide a second feedback corresponding to the target object at a second intensity.

[0433] The operations of FIG. 38 can be applied to the embodiment of FIG. 39. Duplicate explanations are omitted.

[0434] For example, the electronic device (100) may receive a user operation for varying the display (140). The electronic device (100) may identify a window corresponding to the degree of variation of the display (140) based on the user operation. The size of the window may change depending on the degree of variation of the display (140). The electronic device (100) may identify an area to be displayed among the areas in front of the image based on the changed window. The electronic device (100) may identify a target object based on the area to be displayed.

[0435] When a user operation that changes the display (140) stops, the electronic device (100) can identify a window corresponding to the display (140) at the time the user operation stopped. The electronic device (100) can identify an area to be displayed based on the identified window. The electronic device (100) can identify a target object in the identified area. The electronic device (100) can provide feedback corresponding to the target object.

[0436] The electronic device (100) can provide feedback based on the degree of variation (interval or progress) of the display (140). When the variation of the display (140) is at its maximum, feedback indicating that the variation is at its maximum can be provided.

[0437] FIG. 40 is a diagram for explaining a feedback table according to one embodiment.

[0438] Referring to FIG. 40, the electronic device (100) can store a first feedback table (4000). The first feedback table (4000) can include the type of feedback (haptic, sound, visual) or / and the intensity of the feedback provided to the user according to preset criteria.

[0439] The established criteria may include at least one of the following: device, placement status, whether mounted, and connected device.

[0440] The device may represent a type of electronic device (100) that includes a display (140). For example, the device may include a tablet, a smartphone, a TV, etc. The electronic device (100) may provide different feedback based on the form factor of the device.

[0441] The layout state may indicate the layout of the display (140). For example, the layout state may be a horizontal layout or a vertical layout.

[0442] Whether or not the electronic device (100) is mounted may indicate whether or not the electronic device (100) is mounted. For example, whether or not the electronic device is mounted may indicate whether or not the electronic device is mounted by a charging stand or a fixed stand.

[0443] For example, the electronic device (100) can obtain a gyro sensing value through a gyro sensor included in the electronic device (100). The electronic device (100) can identify whether the electronic device (100) is currently in a cradle state based on the gyro sensing value. If the electronic device (100) is cradle-mounted, the electronic device (100) may tilt. The electronic device (100) can identify that the electronic device (100) is tilted based on the gyro sensing value. If the electronic device (100) maintains the tilted state for a threshold time or longer, the electronic device (100) can identify that the electronic device (100) is in a cradle state. If the user directly holds the electronic device (100) by hand, the gyro sensing value will change slightly. However, the gyro sensing value of the electronic device (100) can be maintained in the cradle state. The electronic device (100) can identify whether the electronic device (100) is in a stationary state based on the gyro sensing value.

[0444] For example, when the electronic device (100) is identified as being in a stationary state, the electronic device (100) may provide at least one of sound feedback or visual feedback, excluding haptic feedback. This is because if haptic feedback is provided in a stationary state, the user may feel discomfort due to the shaking of the electronic device (100).

[0445] For example, the electronic device (100) may be connected to an external device (e.g., a keyboard). The electronic device (100) may be in a stationary state. The electronic device (100) may not provide haptic feedback when in a stationary state.

[0446] For example, the electronic device (100) may operate in desktop mode. Desktop mode may be a mode in which a tablet or smartphone is used as a PC environment. A user may use the electronic device (100) in desktop mode, which is not a PC, in the same manner as a PC. Desktop mode may provide a PC interface. Desktop mode may provide a preset UI for a PC environment. Desktop mode may support a keyboard or mouse. Desktop mode may allow connection to an external monitor. In desktop mode, the electronic device (100) may not provide haptic feedback, but may provide at least one of sound feedback and visual feedback.

[0447] For example, the electronic device (100) may operate in split-screen mode (or flex mode). Split-screen mode may be a mode in which one display (140) or multiple displays are each divided into separate screens. Different images may be output on the divided separate screens. In split-screen mode, the electronic device (100) may not provide haptic feedback, but may provide at least one of sound feedback and visual feedback.

[0448] For example, the electronic device (100) may be connected to an external device including a haptic function. When connected to the external device including a haptic function, the electronic device (100) may provide at least one of sound feedback and visual feedback instead of providing haptic feedback. The electronic device (100) may generate a control signal so that haptic feedback may be provided through the external device. The electronic device (100) may transmit the control signal to the external device. The user may receive haptic feedback through the external device. When the connection with the external device is lost, the electronic device (100) may provide haptic feedback to the user through the electronic device (100).

[0449] For example, the electronic device (100) may provide feedback based on at least one of the device form factor, hardware specifications, and the application that provides the image displayed in the window.

[0450] For example, the electronic device (100) may provide feedback based on sound channel information. The electronic device (100) may determine whether to provide two or more audio channels based on the sound channel information. If it is capable of providing two or more audio channels, the electronic device (100) may provide at least one of sound feedback or visual feedback without providing haptic feedback.

[0451] FIG. 41 is a drawing for explaining feedback corresponding to the expansion direction of a window, according to one embodiment.

[0452] Referring to the embodiment (4110) of FIG. 41, the electronic device (100) can identify the expansion direction of the window (10). The expansion direction can include at least one of upper (411), upper-right (4112), right (4113), lower-right (4114), lower (4115), lower-left (4116), left (4117), and upper-left (4118).

[0453] The electronic device (100) may store a second feedback table (4120) indicating feedback corresponding to the expansion direction. The second feedback table (4120) may include feedback corresponding to the expansion direction. The feedback corresponding to the expansion direction may be the same or different depending on some directions.

[0454] FIG. 42 is a diagram illustrating feedback related to an XR device, according to one embodiment.

[0455] Referring to FIG. 42, the electronic device (100) may be an XR (Extended Reality) device. The XR device may include an HMD (Head-Mounted Display) device. The XR device may include a VR (Virtual Reality) device or an AR (Augmented Reality) device. The electronic device (100) may provide feedback in response to a function (or operation) that can be implemented through the XR device.

[0456] Referring to the embodiment (4210) of FIG. 42, the electronic device (100) can obtain rotation angle information of the electronic device (100). The electronic device (100) can identify a central region of a field of view (FOV) based on the rotation angle information. The electronic device (100) can identify the central region of the FOV in the obtained image. The electronic device (100) can provide feedback corresponding to the central region of the FOV.

[0457] For example, the electronic device (100) may include a gyro sensor. The electronic device (100) may obtain rotation angle information based on sensing data obtained through the gyro sensor.

[0458] Referring to the embodiment (4220) of FIG. 42, the electronic device (100) can obtain information about the user's gaze. The electronic device (100) can track the user's gaze. The electronic device (100) can identify the location of the user's iris. The electronic device (100) can identify the user's gaze based on the location of the user's iris. The electronic device (100) can identify an object corresponding to the user's gaze.

[0459] For example, the electronic device (100) can identify an object corresponding to the user's gaze as a target object.

[0460] For example, the electronic device (100) can identify the direction corresponding to the user's gaze as the change direction of the window (10).

[0461] For example, the electronic device (100) may include an image sensor. The electronic device (100) may obtain information about the user's gaze based on sensing data obtained through the image sensor.

[0462] For example, the electronic device (100) can receive user input for reducing (or enlarging) the screen provided through the display (140). The electronic device (100) can provide feedback based on the user input.

[0463] For example, the electronic device (100) can obtain a Z-depth value. The electronic device (100) can provide feedback corresponding to the Z-depth value. The Z-depth value can represent a value indicating the degree to which a user wearing the XR device has moved forward or backward.

[0464] FIG. 43 is a drawing for explaining a method of controlling an electronic device according to one embodiment.

[0465] According to one embodiment, a method for controlling an electronic device includes the steps of receiving an image to be displayed through a first window (S4305), identifying an expansion direction of the first window based on a user input related to the first window (S4310), identifying a target object corresponding to a newly displayed area in the image based on the expansion direction (S4315), and providing at least one of haptic feedback, sound feedback, or visual feedback based on the target object (S4320).

[0466] The control method may include the steps of identifying a first area to be displayed through a first window among the entire area of ​​the image and receiving a user input for expanding the first window while displaying the first area through the first window.

[0467] The step of identifying the expansion direction (S4310) may include, when a user input is received, identifying the expansion direction of the first window based on the user input, the control method may include the step of identifying a second area to be displayed through a second window expanded based on the user input, the step of identifying a target object in the second area, and the step of displaying the second area through the second window.

[0468] The control method includes a step of identifying at least one object to provide feedback based on an image and a step of determining feedback corresponding to the at least one object, wherein the step of identifying a target object (S4315) identifies the target object in a second area among the at least one object, and the step of providing (S4320) provides feedback corresponding to the target object among the determined feedback, and the feedback may include at least one of haptic feedback, sound feedback, or visual feedback.

[0469] The control method includes a step of identifying an anchor region of at least one object, and the step of identifying a target object (S4315) can identify an object corresponding to the identified anchor region in the second region as the target object when the anchor region of the object is identified in the second region.

[0470] The control method may include, when a plurality of target objects are identified, a step of identifying a first overlapping region in which a first anchor region and a second region of a first target object overlap, a step of identifying a second overlapping region in which a second anchor region and a second region of a second target object overlap, and a step of comparing the first overlapping region and the second overlapping region to identify a final target object.

[0471] The control method includes a step of obtaining a difference value between an area of ​​a first overlapping area and an area of ​​a second overlapping area, and a step of identifying an object in a larger area among the first overlapping area and the second overlapping area as a final target object if the difference value is greater than or equal to a threshold value, and the step of providing (S4320) may include a step of providing at least one of haptic feedback, sound feedback, or visual feedback based on the final target object.

[0472] The control method may include a step of identifying a final target object based on a preset priority if the difference value is less than a threshold value.

[0473] The control method includes a step of identifying a depth map corresponding to the image when an image to be displayed is received through a first window, and the step of determining feedback can determine feedback corresponding to at least one object based on the image and the depth map.

[0474] The control method may include, when a user input for minimizing a first window is received, the steps of identifying feedback currently being provided, providing the feedback by gradually reducing the intensity of the feedback, and displaying a preset default screen.

[0475] The methods according to the various embodiments of the present disclosure described above can be implemented in the form of an application that can be installed on an existing electronic device.

[0476] The methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device.

[0477] The various embodiments of the present disclosure described above may also be performed through an embedded server provided in an electronic device, or an external server of at least one of the electronic device and the display device.

[0478] The electronic device (100) can change the window (10) that displays the content during the process of using the content, such as the background screen, web surfing, or watching a video. The electronic device (100) can recognize the area where the content is displayed in real time. The electronic device (100) can identify changes in the direction of the window (10). The electronic device (100) can generate and provide feedback based on the displayed area and changes in direction. The electronic device (100) can provide feedback based on at least one of the variable speed of the window (10), the direction of change, the status of the electronic device, the degree of expansion of the image (the area displayed), or the recognized object.

[0479] The electronic device (100) can provide emotional quality through feedback related to the window (10). When the window (10) is variable, an immersive experience can be provided to the user through feedback. When the window (10) is variable, this may include when the display (140) is variable.

[0480] For example, the electronic device (100) may acquire an image and analyze the acquired image to derive a representative keyword. The electronic device (100) may identify a representative object included in the image based on the representative keyword. The electronic device (100) may identify sub-keywords in the entire area of ​​the image. The electronic device (100) may identify at least one object including the representative object based on the sub-keyword. The electronic device (100) may generate feedback corresponding to at least one object. The electronic device (100) may determine feedback corresponding to at least one object using a pre-stored database (e.g., a feedback table).

[0481] The electronic device (100) can receive a user input for changing a window (10). The electronic device (100) can identify a target variable to be changed based on which variable among a plurality of variables representing the window (10).

[0482] When the window (10) is expanded, the electronic device (100) can provide a feedback effect corresponding to the expanded state of the target side. When the window (10) is reduced, the electronic device (100) can provide a feedback effect corresponding to the reduced state of the target side. For example, the visual feedback can be generated in various formats, such as web programming code or a simple image.

[0483] The electronic device (100) can control the intensity of the feedback based on at least one of the change rate or progress of the target variable.

[0484] The electronic device (100) may include a sliding member. The sliding member may perform a sliding function that changes the display (140). The electronic device (100) may receive a sliding operation through the sliding member. The electronic device (100) may provide feedback according to the sliding operation.

[0485] The electronic device (100) can identify the target change of the window (10) in a direction corresponding to the sliding operation.

[0486] When an event in which a sliding operation is in progress is identified, the electronic device (100) can provide feedback corresponding to the target change at a preset intensity.

[0487] When an event that stops the sliding operation is identified, the electronic device (100) may provide feedback corresponding to the position of the target variable and then no longer provide feedback.

[0488] According to an example embodiment of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device according to the disclosed embodiments, which is a device that can call instructions stored in the storage medium and operate according to the called instructions. When the instructions are executed by a processor, the processor may directly or under the control of the processor use other components to perform a function corresponding to the instructions. The instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain signals and is tangible, but does not distinguish between whether data is stored semi-permanently or temporarily in the storage medium.

[0489] According to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store. In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0490] Each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the respective components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0491] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the scope of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea of ​​the present disclosure.

Claims

1. In electronic devices, Memory that stores instructions; display; and At least one processor comprising processing circuitry; The above instructions, when individually or collectively executed by the at least one processor, Identify a first portion of the image having a size corresponding to the window among the areas of the image, Displaying the first part of the image in the window through the display, Determine the expansion direction of the window based on user input for the window, Based on the above expansion direction, a newly displayed area outside the first partial area in the image is identified, Identifying a target object contained in the newly displayed area of ​​the above image, An electronic device that provides at least one of haptic feedback, sound feedback, or visual feedback based on the target object.

2. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, An electronic device that receives a user input for expanding the window while the first partial area is displayed in the window of the first size.

3. In paragraph 2, The above instructions, when individually or collectively executed by the at least one processor, When the user input is received, the extension direction of the window is identified based on the user input, Identifying a second portion of the image to be displayed in the second size window expanded based on the user input; Identifying the target object in the newly displayed area excluding the first portion of the second portion of the above-mentioned area, An electronic device that controls the display to display the second partial area in the window of the second size.

4. In paragraph 3, The above instructions, when individually or collectively executed by the at least one processor, Identify at least one object to provide feedback based on the image; Determine at least one feedback corresponding to at least one object, Identifying the target object in the newly displayed area within the second partial area among the at least one object, An electronic device that provides feedback corresponding to the target object among at least one of the determined feedbacks.

5. In paragraph 4, The above instructions, when individually or collectively executed by the at least one processor, Identifying an anchor region of at least one object within the image, An electronic device that identifies the object corresponding to the confirmed anchor area as the target object when it is confirmed that the newly displayed area within the second partial area includes an anchor area of ​​the object.

6. In paragraph 5, The above instructions, when individually or collectively executed by the at least one processor, When a plurality of target objects are confirmed in the newly displayed area, a first overlapping area in which the first anchor area of ​​the first target object and the newly displayed area overlap is identified, Identify a second overlapping area where the second anchor area of ​​the second target object and the newly displayed area overlap, An electronic device that identifies a final target object by comparing the first overlapping area and the second overlapping area.

7. In paragraph 6, The above instructions, when individually or collectively executed by the at least one processor, Obtain the difference value between the area of ​​the first overlapping area and the area of ​​the second overlapping area, If the difference value is greater than or equal to the threshold value, the object in the larger area among the first overlapping area and the second overlapping area is identified as the final target object, An electronic device that provides at least one of the haptic feedback, the sound feedback, or the visual feedback based on the final target object.

8. In paragraph 7, The above instructions, when individually or collectively executed by the at least one processor, An electronic device that identifies the final target object based on a preset priority if the difference value is less than the threshold value.

9. In paragraph 4, The above instructions, when individually or collectively executed by the at least one processor, Obtain the image to be displayed in the above window, Identify the depth map corresponding to the above image, An electronic device that determines at least one feedback corresponding to at least one object based on the image and the depth map.

10. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, When user input to minimize the above window is received, identify the currently provided or most recently provided feedback, Provide the feedback by gradually reducing the intensity of the feedback, An electronic device that controls the display to display a preset default screen.

11. In a method for controlling an electronic device, The above control method is, A step of identifying a first partial area of ​​an image having a size corresponding to a window among the areas of the image; A step of displaying the first partial area of ​​the image in the window; A step of determining an expansion direction of the window based on a user input for the window; A step of identifying a newly displayed area outside the first partial area in the image based on the expansion direction; A step of identifying a target object included in the newly displayed area of ​​the image; and A control method comprising: providing at least one of haptic feedback, sound feedback, or visual feedback based on the target object.

12. In paragraph 11, The above control method is, A control method for receiving a user input for expanding the window while the first partial area is displayed in the window of the first size.

13. In paragraph 12, The step of identifying the above expansion direction is: When the user input is received, the extension direction of the window is identified based on the user input, The above control method is, A step of identifying a second partial area of ​​the image to be displayed in the second size window expanded based on the user input; A step of identifying the target object in the newly displayed area excluding the first partial area in the second partial area; and A control method comprising: a step of displaying the second partial area in the window of the second size; 14. In paragraph 13, The above control method is, identifying at least one object to provide feedback based on the image; and comprising a step of determining at least one feedback corresponding to at least one object; The step of identifying the above target object is: Identifying the target object in the newly displayed area within the second partial area among the at least one object, The steps provided above are: A control method for providing feedback corresponding to the target object among at least one of the determined feedbacks.

15. In paragraph 14, The above control method is, A step of identifying an anchor region of at least one object within the image; The step of identifying the above target object is: A control method for identifying an object corresponding to the confirmed anchor area as the target object when it is confirmed that the newly displayed area within the second partial area includes an anchor area of ​​an object.

Images