User interface operation on a foldable screen device

Abstract

User interface (UI) manipulation techniques are disclosed that enable user devices to hide and obscure sensitive information displayed on a flexible, foldable, or otherwise reconfigurable display from onlookers while maintaining or improving exclusive accessibility to a primary user. Examples of the disclosed techniques can manipulate the UI so that it is primarily only viewed as intended when the user device is physically configured at a particular folding angle (and / or viewing angle). In some examples, the UI can be customized to prompt a particular device configuration (e.g., folding angle) that can provide an optimal security configuration for the user's current environment. The UI displayed on the display screen can change form factor (e.g., size intended for viewing by the user) to adapt to the current environment and / or based on the sensitivity of the displayed content.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Application No. 63 / 008,473, entitled "User Interface Manipulation in a Foldable Screen Device," filed April 10, 2020, and U.S. Provisional Application No. 62 / 991,553, entitled "Foldable Phone UI Manipulation for Privacy," filed March 18, 2020. Each of these applications is incorporated herein by reference in its entirety.

[0002] (Technical field) The present disclosure relates generally to data security, and more particularly to protecting sensitive information displayed on mobile end user devices from attempts by onlookers to intercept such information. [Background technology]

[0003] With the proliferation of computers and portable electronic devices, the preferred mode of information presentation has long since shifted from paper to electronic. Typically, such electronic devices include a display screen (e.g., a liquid crystal display (LCD) screen) that presents visual information to a human user.

[0004] In many cases, for example, where financial or business transactions are involved, sensitive information such as social security numbers or bank account numbers may be displayed on the display screen. This state of technology creates a vulnerability for unscrupulous bystanders to steal sensitive personal and financial information from the user by viewing the user's information from the side of the device and taking imagined or actual photographs. [Brief description of the drawings]

[0005] One or more embodiments of the present disclosure are illustrated by way of example, and not limitation, in the figures of the accompanying drawings in which like references indicate similar elements.

[0006] [Figure 1A] FIG. 1A illustrates an example environment in which a foldable phone user interface (UI) as presented herein may be implemented.

[0007] [Figure 1B] FIG. 1B illustrates the folding angles of an example foldable phone.

[0008] [Figure 1C] FIG. 1C shows the viewing angle of an example foldable phone relative to a human user.

[0009] [Figure 1D] FIG. 1D shows a table illustrating different configurations of the disclosed UI manipulation techniques.

[0010] [Figure 2A] FIG. 2A shows examples of how a foldable phone UI as presented herein can be implemented with respect to different privacy scenarios.

[0011] [Figure 2B] FIG. 2B shows an example of visual guidance indicators and textual guidance displayed in the actual UI versus the UI as seen by a user.

[0012] [Figure 2C] FIG. 2C shows additional implementation details for some examples of the foldable phone UI manipulation techniques introduced.

[0013] [Diagram 3] FIG. 3 illustrates a functional block diagram of an example UI manipulation engine that can be utilized to implement UI manipulation techniques for a foldable phone.

[0014] [Figure 4A]FIG. 4A illustrates an example data flow diagram for implementing a foldable phone UI manipulation technique with some example components of the UI manipulation engine of FIG.

[0015] [Figure 4B] FIG. 4B illustrates another example data flow diagram for implementing a foldable phone UI manipulation technique with some example components of the UI manipulation engine of FIG.

[0016] [Figure 4C] FIG. 4C illustrates yet another example data flow diagram for implementing a foldable phone UI manipulation technique with some example components of the UI manipulation engine of FIG.

[0017] [Diagram 5] FIG. 5 shows a flow chart illustrating an example method for implementing UI navigation techniques for a foldable phone.

[0018] [Figure 6] FIG. 6 shows a flow chart illustrating another example method for implementing UI navigation techniques for a foldable phone.

[0019] [Figure 7A] FIG. 7A shows a more detailed example of how a distorted UI may be generated in some embodiments of the foldable phone UI manipulation techniques.

[0020] [Figure 7B] FIG. 7B illustrates additional assumptions for the calculations in the example of FIG. 7A.

[0021] [Figure 7C] FIG. 7C shows an example transformation calculation by changing the folding angle in the example of FIG. 7A.

[0022] [Figure 7D] FIG. 7D shows an example transformation calculation due to a change in the viewing angle in the example of FIG. 7A.

[0023] [Figure 7E] FIG. 7E shows an example of how the UI can adjust to compensate for angular distortions (e.g., how tilted) in the actual, generated UI in response to a change in viewing angle.

[0024] [Figure 7F] FIG. 7F illustrates an example relationship between the amount of adjustment of the angular distortion and the change in viewing angle.

[0025] [Figure 8] FIG. 8 shows a flow chart illustrating an example method for implementing a grip manipulation technique.

[0026] [Figure 9] FIG. 9 depicts a flowchart illustrating an example method for implementing a UI-driven grip manipulation technique.

[0027] [Figure 10A] FIG. 10A is an example data flow for implementing a grip manipulation technique using some example components of the UI manipulation engine of FIG.

[0028] [Figure 10B] FIG. 10B is another example data flow for implementing a grip manipulation technique using some example components of the UI manipulation engine of FIG.

[0029] [Figure 11A] FIG. 11A illustrates how individuals in close proximity to a user device may be able to view sensitive information displayed by the user device.

[0030] [Figure 11B] FIG. 11B illustrates how the UI operations engine may calculate which directions should be obscured to ensure privacy, based on the locations of nearby individuals.

[0031] [Figure 11C] FIG. 11C illustrates how the UI navigation engine may result in the display of visual guidance indicia to assist the user in placing their hands in an optimal grip position.

[0032] [Figure 11D] FIG. 11D illustrates how confidential information displayed by a user device can be hidden from nearby individuals when the hands are placed in an optimal grip position.

[0033] [Figure 12] FIG. 12 depicts a high-level block diagram illustrating an example of a mobile system in which at least some operations associated with the techniques presented herein may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] References herein to "one embodiment," "one embodiment," or the like mean that the particular feature, function, structure, or characteristic being described is included in at least one embodiment of the present disclosure. Appearances of such phrases herein do not necessarily all refer to the same embodiment. On the other hand, referenced embodiments are also not necessarily mutually exclusive.

[0035] As mentioned above, financial and commercial transactions are increasingly being conducted on computers and other portable electronic devices such as mobile phones instead of paper. Typically, such electronic devices are equipped with a display screen (e.g., a liquid crystal display (LCD) screen) that presents visual information to a human user. In many cases, for example, when financial or commercial transactions are involved, sensitive information such as social security numbers or bank account numbers may be displayed on the display screen. As smart phones take an increasingly larger share of the payment market, more and more people are using smart phones for banking applications on a regular basis. People may try to enter and read private data in public places that are highly insecure. This state of technology creates vulnerabilities to malicious people and agents who can physically view the user's interactions with the device and can steal sensitive personal and financial information from the user by viewing the user's information from the side of the device and by taking mental or actual photographs.

[0036] Meanwhile, in the past decade, there has been great progress in the fabrication of flexible displays (e.g., using fabric-based substrates), mainly due to advances in organic light-emitting diode (OLED) display technology. These flexible display devices allow different configurations within a single device, such as allowing variable screen sizes to accommodate application needs and variable physical configurations. As flexible displays improve, a variety of new physical configurations are possible for user interfaces and modes of interaction. The UI elements (e.g., screen(s)) of a foldable mobile phone can be quickly manipulated into different shapes or changed to different folding angles. In addition to flexible displays, even conventional displays continue to become more popular and cheaper to manufacture, and devices with rigid displays but with more than one screen (e.g., clamshell phones with two screens) are also appearing on the market.

[0037] Thus, presented herein are user interface (UI) manipulation techniques that allow a user device to hide and obscure sensitive information displayed on a flexible, foldable, or otherwise reconfigurable display from bystanders while maintaining or improving its accessibility only to primary users. More specifically, one or more embodiments of the disclosed technology can manipulate the UI such that it is primarily only viewable as intended when the user device is physically configured at a certain folding angle. In some examples, the UI can be customized to prompt a particular device configuration (e.g., folding angle) that can provide an optimal security configuration for the user's current surroundings. The UI displayed on the display screen can change its form factor (e.g., intended size as seen by the user) to adapt to the current environment (e.g., when the user moves from one environment to another or when the environment changes around the user). Additionally or alternatively, the displayed UI can change its form factor based on the sensitivity of the displayed content.

[0038] Through UI manipulation, the technology presented here allows the user to hide or render invisible information displayed on the device in an intuitive manner that maintains ease of use and readability, thereby implementing an effective user interface system that allows users to easily achieve optimal screen configurations for security and helps prevent unintentional data loss to bystanders.

[0039] In the following description, for illustrative purposes only, an example of a foldable mobile device is used to explain various aspects of the technology. However, it should be noted that the technology introduced herein is not limited in its applicability to foldable mobile phones or any other particular type of device. Also, while the following description focuses on implementing various aspects of the disclosed technology on a mobile phone, other electronic devices or systems (e.g., laptops or tablets) may also be adapted to the technology in a similar manner. The term "foldable display" or "foldable screen" is used interchangeably herein, unless otherwise clear by context, to refer to any variation of a display in which the display area is foldable, which should include at least two variations: (1) a display having at least one flexible screen (e.g., one flexible OLED display screen) configured to be foldable, and (2) a display having at least two sections, each of which has a screen, but the two sections are configured to function together (or collectively) as one screen, and the two sections are mechanically coupled to each other via a folding mechanism. Some examples of foldable screens include bi-fold displays, tri-fold displays, etc.

[0040] In this specification, the term "user" generally refers to a person who physically operates a mobile device, and the term "bystander" generally refers to a person who is in physical proximity to the user but does not physically operate the mobile device. It should also be noted that the UI manipulation techniques introduced here are distinct from techniques that simply use less than the entire area to display information (e.g., by using fewer pixels to display, such as simply shrinking a 1900x1200 pixel display to a 1680x1050 pixel display), and that, as discussed here, the UI manipulation techniques utilize visual transformations that intuitively require the user to place the phone in a particular physical device configuration (e.g., folding angle) before the user can properly view the UI. Thus, the term "form factor" in this disclosure and in the context of displays generally refers to how large or small a display should appear visually to the user (i.e., when viewed from the user's perspective), as opposed to a third party (e.g., bystander). In a similar vein, for purposes of discussion herein, unless otherwise made clear by the context of the specification, the terms "seen" or "perceived" are used with respect to the perspective of the user of the device in question, and not from another party.

[0041] It should be noted that the figures presented herein are merely examples to facilitate review of the disclosed UI manipulation techniques, and that they are not drawn to scale. [Application environment]

[0042] FIG. 1A illustrates an environment 100 in which a foldable phone user interface as introduced herein may be implemented. Illustrated in the environment 100 is a user 110 interacting with a foldable screen 120 on a foldable mobile device 130 in the presence of multiple bystanders 140. Due to the presence of bystanders 140 in the environment 100, there is a high possibility that the bystanders 140 may steal the user's information displayed in the user interface of the mobile device 130 (i.e., directly displayed on the foldable screen 120 in a conventional manner). As illustrated in FIG. 1A, the mobile device 130 displays the user's private and sensitive information, where there is a risk to the user's privacy. Therefore, it is necessary to protect the user's 110 information from being stolen by bystanders 140.

[0043] Thus, as discussed in more detail below, the UI manipulation techniques disclosed herein can manipulate the UI in a manner that is substantially visible to a user only when the device is configured / reconfigured by the user into a selected device configuration (e.g., folded at a certain folding angle and / or, in some embodiments, viewed from a certain viewing angle). In other words, the disclosed UI system can manipulate the UI such that when the user device is physically configured at a particular folding angle and / or viewing angle, the resulting UI appears visually at a security-enhancing size and / or angle from the perspective of the user 110.

[0044] In one or more examples, the disclosed UI operation system can determine a level of privacy risk in the user's surroundings by determining the likelihood and potential number of bystanders in the user's vicinity. Additionally or alternatively, the UI system can determine a level of sensitivity of the private and sensitive information presented on the user's mobile device 130. Then, in some embodiments, based on the level of privacy risk in the user's surroundings and / or the level of sensitivity of the private and sensitive information presented on the user's mobile device 130, the UI system can determine a form factor for how the sensitive information is displayed on the foldable screen 120 such that only the user 110 is in a position to get an undistorted view of the sensitive information. In these embodiments, the sensitive information may remain distorted even to the user 110 unless the foldable screen 120 is manipulated to a selected folding angle and / or a selected viewing angle. In other words, in some examples, the UI can be customized to prompt a particular device configuration (e.g., folding angle and / or viewing angle) that can provide an optimal security configuration for the user's current environment 100. [Fold angle]

[0045] FIG. 1B illustrates a folding angle 138 of a foldable phone (e.g., user device 130). As illustrated in FIG. 1B, folding angle 138 is an angle between sections of a multi-section foldable mobile device (e.g., user device 130). More specifically, folding angle 138 is formed by an angle between two axes: a first axis 132 extending along a surface of a first section (e.g., left section) of foldable phone 130, and a second axis 134 extending along a surface of a second section (e.g., right section) of foldable phone 130. The first and second sections of foldable phone 130 are mechanically connected and folded along a hinge axis 136. As illustrated in FIG. 1B, the UI on device 130 is collectively displayed by the left and right sections of the foldable display.

[0046] For ease of discussion, the bi-fold display embodiments described herein focus on horizontal (e.g., left-to-right) folding configurations, but it should be noted that the disclosed technology is equally applicable to those embodiments having vertical (e.g., top-to-bottom) folding configurations. In some of those embodiments, the configurations may be interchangeable, e.g., via manual or automatic screen rotation functionality, so that the configurations can be better tailored to user preferences and current device orientation.

[0047] According to this embodiment, a folding angle 138 can be selected and the UI distorted accordingly such that when the foldable screen (e.g., screen 120) of the foldable phone 130 is folded or manipulated to the selected folding angle (or desired folding angle), the user interface is optimally readable by the user 110 but distorted for the remainder of the vicinity (e.g., bystanders 130). The selected folding angle can also be expressed as an optimal inner angle around which the display screen 120 folds. For example, if the display is a bi-fold display (e.g., single hinge), the folding angle is the inner angle around which the display folds.

[0048] In some embodiments, the desired folding angle may be determined to block the view of the user's private on-screen information based on the sensitivity of the private information displayed in the user interface and / or the presence of potential bystanders in the user's vicinity. For example, if a user wishes to access private or secure data, phone camera data may be used to determine the presence of any potential bystanders in the user's vicinity.

[0049] Once the optimal folding angle of the phone is determined, a user interface can be generated / distorted / customized such that it is readable at the optimal folding angle. The user folds the phone to a certain degree in order to make the user interface readable at the user's position, blocking the view of potential bystanders over the user's personal information. For example, the user wants to view their bank account information while riding a bus. When the user accesses a software application (e.g., the bank's proprietary application) installed on the foldable phone presented here, the phone presents the user with a user interface that appears distorted. The user closes the phone to a narrower angle until the user interface is visually readable from the user's perspective (e.g., edges and fonts do not appear slanted / distorted), while blocking the content from potential bystanders around the user (by closing the phone to a narrower angle). [Viewing angle]

[0050] FIG. 1C illustrates a viewing angle 148 for a user (e.g., user 110) of an example foldable phone (e.g., user device 130). In general, the viewing angle is the angle of orientation of the mobile device 130 relative to the user 110. The viewing angle can also be understood as the eye position of the user 110 relative to the foldable device 130. As shown in FIG. 1C, in the example of a bi-fold phone, the viewing angle 148 can be the angle between a first axis 142 that is perpendicular to the non-folding edge of the device 130 (i.e., 90°) and the line of sight of the user 110. The bi-fold device of FIG. 1C can be folded around a hinge axis 146 that is centrally located (e.g., of the same configuration as the hinge axis 136).

[0051] Note that for convenience, the disclosure herein assumes that the point of gaze (i.e., where the user's eyes focus) is the center of the screen, as shown in FIG. 1C. The point of gaze defines the starting edge from which the field of view 148 may be measured. This field of view definition and / or point of gaze assumption may vary depending on the implementation (e.g., the point of gaze may be at the top or bottom of the screen, or in some implementations the point of gaze may be dynamically determined by a sensor), but the UI manipulation techniques disclosed herein are similarly applicable regardless of changes in the field of view definition and / or location of the point of gaze. [Dynamic vs. static UI adjustments]

[0052] The UI generated by the UI manipulation techniques herein generally has the effect of being viewed by the user (i.e., from the user's perspective) as not visually distorted or distorted only when the foldable screen is folded to the intended folding angle and the screen is viewed from the intended viewing angle. Thus, in some embodiments, the user device may have the option to dynamically or statically adjust the UI distortion. For purposes of discussion herein, the term "static UI manipulation" means that the UI is only manipulated / bent (e.g., in the manner described with respect to FIG. 2A below) to be viewed at an optimal folding angle and / or from an optimal viewing angle, and is "static" in the sense that the UI rendering does not change based on the actual folding angle of the device or the actual position of the user's eye position / point of gaze. In contrast, the term "dynamic UI manipulation" means that the UI rendering changes and is subject to adjustment based on the actual folding angle of the device and / or the actual position of the user's eye position / point of gaze.

[0053] More specifically, as shown in table 150 of FIG. 1D, there are four different UI adjustment configurations: (A) both folding angle UI adjustment and viewing angle UI adjustment are static, (B) static folding angle UI adjustment and dynamic viewing angle UI adjustment, (C) dynamic folding angle UI adjustment and static viewing angle UI adjustment, and (D) both folding angle UI adjustment and viewing angle UI adjustment are dynamic. For embodiments that implement static folding angle adjustment, the generated UI distortion does not need to take into account the actual folding angle of the device; rather, the UI is generated based on the selected optimal folding angle, and it is up to the user to reconfigure the device to the optimal folding angle (e.g., by folding the screen) to see the correct UI. Similarly, for embodiments that implement static viewing angle adjustment, the generated UI distortion does not need to take into account the user's eye position / point of gaze; rather, the UI is generated based on the selected optimal viewing angle, and it is up to the user to adjust the device to a holding position / orientation / etc. and / or move the user's eyes relative to the device so that the device is viewed from the optimal viewing angle in order for the user to see the correct UI.

[0054] In contrast, for embodiments implementing dynamic folding angle adjustment, the UI distortion is dynamically generated and adjusted by considering the actual folding angle of the device, i.e., the UI can deform as the actual folding angle changes. Similarly, for embodiments implementing dynamic viewing angle adjustment, the generated UI distortion considers the user's eye position / gazing point. In other words, the UI is dynamically generated and adjusted based on the actual position / gazing point of the user's eyes. With dynamic viewing angle adjustment, the user does not need to adjust the holding position / orientation / etc., and can generally see the correct UI from any viewing angle available to the user.

[0055] For the sake of brevity, unless otherwise clear by context, the following description of UI operation techniques will mainly focus on the configuration employing static folding angle UI adjustment and dynamic viewing angle UI adjustment, i.e., configuration (B). Here, it is recognized that configuration (B) may be preferred in many field applications for at least two advantages: first, the static folding angle forces the user to change the folding angle, thus facilitating protection of confidential data against bystanders; and second, the dynamic viewing angle, which changes the UI distortion based on where the user's eyes are located, provides convenience and improves usability for the user. In comparison, configuration (A) may be preferred in low-cost and / or low-power implementations where an extra sensor system, such as a user's eye tracking system, may be costly and / or consume too much power. On the other hand, configuration (D) relatively provides the most seamless user experience since the UI remains visible to the user regardless of the user's current folding angle or point of gaze, but it would be less secure than a configuration with a static folding angle because if the UI visibility were the same to the user, the user would be less inclined to make any adjustments (e.g., folding the phone narrower).

[0056] Although configuration (B) is preferred, the disclosed techniques are equally applicable to configurations (A), (C), and (D). In particular, in the following description where determining and generating an optimal folding angle is discussed, the same or similar considerations (e.g., regarding internal and / or external information of the user device) can be applied to determining and generating an optimal viewing angle (e.g., for those implementations involving static viewing angle adjustment). In some embodiments, a user may be given the option to select among different configurations.

[0057] Additionally, note that in some embodiments, the UI distortions may be "updated" based on the determined level of sensitivity of the user's personal information and / or the determined level of privacy risk. These "updates" of UI distortions may also be generally understood as "dynamic" in some sense, since they occur in response to changes in either the sensitivity or surrounding privacy risk display data, or both, but this type of consideration is made clear in context and is not to be confused with the general consideration of dynamic UI operations that are responsive to the user's gaze point and / or fold angle (i.e., parameters that are within the user's control). [User interface operations for protecting confidential data]

[0058] FIG. 2A illustrates an example of how a user interface of a foldable phone as presented herein may be implemented for privacy and confidential data protection. Shown in FIG. 2A is a foldable mobile phone 230 having a bi-fold display 220 that folds around a hinge axis 236. Also shown in FIG. 2A are example UIs 201-208 illustrating four sets of combinations (i.e., the first set is example UIs 201-202, the second set is example UIs 203-204, etc.) of how the foldable mobile phone 230 may be operated for different folds in four different example privacy scenarios - (1) a private space, (2) a relatively safe space, (3) a public but quiet space, and (4) a crowded public space. In the following, the privacy scenarios and their corresponding holding angles are merely for discussion, and depending on the implementation, more or less scenarios may be adopted, and different holding angles may be adopted.

[0059] Example UIs 201, 203, 205, and 207 show UIs as they appear to a user (e.g., user 110) in different privacy scenarios, while example UIs 202, 204, 206, and 208 show actual UIs displayed on bi-fold screen 220. Specifically, the actual UI representations in example UIs 202, 204, 206, and 208 show how a user's private / sensitive information may be laid out on a user interface of a phone having a bi-fold display (e.g., bi-fold display 220). In other words, the "actual UI" representations show exactly how the user interface would appear to a user when foldable phone 230 is fully unfolded in each scenario. In comparison, the "user-perspective UI" representations show how the user's private / sensitive information would visually appear to a user on the user interface of phone 230 when bi-fold display 220 is folded at an optimal folding angle (and, in some embodiments, viewed from an optimal viewing angle) for each of the scenarios.

[0060] Take the first privacy scenario - private space - as an example. The example UI 202 represents the actual UI rendering of the user's private / confidential information on the phone 230 when the bi-fold display 220 of the foldable mobile phone 230 is fully unfolded. In comparison, the user-visible UI, the example UI 201, shows how the actual UI 202 visually appears to the user when two sections (i.e., the left and right sections) of the bi-fold display 220 are fully unfolded, i.e., occupying the entire area of ​​the foldable display 220. It is understood that a user would generally prefer to keep the foldable mobile phone 230 fully unfolded in the user's private space, such as at home, since it fully utilizes the display area of ​​the foldable display 220. It should be noted that in the fully unfolded state, the actual UI (i.e., the example UI 202) is the same as the user-visible UI (i.e., the example UI 201) at the user's optimal viewing angle.

[0061] A second example privacy scenario - an office cubicle - is a location that may generally be classified as relatively safe, but not as safe as the user's home, so it may not be advisable for the user to have the phone 230 fully open, as there is still a small chance that bystanders may view the user's phone. In such a case, having the phone 230 folded somewhat but at a large angle may be sufficient to protect the user's private / sensitive information from being stolen by bystanders. The example UI 203 represents an actual UI rendering of the user's private / sensitive information on the phone 230 for a scenario where the user is located in a relatively safe space. In such an environment, an optimal view for the user may be achieved when the two sections of the bi-fold display 220 are folded at a large angle (e.g., 135° or more). In comparison, the UI visible to the user, the example UI 203, shows how the actual UI 204 appears to the user when the two sections of the bi-fold display 220 are folded at an optimal folding angle for a relatively (but not completely) safe space. It should also be noted that UI203 as seen by the user represents how the actual UI204 appears to the user when the foldable mobile phone 230 is viewed from an optimal viewing angle (see discussion above with respect to FIGS. 1C-1D). UI203 as seen by the user is a user interface that appears smaller than the full area of ​​the display (e.g., compared to UI201) at the optimal folding angle in a relatively safe space, but is undistorted to the user, whereas the same user interface appears distorted to an onlooker (e.g., UI204).

[0062] Similarly, in a third privacy scenario, a location that may generally be classified as a public but still quiet space, such as a park, the user may be better off folding the phone 230 to a narrower angle, especially compared to the second scenario when the user was located in a safer location, such as an office. This is because bystanders are more likely to see the user's phone in a park compared to a safer location, such as an office. In such a case, folding the mobile phone at a narrower angle may protect the user's private / sensitive information from being stolen by bystanders. The example UI 206 is an actual UI rendering of the user's private / sensitive information on the phone 230 for a scenario when the user is located in a public but still quiet space. In such an environment, an optimal view for the user may be achieved when the two sections of the bi-fold display 220 are folded to a narrow angle (e.g., 135°-75°). The UI 205 from the user's perspective illustrates how the actual UI 206 appears to the user when the foldable mobile phone 230 is folded to an optimal folding angle for a public but still quiet space. In this way, at the optimal folding angle, the UI 205 as seen by the user remains undistorted to the user, even though the display area perceived by the user (e.g., compared to the UI 201) becomes smaller than the full area of ​​the foldable display 220. However, the same user interface becomes more hidden to an onlooker due to its narrower folding angle, and appears more distorted to an onlooker compared to the second scenario.

[0063] Finally, in a fourth privacy scenario, a location that may generally be classified as a crowded public space, such as a bus or subway, the user may be better off folding the phone 230 to a tighter angle compared to the previous scenario (e.g., a park). This is because bystanders are more likely to see the user's phone on a bus compared to a park. In such a case, folding the phone 230 to a tighter angle may protect the user's private / confidential information from being stolen by bystanders. The example UI 208 is an actual UI rendering of the user's private / confidential information on the phone 230 for a scenario when the user is in a crowded public space. In such an environment, an optimal view for the user may be achieved when the two sections of the bi-fold display 220 are folded to the narrowest angle (e.g., 75° or less) in all four scenarios. The user-perspective UI 207 shows how the actual UI 208 appears to the user when the foldable mobile phone 230 is folded to the optimal folding angle for a crowded public space. As shown in FIG. 2A, at the optimal folding angle, the UI 207 seen by the user can remain undistorted to the user, even though the viewing area perceived by the user is the smallest of all four (compared to UIs 201, 203, and 205). However, compared to the first three scenarios, the example UI 208 is the most hidden to the bystander due to the narrowest folding angle, and appears to the bystander as the most distorted compared to all previous scenarios. Note that in embodiments where the foldable display is configured to fold at or near the center (e.g., bi-fold display configuration), the user interfaces (e.g., UIs 204, 206, and 208) implementing the present technology may be deformed symmetrically about an axis along the hinge (e.g., axis 236) of the foldable display (e.g., display 220).

[0064] In this manner, the disclosed UI manipulation / distortion techniques provide an intuitive user interface system in which the content of the user interface is visually distorted in the actual UI representation and can only appear undistorted and readable when the display(s) (e.g., display 220) is folded to a particular angle. This allows for protection of the user's private and confidential information from being stolen by bystanders (e.g., bystander 140), since the bystanders can only see distorted user interface displays such as those examples shown as actual UI representations (e.g., example UIs 204, 206, and 208). Furthermore, bystanders are further hindered from stealing the user's private and confidential information, since the user folds or closes the display at a particular folding angle to block the bystander's view when the user is viewing a readable UI such as those examples shown as user-perspective UI representations (e.g., example UIs 203, 205, and 207).

[0065] 2B shows an example of visual guidance indicators 250 and text guidance 252 as they appear in the actual UI versus the UI as seen by the user. As discussed herein, the UI displayed on a foldable or flexible display can be generated based on a selected folding angle, such that the UI will only be visually visible to the user as normal when the screen of the foldable phone is folded to the selected folding angle. The UI generated from this technique is intuitive to the user at least when the phone is not in the selected configuration (e.g., folded to the selected folding angle) because content would otherwise appear abnormal (e.g., distorted or skewed).

[0066] Nonetheless, to further enable a user to intuitively configure and / or orient the phone and reduce reliance on distortion of the displayed content to guide the user, the generated UI may additionally include visual guidance indicators (e.g., guidance indicators 250) that guide the user to manipulate the foldable display to a selected folding angle. In some implementations, the visual guidance indicators 250 may be displayed at the top of the display or in a location that does not obstruct normal content in the UI from the user. The visual guidance indicators 250 may be a selected geometric shape, such as a circle, a square, a triangle, a diamond, etc. With implementation of the disclosed UI manipulation techniques, the visual guidance indicators will appear to the user as the selected geometric shape (e.g., a circle as shown in the "UI from the user" representation of FIG. 2B) only when the foldable display is manipulated to the optimal folding angle. For example, if the selected geometric shape is a circle, the actual UI may appear to the user as a shape other than a circle (e.g., a tilted ellipse as shown in the "Actual UI" representation of FIG. 2B) when the foldable display is not manipulated to the optimal folding angle. In other words, the geometric shape on the intuitive user interface appears as a circle only when the foldable display is manipulated to the optimal folding angle. In some embodiments, the actual UI may include an additional hint on the display (which may also be part of the text guidance 252 described below) that indicates to the user what the selected geometric shape is. The user can then adjust the phone to a tighter angle until the oval appears as a circle. The user interface is readable and visually normal from the user's perspective, but bystanders in the user's vicinity would otherwise have content blocked or visually distorted.

[0067] Additionally or alternatively, the generated UI may include visual guidance indicators that guide the user to manipulate their grips according to the grip manipulation techniques discussed with respect to FIGS. 8-11D. Thus, in one or more embodiments, the generated UI may include visual guidance indicators for a UI manipulation technique and visual guidance indicators for a grip manipulation technique. However, the visual guidance indicators may be generated independently of each other. The visual guidance indicators for the grip manipulation technique may be generated independently of the UI manipulation technique (i.e., without visual distortion) or in conjunction with the UI manipulation technique (i.e., with visual distortion).

[0068] Additionally or alternatively, one or more embodiments of the UI manipulation system herein may include text guidance (e.g., text guidance 252) on the user interface to guide the user to change the current folding angle to the optimal folding angle. For example, the text guidance 252 may instruct the user with words such as "Open the foldable display more" or "Close the foldable display more." In some variations, the text guidance may be displayed in a currently readable font (i.e., in an undistorted manner), such as an instruction to "Close the phone until the oval becomes a perfect circle."

[0069] In this aspect, visual guidance 250 and / or textual guidance 252 can guide the user to fold the phone just enough to block a potential bystander's view of the user's personal information and make the user interface readable from the user's location. As a practical example, a user wants to view their bank account information while riding a bus. When the user accesses that information on the foldable phone, the phone presents the user with a UI that appears distorted. At the top is an oval, now written in an easy-to-read font, with instructions: "Fold your phone to a tighter folding angle until the black oval becomes a perfect circle (and the UI appears straight and horizontal)." The user closes the phone to a tighter angle against the phone until the oval appears as a circle. After the adjustment, the user interface is now fully readable from the user's location, but the content above it is blocked from potential bystanders.

[0070] Further, in some examples, the UI operation system can require that the user device be placed in an optimal configuration before sensitive information can be displayed to the user on the UI. For example, the UI operation system can be configured to display only auxiliary UI elements and actively block, blur, mask, otherwise prevent, or simply prohibit the display of sensitive data on the screen of the user device unless / until the foldable display is manipulated to an optimal device configuration (e.g., optimal folding angle). For example, the UI can only display guidance information (e.g., visual indicators 250 and / or textual guidance 252) to the user until the user follows the guidance instructions and reconfigures or manipulates the foldable phone to the selected optimal folding angle. Only then does the UI operation system display sensitive information on the UI (e.g., in the manner described above).

[0071] FIG. 2C illustrates additional optional implementation details for some examples of UI manipulation techniques for a foldable phone. As discussed herein, because at least some examples of UI manipulation techniques introduced have a visual effect of reducing or shrinking the display area of ​​the foldable display as perceived by the user (e.g., comparing UI 201 to UIs 203, 205, and 207), in some embodiments, the content of the display may be automatically adjusted based on the determined effective form factor for the foldable display. Specifically, in various embodiments, the original layout of the sensitive information may be changed or rearranged to an alternative layout according to the determined form factor, where the alternative layout is suitable for a smaller form factor compared to the total area of ​​the foldable display. For example, the original layout (e.g., which may be designed for a larger display such as a desktop / tablet display) may be changed to an alternative layout (e.g., which may have a different font size, graphics, and / or navigation arrangement) designed for a mobile device with a small or smaller screen.

[0072] 2C are two example UIs 270 and 272, where example UI 270 shows the original layout of content to be displayed when the screen is fully open. When a device implementing the UI manipulation techniques is folded, the displayed layout may automatically change to an alternative layout shown by example UI 272. Additionally or alternatively, additional UI elements, such as scrollbar 274, may be further added to the displayed UI to assist the user in navigating content in the simulated reduced screen size. [Example system components and data flows]

[0073] FIG. 3 illustrates a functional block diagram of an example UI manipulation engine 300 that can be utilized to implement the foldable phone UI manipulation techniques presented herein.

[0074] An example UI manipulation engine 300 may be implemented in and executed on a user's mobile device (e.g., mobile device 130, 230). As shown in FIG. 3, the UI manipulation engine 300 may include a UI sensitivity extraction system 310, an ambient state evaluation system 320, an optimal display configuration finder 330, an actual display configuration finder 340, a user state evaluation system 350, a UI distortion generator 360, and an auxiliary UI element generator 370. It should be noted that the components shown in FIG. 3 are merely exemplary, and certain well-known components, such as a central processing unit (CPU), memory modules, screen display(s), and communication circuits, are not shown for simplicity. It should be noted that one or more of the systems, subsystems, components, and / or modules introduced herein may be implemented as or include software applications. However, any suitable combination of these systems / subsystems / modules may be implemented as hardware or firmware components to achieve the same or similar functions. Similarly, the example components in FIG. 3 are shown for ease of discussion herein, and more or fewer components may be used, and components may be combined or divided, depending on the actual implementation.

[0075] According to some implementations, the UI operations engine 300 can begin operation upon receiving an indication that sensitive information is to be displayed (e.g., on a foldable display of a mobile device). For example, an application programming interface (API) can be implemented for the engine 300 such that the engine 300 can receive software function calls (e.g., from a software application) as instructions. In a variant, the engine 300 can be implemented as part of a function set provided by an operating system (OS), in a system software library, or the like. In additional or alternative embodiments, the engine 300 can be standalone software that resides in memory and intercepts / detects the sensitive information to be displayed.

[0076] After receiving an indication that there is sensitive information to be displayed, the UI manipulation engine 300 determines an optimal display configuration therefor, and to do so, the engine 300 considers one or more factors, including those based on external and / or internal information. More specifically, there are generally two categories of information that may affect the determination of how small the perceived display screen should be (i.e., its form factor) in accordance with the introduced UI manipulation techniques: (1) information that is internal to the phone, and (2) information that is external to the phone. The former (i.e., internal information) can be extracted by the UI sensitivity extraction system 310, and the latter (i.e., external information) can be extracted by the ambient condition assessment system 320. [A.UI Sensitivity Extraction System]

[0077] Figure 4A is an example data flow 400 for implementing a foldable phone UI operation technique with some example components of the UI operation engine of Figure 3. Data flow 400 illustrates an example of how data may flow between various blocks within a UI operation system (e.g., engine 300). With simultaneous reference to Figures 3 and 4A, data flow 400 will now be described in conjunction with a functional block diagram of UI operation engine 300.

[0078] The UI sensitivity extraction system 310 can determine the privacy sensitivity of data displayed on a user device and output the UI sensitivity data. In other words, the UI sensitivity extraction system 310 can determine the level of sensitivity of information presented on a user's mobile device.

[0079] In some examples, the UI sensitivity data may describe various sensitivity levels of different features currently displayed on the screen. Examples of high sensitivity items may include password entry boxes, bank account information, credit card information, etc. In some implementations, intermediate sensitivity may include personal information such as social security numbers, birth dates, home addresses, or phone numbers. The UI sensitivity extraction system 310 may also identify low sensitivity items such as public domain information, generic website text, or images. In many embodiments, the user may have the option to adjust or edit these items to the user's own preferences. Additionally or alternatively, the UI sensitivity data may describe an overall sensitivity rating of the screen that matches the most sensitive features on the page.

[0080] According to some embodiments, the UI sensitivity extraction system 310 can obtain UI sensitivity data from local and / or remote resources. Examples of these resources can include lookup tables or databases in which sensitivity ratings for commonly found software applications / components / functions can be recorded. For example, if the software mobile application is a banking application, if the application window shows a password entry screen, if a web page is triggered that provides a specific function (e.g., displaying a bank statement), or if the UI element includes a password entry field, the resource(s) can be identified as high sensitivity. As an additional or alternative embodiment, the UI sensitivity extraction system 310 can obtain UI sensitivity data via metadata, such as message metadata regarding who the sender is, metadata regarding the nature of the message, the urgency of the message, and / or the address of the recipient (e.g., a sensitivity distinction based on work or private email). Additionally, in some variations in which an API is implemented for the UI operations engine 300, the UI sensitivity extraction system 310 can receive UI sensitivity data from the API (e.g., via one or more parameters of a function call).

[0081] In one or more embodiments, the UI sensitivity extraction system 310 can obtain UI sensitivity data through content analysis. In a particular example, the UI sensitivity extraction system 310 can determine a level of sensitivity based on the presence of a keyword (e.g., “highly confidential”) in a sent or received message or through analyzing graphic / photographic content (e.g., by using known image processing and recognition algorithms). Some embodiments of the UI sensitivity extraction system 310 can also generate UI sensitivity data by using machine learning techniques (e.g., supervised machine learning). For example, the UI sensitivity extraction system 310 can be initially trained with a set of training data labeled with its sensitivity to establish a UI sensitivity determination data model. Examples of training data can include actual content (e.g., text, documents, images, or videos) and their corresponding sensitivity ratings. In this manner, the UI sensitivity extraction system 310 can generate sensitivity ratings of sensitive UI content on the fly.

[0082] Additionally, some variations of the UI sensitivity extraction system 310 may generate UI sensitivity data through user sentiment analysis, such as by inferring a likely sensitivity level from a user's mood changes. For example, if a user behaves more secretively and changes facial expressions, mood, etc., as the user interacts with an application, the UI sensitivity extraction system 310 may infer a higher level of sensitivity for that application than for general applications. These inferential sensitivity level determinations may be based, for example, on gesture recognition, facial expression recognition, and / or electroencephalography. [B. Surrounding Condition Assessment System]

[0083] The ambient condition assessment system 320 includes a set of system hardware, firmware, and / or software components that may be used to determine and analyze the privacy conditions of a user's surroundings. As shown in FIG. 3, the ambient condition assessment system 320 includes three exemplary components: an ambient condition sensor 322, an ambient condition data preprocessor 324, and an ambient condition finder 326.

[0084] More specifically, the ambient condition assessment system 320 may be used to identify a level of privacy risk in the mobile device's ambient environment, for example, based on readings from the ambient condition sensor 322. This identified privacy risk level may be used by the optimal display configuration finder 320 and the UI distortion generator 360 to determine an optimal form factor of the user's private / sensitive information to be displayed on the mobile device. In some embodiments, the form factor of the user's private / sensitive information to be displayed on the mobile device may be inversely correlated to the identified level of privacy risk. That is, the higher the level of identified privacy risk, the smaller the form factor of the UI as seen by the user will be. For example, as described with respect to FIG. 2A above, there may be four levels of privacy risk - private space, relatively safe space, public but still quiet space, and crowded public space - and the more crowded the surroundings, the smaller the UI will be as perceived by the user.

[0085] The ambient condition sensors 324 can include one or more sensors that can be used to obtain information about the user's surroundings (i.e., "ambient data"). In various examples, ambient data can include visual images or records of the user's location, specific geographic location information (e.g., Global Positioning System (GPS) coordinates), audio information (e.g., background sound data and / or audio data from the surroundings), device connectivity information (e.g., information for detecting devices of people in the user's surroundings, such as connectivity data), and / or other suitable data that can be used to determine the presence and status of people in the user's surroundings. To obtain applicable ambient data, examples of ambient condition sensors 324 can include sensors such as rear and / or front cameras or other light-based (e.g., infrared) imaging technologies that can generate a light feed from the user's location, GPS sensors that can generate satellite position information, microphones or audio sensors that can generate audio signals collected from the user's location, and / or wireless network transceivers (e.g., Bluetooth) that can generate a signature of devices within the vicinity of the user's mobile device (e.g., 10 meters based on the range of the transceiver). In some embodiments, these sensors may be on-board the user's mobile device or may be separately connected to the device.

[0086] Specifically, according to one or more embodiments, the ambient condition sensor 322 can then transmit the ambient data to the ambient condition data pre-processor 324, which can pre-process the ambient data (e.g., to increase the signal-to-noise ratio (SNR) to an appropriate quality) for subsequent analysis. Additionally, the ambient condition data pre-processor 324 can apply various applicable types of pre-processing to the ambient data depending on the type of data collected. For example, in some embodiments, the ambient condition data pre-processor 324 may pre-process optical data generated from an optical sensor (e.g., a camera) in the ambient condition sensor 322 to correct for scene illumination, focus, vibration, motion artifacts, reflections, and other characteristics. Additionally, in many of these embodiments, the ambient condition data pre-processor 324 can pre-process the optical data using machine vision techniques to identify and label (e.g., by adding metadata or tags) key features such as people or faces.

[0087] Additionally, some implementations of the ambient data pre-processor 324 can pre-process the audio data to further improve audio quality by removing background noise (e.g., wind or road noise), amplifying certain audio features (e.g., voices), and increasing the SNR. Similar to the video pre-processing embodiments described above, in certain embodiments the ambient data pre-processor 324 may pre-process the audio data to add metadata or tags to key features such as clear speech or phrases and words heard.

[0088] Additionally, in certain embodiments, the surrounding state data pre-processor 324 can pre-process the connection data to identify, count, and potentially locate unique device signatures in the user's vicinity. For example, the surrounding state data pre-processor 324 can utilize information contained in Wi-Fi management frames, requests, beacons, and pings to identify, count, and / or locate unique device signatures and / or geographic locations. The surrounding state data pre-processor 324 can also pre-process GPS / satellite data in some examples to identify certain qualitative characteristics (e.g., store names, road names, or landmarks) of the location where the user is currently located. These GPS / satellite / location data can be considered in combination with the phone's existing map services (e.g., Google Maps). In some implementations, the pre-processing of the location data can utilize machine learning techniques (e.g., to filter and classify features).

[0089] The ambient condition finder 326 can then assess the current state of the user's surroundings (i.e., the "ambient condition") by analyzing the preprocessed ambient data generated by the ambient condition data preprocessor 324. The ambient condition finder 326 can then determine a level of privacy risk in the user's surroundings based on the user's ambient condition. In one or more implementations, the ambient condition finder 326 can extract relevant information from the ambient data in determining the ambient condition.

[0090] In particular, in one or more examples, the ambient data can include the number of other people (e.g., bystanders) in the vicinity. In some of these examples, the ambient condition finder 326, in determining the number of other people in the vicinity, can perform a count of different people feature tags added to the camera feed during preprocessing by the ambient condition data preprocessor 324. In other embodiments, the ambient condition finder 326 can determine the number of other people in the vicinity by using machine vision facial recognition technology to count the number of nearby users. In a variation, the number of other people in the vicinity can be determined by counting the number of nearby devices detected via Bluetooth technology (or other suitable short-range wireless connection technology). Additionally or alternatively, the ambient condition finder 326 can determine the number of bystanders by counting the number of unique human voices identified during preprocessing of the ambient data. Certain variations of the embodiments can provide a simple estimate of the total sound level in the vicinity based on the change in audio peak amplitude over a given time interval in the ambient data.

[0091] Additionally, in some embodiments, the surrounding data may include the spatial distribution and line of sight of people nearby. In some of these examples, the surrounding condition finder 326 may detect this information using camera data and machine vision techniques to, for example, identify that someone is standing behind the user and facing the user device (which may be a security threat) as opposed to someone standing to the left or right of the user but perhaps facing away from the user device (which may be a lesser security threat). In many implementations, the surrounding data may include predicting how crowded / public the user's location is based on known properties associated with the user's location, such as a predictive evaluation of the user's GPS coordinates. For example, the surrounding condition finder 326 may detect from the user's device's GPS data that the user is in a public location, thereby inferring that other people are likely to be nearby. In another example, the surrounding condition finder 326 may discover through GPS data that the user is in a field in a large state park, thereby inferring that other people are unlikely to be nearby. In some embodiments, the surrounding condition finder 326 may further associate known properties of the user's location from the user's device's network connection. For example, when the user device connects to “Home Wi-Fi” or “Car Wi-Fi”, the surrounding condition finder 326 may determine a high privacy level (i.e., a low risk level), and in contrast, when the user device connects to “Public Wi-Fi” or “Airport Free Wi-Fi”, the surrounding condition finder 326 may determine a low privacy level (i.e., a high risk level).

[0092] Based on the ambient state data and through the exemplary methods described above, the ambient state finder 326 can generate an ambient state output indicating how likely it is that the user's data will be exposed when using the user device. In other words, the ambient state output from the ambient state finder 326 can communicate a level of privacy risk in the user's vicinity to other components of the UI operations engine 300. In some embodiments, the communication of the ambient state to subsequent system components can be in the form of a specific description of a number of people that provide a data exposure risk to the user, including their location, line of sight angle, and distance to the user, among other information. In a variation, the communication of the ambient state to subsequent system components can be in the form of a generic metric regarding the user risk (e.g., "data exposure risk"), and the ambient state finder 326 can apply weighting to aspects of the analyzed ambient data to determine the likelihood that the user's data may be exposed at the current location. In some additional or alternative examples, the ambient state finder 326 can directly generate an assessment or level of privacy risk in the user's vicinity. [C. Optimal Display Configuration Finder]

[0093] The optimal display configuration finder 330 can determine an optimal physical configuration (i.e., an "optimal display configuration") of a foldable or flexible screen to prevent viewing by other people nearby. In particular, the optimal display configuration finder 330 can use (1) the level of sensitivity of private and confidential information presented on the user's mobile device and / or (2) the level of privacy risk of the user's surrounding environment to identify a physical configuration in which the probability of the screen's contents being viewed by unscrupulous bystanders is reduced, minimized, or even eliminated. In other words, the optimal display configuration finder 330 can use UI sensitivity data (e.g., generated by the UI sensitivity extraction system 310 described above) and / or surrounding conditions (e.g., generated by the surrounding conditions assessment system 320 also described above) to determine an optimal privacy configuration (e.g., folding angle) of the display to avoid the screen's contents being peeked by bystanders.

[0094] Specifically, in some embodiments, the optimal display configuration finder 330 can determine the optimal folding angle based on information inside the user device, i.e., a determined level of confidentiality of the user's private information generated by the UI confidentiality extraction system 310. In some other embodiments, the optimal display configuration finder 330 can determine the optimal folding angle based on information outside the user device, i.e., a determined level of privacy risk generated by the ambient condition assessment system 320. Additionally or alternatively, the optimal display configuration finder 330 can determine the optimal folding angle based on both a determined level of confidentiality of the user's private information and a determined level of privacy risk. In those embodiments that perform static viewing angle adjustment, the optimal display configuration finder 330 can further determine the optimal viewing angle.

[0095] Some embodiments of the optimal display configuration finder 330 can then determine an optimal folding angle, i.e., an optimal internal angle at which the display is folded (see FIG. 1B and related discussion above). In those embodiments in which the user device's display includes a single joint (e.g., a single-hinged display or multiple displays), the optimal display configuration can be the optimal folding angle at which the display is folded. For example, an acute folding angle may provide higher display privacy (at the expense of reducing the perceived size of the display) and is optimal for high privacy risk areas and / or high sensitivity content, whereas an obtuse folding angle may provide lower display privacy while providing a larger perceived size of the display and is optimal for low privacy risk areas and / or low sensitivity content.

[0096] The optimal display configuration can vary depending on the physical characteristics of the user device. For example, in some embodiments, the device may be a device with two or more folding sections designed with a specific pre-defined folding configuration. In such devices, the folding angle and rotation direction of the screen may be limited, and therefore the optimal display configuration may be a combination of the folding angle plus a vector of a specific rotation direction for each folding section. Furthermore, in certain examples, the device may be a device with a generally flexible display fabric. Such a device may be a single piece folding screen with a dynamically configurable folding behavior that is not limited to a pre-defined folding behavior. In such devices, the folding angle and rotation direction of the screen are less limited. In this way, the optimal display configuration may include a specific, selected shape, and the configuration may further include, for example, the direction in which the screen should be folded and the location of the center of folding (e.g., the fold line, etc.) on the screen surface. This combination of folding angle and fold line can be utilized, for example, by the auxiliary UI element generator 370 (described below) to guide the user to fold the display fabric into the selected shape. In this context, many examples of fold lines can be similar to the hinge axes introduced herein for foldable (eg bi-fold) displays.

[0097] It should be noted that in some examples, data from an accelerometer / gyroscope sensor built into the user device can be used to identify how the user is currently holding the device, through which the optimal display configuration finder 330 can determine whether the optimal folding angle can be physically achieved by folding one or both sides of the device.

[0098] After the optimal display configuration is determined, it may be output by optimal display configuration finder 330. For purely static UI operation embodiments (i.e., where the UI does not deform / change with changes in actual folding angle or user's eye position), e.g., configuration (A) above, the optimal folding angle and / or viewing angle may be output to UI distortion generator 360 for UI rendering and display. [D. Actual Display Configuration Finder]

[0099] FIG. 4B is an example data flow 402 for implementing a foldable phone UI operation technique using some example components of the UI operation engine of FIG. 3. Data flow 402 illustrates one example of how data can flow between various blocks within a UI operation system (e.g., engine 300). With simultaneous reference to FIG. 3 and FIG. 4B, data flow 402 will now be described along with a functional block diagram of UI operation engine 300.

[0100] As introduced above, many embodiments of the disclosed UI manipulation techniques employ dynamic UI manipulation, i.e., UI distortions change or deform when the user's actual point of gaze changes (e.g., in configuration (B)), or when the actual folding angle of the user device changes (e.g., in configuration (C)), or both (e.g., in configuration (D)). One or more system components (e.g., real display configuration finder 340, or user state assessment system 350) on which these embodiments rely as a basis for these dynamic UI adjustments are introduced below.

[0101] The actual display configuration finder 340 can determine the current configuration (or "actual display configuration") of the user device. For example, the actual display configuration finder 340 can determine the actual display configuration using device data obtained from one or more display configuration sensors built into the device. Examples of information obtained from these display configuration sensors can include the current folding angle and / or folding orientation configuration, device position, and / or device orientation (e.g., tilt angle). In some examples, the device orientation can be obtained by data from an accelerometer, gyroscope, or other suitable sensor carried by the user device. These data can be used to determine how the user is currently holding the device and / or how the user should move the mobile device to achieve an optimal viewing angle (e.g., for embodiments in which the viewing angle adjustment is static).

[0102] In some embodiments, the display configuration sensor used by the actual display configuration finder 340 to collect device data may include a mechanical sensor. For example, in a two-panel (e.g., two-section or bi-fold) flexible display, the mechanical sensor may be integrated into the hinge joining the two panels. In a particular example, the sensor may include a flexible electronic material that changes resistance as the hinge closes and opens, so that it can measure the folding angle of the screen. The folding angle data may be used to determine how the foldable screen is currently folded and / or how the user should reconfigure the foldable screen to achieve an optimal folding angle (e.g., for embodiments in which the folding angle adjustment is static).

[0103] Generally, for flexible displays (e.g., fabric displays), some embodiments of the actual display configuration may include a sufficiently high density of display distortions at multiple points across the surface of the display to describe the surface distortion across the entire surface. For example, in many of these embodiments, the actual display configuration may include data representing a vector field having scalar magnitude and angle of distortion for each point on the surface.

[0104] Additionally, some variations of the display configuration sensor may also include one or more cameras. For example, in general flexible display embodiments, optical data collected from a camera (which may be built into the device or obtained from a third-party device) may be used by the real display configuration finder 340 to create a normal map or vector field of the current device surface. In some additional or alternative embodiments, the camera(s) may also be used by the real display configuration finder 340 to determine the relative device orientation and estimate the device's spatial position. Furthermore, in some embodiments, the real display configuration finder 340 may utilize camera data and image processing techniques to capture the real display configuration in 3D, which may result in a 3D model file, e.g., an STL file.

[0105] In various implementations, the display configuration sensor may include a time-of-flight (ToF) sensor that measures distance to an object using the time it takes for light to travel to the object and reflect back from the object. For example, an integrated ToF sensor may be used by the real display configuration finder 340 to obtain device location by performing distance measurements between the device and a person or other physical feature. Additionally or alternatively, the display configuration sensor may include radar. For example, radar or other object or motion sensing technology may be used by the real display configuration finder 340 to determine spatial relationships between display elements or between display elements and a user. [E. User Status Evaluation System]

[0106] The user state assessment system 350 may be a combination of system components used to determine and analyze a user's physical state. The user state assessment system 350 may include example components such as a user state sensor 352 and a user state finder 354.

[0107] The user state sensor 352 includes one or more sensors that may be used by the user state assessment system 350 to obtain data regarding the user's state (or "user state"). More specifically, examples of user states that may be obtained by the user state sensor 352 may include a user gaze point, which is the position on the screen where the user is focusing, a head angle, which is the angle of the user's head relative to the display, a viewing angle, which is the angle of the user's eyes relative to the display, and / or a head orientation, which is the orientation of the user's whole body relative to the display. Depending on the implementation, the user state sensor 352 may be a dedicated, eye-tracking sensor system, or may be a combination of sensors on-board the user device. Exemplary components of the user state sensor 352 may include, for example, one or more front-facing cameras, a motion sensor (which may be on the user device or worn by the user, such as a smart band), a suitable projector, and / or a time-of-flight sensor. For example, a projector can project a particular light or light pattern (e.g., infrared or near infrared) onto the user's eyes, a camera can take an image of the user's eyes and the pattern, and then image processing and machine vision techniques can be applied to determine the user's eye position and gaze point. In another example, a time-of-flight sensor can detect the user's distance from the screen. [F.UI Distortion Generation Section]

[0108] The UI distortion generator 360 generates visual distortions (also referred to herein as "UI distortions," "UI transformations," or "UI manipulations") in the user interface. The UI distortions manipulate the visibility of the displayed UI such that an optimal view of the user interface is obtained only from the user's current viewpoint at the optimal display configuration (e.g., optimal folding angle). Visibility of the display from any other bystander's viewpoint is reduced (due to the closed folding angle) and obscured (due to the visual distortions), making it less likely that bystanders will miss and thereby steal the user's personal information from the user's device.

[0109] Depending on the implementation, the UI distortions can include a combination of different transformations and distortions. For example, UI distortion may include a series of transformations (e.g., translation, rotation, or tilt) applied to the user interface. According to a number of the present embodiments, the UI distortion generator 360 can apply "optical illusions" to the user interface (e.g., such as those discussed in FIG. 2A) that can render the UI and target information on the UI readable only from a desired angle and / or distance. For certain embodiments having "holographic display" hardware, the UI distortion generator 360 can generate a set of instructions including, for example, to which angles the pixel lights of different pixels should be adjusted so that privacy-sensitive UI elements are only visible from certain angles and / or physical device configurations. As discussed above with respect to FIG. 2A, the type of distortion and the degree of distortion take into account the level of sensitivity of the displayed information. The transformations or distortions applied to high sensitivity UI content can be more extreme such that the UI content is much less readable to those other than the target user (e.g., UI example 207 compared to UI example 203).

[0110] In certain examples, the UI distortion generator 360 takes into account user state data and / or actual display configuration when determining and generating the UI distortion. For example, using user state data (e.g., a user's gaze point), the UI distortion generator can adjust the viewing angle of the UI distortion so that the UI distortion remains easy for the user to see as the user's eye position moves. Using the actual display configuration, many embodiments of the UI distortion generator 360 can also continuously modify the UI distortion in response to any changes in the folding angle. In this way, the perceived UI form factor (e.g., size) generated through the UI distortion can be updated and adjusted (e.g., in a real-time or near real-time manner) in response to new actual display configurations and / or user states.

[0111] Additionally, some embodiments of the UI distortion generator 360 can generate instructions for the device hardware to compensate for the actual display configuration. In some implementations, the UI distortion generator 360 may generate instructions to increase pixel brightness in certain parts of a UI element to be viewed, for example, to compensate for visual degradation due to viewing the screen at the outer edge of the viewing cone. In other variations, the UI distortion generator 360 may generate instructions for recalibrating the device's 3D control gestures so that the gestures still work in the new aspect ratio, UI positioning, distortion of UI elements, etc., in the new UI.

[0112] Additionally, for improved user experience and usability, some embodiments of the UI distortion generator 360 can modify one or more characteristics of the UI. For example, the UI distortion generator 360 may modify the dimensions of user interface features. Additionally or alternatively, the UI distortion generator 360 can modify the positioning of UI elements on the display. For example, the UI distortion generator 360 may move elements from the edge of the display to the center to reduce the amount of distortion required for extreme folding. In some embodiments, the UI distortion generator 360 may modify many user interface characteristics, including, for example, color, brightness, or pixel distribution. In some examples, the UI distortion generator 360 can rearrange the UI to a layout that is appropriate for a smaller display (i.e., as perceived by the user) when the device is closed to a narrower folding angle (such as the example described above with respect to FIG. 2C). [G. Auxiliary UI element generation section]

[0113] FIG. 4C is an example data flow 404 for implementing a foldable phone UI operation technique using some example components of the UI operation engine of FIG. 3. Data flow 404 illustrates an example of how data can flow between various blocks within a UI operation system (e.g., engine 300). With simultaneous reference to FIG. 3 and FIG. 4C, data flow 404 will now be described along with a functional block diagram of UI operation engine 300.

[0114] The auxiliary UI element generator 370 can generate UI elements or functions and / or hardware enabling functions (e.g., standalone LED indicators) that can be used to guide the user to achieve an optimal display configuration. More specifically, in some examples, the auxiliary UI element generator 370 can select to display one or more auxiliary UI elements (e.g., using the UI distortion from the UI distortion generator 360 and / or the optimal display configuration parameters from the optimal display configuration finder 330) such that characteristics of the auxiliary UI elements can change to indicate to the user when the user has correctly manipulated the folding screen to achieve an optimal display configuration (e.g., an optimal folding angle).

[0115] More specifically, in some embodiments, the auxiliary UI element may include dimensional changes to a shape or pattern that the user may intuitively understand, such as distortions to the UI itself and / or additional visual guidance indicators (e.g., a selected geometric shape). For example, the auxiliary UI element may include a distorted circle (or ellipse, e.g., indicator 250 in FIG. 2B) that becomes fully round only when the screen is changed to the optimal display configuration. In some examples, additional text guidance may be generated on the display that shows the user what the selected geometric shape is (e.g., a circle). To provide more useful assistance to the user, one or more embodiments provide that this additional text guidance is displayed in an orientation that the user can see without (or before) the device being configured (e.g., folded) into the optimal display configuration (e.g., at the optimal folding angle) - such an example is shown as text guidance 252 in FIG. 2B. The content of the text guidance may be generated in some implementations based on the difference between the actual display configuration (e.g., the current folding angle) and the optimal display configuration (e.g., the selected folding angle). For example, text guidance can instruct the user to change the folding angle, i.e., more open or more closed. In some examples, images can be used as a substitute for geometric shapes.

[0116] As an additional embodiment, the auxiliary UI elements may include hardware enablement features such as an LED indicator when the display is in the optimal display configuration. In a variation, a holographic display indicator may be included, which can display the correct color or brightness when the display is in the optimal display configuration. Additionally or alternatively, the auxiliary UI elements may include a pattern of pixels, or points of illumination, that will only appear as a particular shape or image once the display is in the optimal display configuration. For example, in a fabric-like display, the pixels may appear as a machine-readable one- or two-dimensional barcode (e.g., a QR code) only when the fabric is manipulated into the optimal shape, thereby serving as confirmation that the optimal display shape has been achieved.

[0117] Further, in some variations, the auxiliary UI element generator 370 can display only auxiliary UI elements and actively block, blur, mask, otherwise prevent, or simply prohibit the display of sensitive data on the screen of the user device unless / until the foldable display is manipulated to an optimal device configuration (e.g., optimal folding angle). For example, the auxiliary UI element generator 370 can control the UI distortion generator 360 such that UI displaying sensitive material is blocked and only guidance information (e.g., visual indicators 250, text guidance 252 (in FIG. 2B), and / or LED indicators as described above) is shown to the user, and thereafter, only if the user follows the instructions and changes the foldable phone to the selected optimal folding angle, the UI manipulation engine 300 shows the sensitive information on the UI (e.g., in the manner described above with respect to FIG. 2A). [Methodology]

[0118] 5 shows a flow chart illustrating an example method 500 for implementing UI manipulation techniques for a foldable phone. Method 500 can be performed by various components (e.g., components 310-370; FIG. 3) of a UI manipulation engine (e.g., engine 300; FIG. 3) for UI distortion and rendering on a user device (e.g., device 130, 230; FIGs. 1A, 2A) to protect sensitive, private information and data on a foldable UI display from onlookers. Method 500 is introduced with simultaneous reference to FIGS. 2A-2C and 3.

[0119] First, the UI operations engine can receive (510) an instruction to display sensitive information on a display (e.g., display 220) of a mobile device (e.g., device 230). One example of such an instruction can be a software function call. In some examples, the mobile device can include a foldable display. Next, the UI operations engine can determine (520) an optimal display configuration for how the sensitive information should be displayed. In many embodiments, the UI operations engine first identifies (522) a level of sensitivity of the information to be displayed on the user's screen, and then determines a form factor for how the sensitive information should be displayed. Additionally or alternatively, the UI operations engine can determine (524) a privacy risk (or how crowded) the surrounding environment of the user device is when determining the form factor for the perceived UI. The form factor, as described above, generally includes the perceived display size, i.e., the size of the display as seen by the user when the user manipulates (e.g., folds) the device into the optimal device configuration (e.g., by folding a bi-foldable display to the optimal folding angle) after UI distortion techniques. In general, the more sensitive the data being displayed or the higher the privacy risk of the surrounding environment, the smaller the perceived form factor (e.g., size) of the displayed UI. That is, the form factor may be inversely correlated to the risk from the surrounding environment and / or the sensitivity of the displayed data. In accordance with one or more embodiments of the disclosed UI manipulation techniques, the determined form factor may be smaller than the full area of ​​a foldable display, for example, for privacy scenarios where bystanders may be present.

[0120] The UI operations engine can then generate a UI to be displayed on the foldable display based on the determined form factor (530). For example, the UI operations engine can receive the actual phone configuration (532), and the UI displayed by the UI operations engine displays the sensitive information in the form factor (e.g., perceived size) but is visually distorted to the user unless the foldable display is manipulated to a selected physical phone configuration (e.g., either a selected folding angle and / or in some embodiments a selected viewing angle). As shown in the example UI of FIG. 2A, in foldable display embodiments, the UI may be collectively displayed by multiple sections of the foldable display. In certain implementations, the foldable display may be a bi-fold display.

[0121] Specifically, in one or more embodiments with a foldable display, the UI distortion is generated such that the user interface appears distorted to the user unless the display section of the mobile device is folded to the folding angle. That is, after the optimal folding angle is determined, the user interface appears distorted to the user unless the display section of the mobile device is folded to the folding angle. In one or more examples, the UI is deformed symmetrically about an axis along the hinge of the foldable display (e.g., similar to those UI examples 204, 206, and 208). In other examples, the UI distortion engine can receive a current orientation state of the mobile device, for example, from one or more orientation sensors on the mobile device, and can further adjust the UI in response to the received current orientation state of the mobile device.

[0122] Further, in some embodiments, the UI operations engine can detect a user state, which may be, for example, a user's gaze point (534). Based on the user state, the UI operations engine can determine a viewing angle at which the user interface will appear distorted to the user unless the mobile device is pointed at the viewing angle relative to the user (e.g., in those embodiments that perform dynamic viewing angle adjustments). In some of these examples, the UI operations engine can determine the user's eye position and adjust the UI in response to the detected position of the user's eyes. In certain implementations, the UI operations engine can employ an eye detector on the mobile device facing the user and further adjust the UI such that a new optimal viewing angle for the UI reflects the detected position of the user's eyes.

[0123] 6 is a flow chart illustrating another example method 600 for implementing UI manipulation techniques for a foldable phone. Method 600 can be performed by various components (e.g., components 310-370; FIG. 3) of a UI manipulation engine (e.g., engine 300; FIG. 3) for updating UI distortion and rendering on a user device (e.g., device 130, 230; FIGs. 1A, 2A) to protect sensitive, private information and data on a foldable UI display from onlookers. Method 600 is introduced with simultaneous reference to FIGS. 2A-2C and 3.

[0124] The UI manipulation engine can perform UI distortion updates when it receives or obtains 610 an update regarding the actual current physical configuration of the user device (e.g., the current folding angle). More specifically, in some embodiments, the UI manipulation engine can detect one or more of the following new information: it can detect a new indication of sensitive data to be displayed 612, it can detect a new level of privacy risk in the user device's surroundings 614, it can detect a new user state (e.g., a new point of gaze or a new eye position of the user) 616, and / or it can detect a new actual device configuration (e.g., a new actual folding angle or a new device orientation) 618.

[0125] Next, consistent with aspects described above (e.g., with respect to FIG. 3), the UI operations engine may determine (620) a new optimal device configuration. In some examples, when there is a new or updated level of sensitive data for the display (612) and / or when there is a new level of ambient privacy risk (614), the UI operations engine may determine (620) a new optimal device configuration (e.g., a new optimal folding angle and / or, in some embodiments, a new optimal viewing angle). Additionally, for those implementing dynamic viewing angle adjustment, when there is a new detected user state (e.g., a new point of gaze) (616), the UI operations engine must determine (620) a new optimal device configuration (e.g., a new viewing angle) for UI distortion generation. Similarly, for those implementing dynamic folding angle adjustment, when there is a new detected actual device configuration (e.g., a new actual folding angle) (618), the UI operations engine must determine (620) a new optimal device configuration (e.g., a new folding angle) for UI distortion generation.

[0126] The UI manipulation engine then adjusts the UI according to the updated new optimal device configuration (630). In one or more implementations, the UI distortion updates are performed substantially in real-time as the user manipulates the folding display. In one or more examples, the selection steps of steps 610-630 may be performed iteratively or recursively. [Example]

[0127] FIG. 7A illustrates a more detailed example of how a distorted UI may be generated (e.g., by UI distortion generator 360; FIG. 3) on a bi-fold display (e.g., display 120, FIG. 1A; 220, FIG. 2A) of a foldable screen device (e.g., device 130, FIG. 1A; device 230, FIG. 2A). More specifically, after a UI form factor of how the sensitive information should be displayed is determined (e.g., as described above with respect to FIG. 3), the actual UI to be displayed on the foldable display is generated based on the determined form factor. To determine how the UI should be distorted and displayed, provided below are equations and descriptions of an example method of computing the UI transformation given the above input parameters of FIG. 7A.

[0128] As shown in FIG. 7A, the mobile device 730 includes a bi-foldable display 720 that folds around a central axis 736 (e.g., a hinge). In the following calculation example, it is assumed that the distance (X1) from the central axis 736 to the edge of the bi-foldable display 720 is known. It is further assumed that the distance (X2) from the central axis 736 to the user's eyes can be detected and / or estimated (e.g., based on a sensor such as an eye tracker and / or a ToF sensor that may be mounted on the device 730). Finally, in the example shown in FIG. 7A, the folding angle is represented by (α) and the viewing angle is represented by (β).

[0129] Let the resulting length of the edge of the manipulated UI be (Y1) and the resulting length of the center of the manipulated UI be (Y2) (as shown in FIG. 7A). Let the initial length of the edge of the original undistorted UI be (Y0) (as shown in FIG. 7B). The relationship between Y0, Y1, and Y2 can be expressed by the following transformation:

number

number

[0130] Figure 7C shows an example of transformation calculation by changing the folding angle α in the example of Figure 7A. Specifically, the following shows an example calculation method for T1:

[0131] 7C and 7A, assuming the viewing angle β is constant, point P1 in FIG. 7C (at the center of the display, where the central axis or hinge is) is assumed to be stationary relative to the original plane of the device. Assuming the thickness of the device is negligible compared to ΔX2, transformation T1 can be applied to compensate for the increased perceived change in height of the UI feature at point P2 (at the edge of the display) from the viewpoint of an observer at distance X2.

[0132] In this coordinate system, the size of UI features at the center of the display, the folding axis (P1) (eg, axis 736), remains unchanged, so Y2=Y0 and T1 does not need to be applied to UI features at the center axis.

[0133] To determine a UI feature at a point P2 that is a distance X1 away from the origin (e.g., at an initial height Y0), a transformation T1 can be applied to compensate for the perceived increase in size due to the feature being closer to the observer by an amount ΔX2. Thus, a transformation can be applied to the feature at P1 (assuming height Y0) to obtain a smaller height Y1. Such a transformation can be expressed as:

number

number

[0134] Thus, based on the example coordinate system of FIG. 7C, transformation T1 can be expressed as follows:

number

[0135] Figure 7D shows an example of conversion calculation according to the change in viewing angle β in the example of Figure 7A. Specifically, the following shows an example of the calculation method in T2.

[0136] 7D and 7A, assuming that the folding angle α is constant, point P1 (at the center of the display, where the gaze point is) in FIG. 7D is assumed to be static with respect to the original plane of the device. Assuming that the thickness of the device is negligible compared to ΔX2, a transformation T2 can be applied to compensate for the increase in the perceived change in height of the UI feature at point P2 (at the edge of the display) from the viewpoint of the observer at distance X2.

[0137] The viewing angle β in three-dimensional (3D) space has three components, one each for rotation in the xy, yz, and zx planes. Each can be considered independently, but similar equations can be applied for the calculation.

[0138] In the above coordinate system, Y2=Y0, and the size of the UI feature at the center axis (P1) remains unchanged, so that T2 does not need to be applied to the UI feature at the center.

[0139] For a UI feature at a point P2 that is a distance X1 away from the origin (e.g., at an initial height Y0), a transformation T2 may be applied to compensate for the perceived increase in size due to the feature being closer to the observer by an amount ΔX2. Thus, a transformation may be applied to the feature at P1 (assuming height Y0) to obtain a smaller height Y1.

number

number

[0140] Therefore, based on the example coordinate system of FIG. 7D, the transform component T 2xy can be expressed as follows:

number

[0141] A similar transformation can be done for the other components of β: yz and β zx may also be applied to the transformation component T 2yz and T 2zx which are then used to represent the transformation T2 (using equation (2)).

[0142] FIG. 7E shows an example of how a UI can be adjusted to compensate for angular distortion (e.g., how tilted) in an actual, generated UI in response to a change in viewing angle. First, shown in FIG. 7E is an example UI 701 that indicates how UI elements should visually appear to a user (e.g., as determined by the optimal display configuration finder 330, FIG. 3). The example UI 701 can be displayed on a flat section of a display, which can be, for example, one section of the display sections that together form the foldable display discussed here. For ease of discussion, the following description focuses on adjusting the angular distortion of one section, but the same technique can be applied to multiple sections of a foldable display as well. Assume that the UI 701 is displayed on a yz plane, with the x-axis extending into the page shown in FIG. 7E. As shown, the UI 701 includes corners that are indicated by rounded dots and would visually form right angles (e.g., 90°). The following discussion describes how this angle can be used to perform an example angular distortion adjustment in response to changes in viewing angle β. Assume the angle that this angle forms is (γ).

[0143] Also shown in FIG. 7E are example UIs 702, 703, 704, and 705. Example UIs 702-703 illustrate when the display section is viewed by a user from an acute viewing angle β (i.e., less than 90°), and UIs 704-705 illustrate when the display section is viewed by a user from an obtuse viewing angle β (i.e., greater than 90°). It is noted here that without compensation for viewing angle changes, as shown in example UI 702, when the display section is viewed from an obtuse viewing angle β, the angle γ will be acute (as opposed to a right angle in UI 701). Similarly, without compensation for viewing angle changes, as shown in example UI 704, when the display section is viewed from an acute viewing angle β, the angle γ will be obtuse (as opposed to a right angle in UI 701).

[0144] Thus, according to at least some embodiments (e.g., those that perform dynamic viewing angle UI adjustments as described above), the UI can undergo angle distortion adjustments (e.g., changes in tilt angles, such as in the corners of box UI elements) to account for perceived changes in tilt angles due to changes in the user's viewing angle. With viewing angle UI adjustments, the angular distortion resulting from the viewing angle changes can be compensated for, as shown in example UIs 703 and 705. In other words, some embodiments disclosed herein can employ a mechanism that determines how much these tilt angles in the UI need to be adjusted based on changes in viewing angle β.

[0145] Specifically, in some examples, from a user's perspective, the angular angle γ can vary with the viewing angle β based on a hyperbolic tangent function (also known as a "tanh()" function), i.e., in certain embodiments, the relationship between the displayed angular angle γ and the viewing angle β can be described based on, for example, a formula based on tanh().

number

[0146] FIG. 7F shows an example of the relationship between the adjustment amount of the angle distortion and the change in the viewing angle, more specifically, Equation (9). As shown in FIG. 7F, the angle on the UI can be increased or decreased by an amount inversely proportional to tanh(β). Note that when the viewing angle β is 90°, the angle γ of the displayed angle on the actual UI is also 90°, i.e., when the viewing angle β is vertical, no adjustment of the angle γ is required.

[0147] It should be noted that in an actual implementation, one or more parameters (i.e., "a", "b", "c", and / or "d") in equation (9) may be optimized or adjusted based on the actual device design and application environment, such as the particular display characteristics, display size, and / or sensor input (such as how far and / or where the user's eyes are). In the embodiment shown in FIG. 7F, a=1.5, b=2, c=π, and d=1. In some implementations, the parameters may be actively adjusted for each user to better suit different people's user habits and personal characteristics (e.g., holding angle or distance).

[0148] It should also be noted that the tanh() function described above is merely an example, and that in one or more embodiments, functions other than the tanh() function may be used to achieve a similar effect. For example, in some variations, a "sigmoid" function (whose plot resembles "tanh") may be used. Additionally, some examples may use a line, or a combination of lines with different slopes, to approximate the visual effect described herein. It is recognized that a suitable function may be characterized by a plot that (1) crosses the origin at 90° (i.e., when β is 90°, γ is also 90°), and (2) is symmetric about the origin. In some embodiments, it may be further characterized by (3) an increase in β results in a decrease in γ, and vice versa (i.e., β and γ are inversely correlated). In some embodiments, when β is near 0°, γ is near 180°, and conversely, when β is near 180°, γ is near 0°.

[0149] In the above-described aspects, the disclosed UI manipulation techniques allow for sensitive information displayed on a flexible, foldable, or otherwise reconfigurable display to be hidden or obscured from bystanders while maintaining or improving the accessibility of the UI to its primary user. In this manner, not only is loss of user data prevented and the usability and readability of the UI maintained, but the configuration of the device can be dynamically customized to provide an optimal security configuration for the user's current environment. Furthermore, an intuitive UI system can be implemented with auxiliary UI elements that can assist the user in easily achieving an optimal screen configuration for security.

[0150] Here, in at least some embodiments, the presented UI manipulation techniques may be implemented based on one or more existing software development kits (SDKs) that can help perform transformations (e.g., as described above) for a UI given input parameters (e.g., dimensions, folding angles, or viewing angles). In general, the class of transformations on which the presented UI techniques can be built may be referred to as perspective transformations, which may be a subcategory under SDKs related to geometric image transformations. One example of such an SDK with functions available for perspective transformations is OpenCV. TM In these SDKs, there can be numerous functions that can receive inputs describing the dimensions of an input image (e.g., the original undistorted length of an edge of the UI, such as Y0 in FIG. 7B), a perspective reference (e.g., a viewing angle, or an object plane), and a desired output. Often, a function can be utilized, programmed, or otherwise instructed to "straighten" images on one plane and present them on a plane facing the viewer. In some embodiments of the present disclosure, the desired output can be from a "pre-distorted" graphic that takes into account changes in viewing angle and / or folding angle. [Grip operation to protect confidential data]

[0151] As discussed above, one approach to obscuring sensitive information from nearby bystanders is to have visual guidance indicators displayed on the foldable display of the user device prompting the user to manipulate the folding angle. The visual guidance indicator may be designed such that it appears as a selected geometric shape (e.g., a circle) only when the foldable display is manipulated to a selected folding angle. The visual guidance indicator may appear as another shape (e.g., a tilted oval) when the foldable display is not manipulated to an optimal folding angle.

[0152] Another approach to blocking sensitive information from nearby bystanders is to trigger the display of visual guidance indicators that prompt the user to adopt a grip position of the user device that blocks the view of nearby bystanders. An appropriate grip position may be determined based on optimal placement of hands and fingers to reduce the risk of data loss. The risk of data loss may be based on the determined angles at which multiple bystanders are present around the user device and therefore what portions of the display may currently be visible to those bystanders.

[0153] For example, a selected geometric shape (e.g., a circle) may be shown on a foldable display of a user device along with instructions to place a selected finger (e.g., index finger) of a selected hand (e.g., left hand) on the selected geometric shape. In such an embodiment, the location of the selected geometric shape may be based on the folding angle and the location(s) of nearby bystander(s). Although the embodiments may be described in the context of a user device having a foldable display, the features are equally applicable to user devices having flexible or otherwise reconfigurable displays. Thus, the process described below may be employed by user devices having reconfigurable displays that are not flexible or foldable to prevent loss of sensitive information in a seemingly natural manner.

[0154] As discussed further below, data regarding potential security and privacy risks surrounding the user device and / or data regarding the sensitivity of information displayed by the user device may be used to determine an optimal grip position to avoid information loss to nearby individuals. The optimal grip configuration may be used to generate UI element(s) that encourage or assist the user to adopt the configuration. For example, current grip position data may be used to determine which UI element(s) should be generated to encourage the user to move his / her grip to the optimal grip position. The current grip position may be determined based on readings generated by, for example, touch-sensitive elements, pressure-sensitive elements, proximity sensors, ambient light sensors, etc.

[0155] 8 shows a flow chart illustrating an example method 800 for implementing a grip manipulation technique. Method 800 may be implemented by various components (e.g., components 310-370; FIG. 3) of a UI manipulation engine (e.g., engine 300; FIG. 3) to protect sensitive information displayed by a user device from nearby individuals. At a high level, method 800 may be described as an algorithmic approach to determine a hand posture that would optimally prevent unintentional disclosure of sensitive information displayed by a user device.

[0156] First, the UI operations engine can receive (810) an instruction to display sensitive information on a display of a user device. One example of such an instruction can be a software function call. As described above, the user device can include a foldable, flexible, or otherwise reconfigurable display. Next, the UI operations engine can determine (820) an optimal display configuration (also referred to as an "optimal grip position") for how the sensitive information is displayed. In numerous embodiments, the UI operations engine identifies (822) a level of sensitivity of the information to be displayed and then determines a form factor for how the sensitive information is displayed. Additionally or alternatively, the UI operations engine can determine (824) a privacy risk in the surrounding environment of the user device in determining a form factor for the UI. In general, the higher the level of sensitivity of the data to be displayed or the higher the privacy risk of the surrounding environment, the smaller the form factor of the UI. That is, the form factor may be inversely correlated to the risk from the surrounding environment and / or the sensitivity of the information.

[0157] Additionally, the UI operations engine can determine an optimal grip configuration for obscuring the sensitive information (830). The optimal grip configuration (also referred to as an "optimal grip position") may be based on the location of nearby individuals as determined by a surrounding condition finder (e.g., surrounding condition finder 326; FIG. 3). For example, the surrounding condition finder may use machine vision facial recognition technology to determine the number of nearby individuals, with the goal being to block the line of sight of these individuals to prevent viewing of the display. If the display of the user device is flexible or foldable, the UI operations engine may further determine the optimal grip position based on the current shape or folding angle. Thus, the UI operations engine may use data regarding the surrounding environment and / or information sensitivity to determine the privacy requirements of the display and identify the optimal grip position for obscuring the sensitive information shown thereon.

[0158] In some embodiments, the optimal grip position is further informed by the user's physical dimensions and capabilities (collectively referred to as "hand characteristics"). Examples of hand characteristics include the size (e.g., measured in terms of width and length) and flexibility of the hand, fingers, and palm. These hand characteristics may be manually input by the user, or these hand characteristics may be determined algorithmically based on past interactions with the mobile device. Alternatively, these hand characteristics may be estimated for a given user based on demographic averages. This estimate may be used directly by the algorithm, or this average may be further refined using the approaches described above.

[0159] As described above, optimal grip positions may be determined based on the positions of nearby individuals whose views of the display should be blocked. More specifically, the UI operations engine may determine optimal grip positions for blocking the lines of sight of these nearby individuals (e.g., as determined from the peripheral data) and preventing them from seeing sensitive portions of the display (e.g., as determined from the UI sensitivity data). As an example, the UI operations engine may determine the viewing angle of each nearby individual and then determine which viewing paths need to be blocked to protect the sensitive information. The UI operations engine may then generate one or more grip patterns that (i) block the viewing paths and (ii) are feasible given the characteristics of the user's hand.

[0160] The UI operations engine can then generate a UI to be displayed on the display based on the optimal display configuration and the optimal grip configuration (840). For example, the UI operations engine may receive data regarding the actual configuration of the user device and the location(s) of nearby individual(s) (842) and then cause UI element(s) to be displayed that indicate where the user should place their hands to obscure at least the portion of the display where sensitive information is presented. Thus, depending on the nature of the display, the UI operations engine may obtain data indicative of the orientation, folding angle, or shape. As shown in the example UI of FIG. 10, one or more UI elements may be displayed that indicate where selected fingers should be placed to ensure that the hands are in an optimal grip position.

[0161] 9 shows a flowchart illustrating an example method 900 for implementing a UI-driven grip manipulation technique. The method 900 may be performed by various components (e.g., components 310-370; FIG. 3) of a UI manipulation engine (e.g., engine 300; FIG. 3) to protect sensitive information presented by a user device from nearby individuals.

[0162] When the UI operations engine receives an update regarding the actual current physical configuration of the user device (910), it can perform a UI update. For example, the UI operations engine can detect one or more of the following new information: it can detect a new display of sensitive data being displayed (912), it can detect a new level of privacy risk in the user device's surroundings (914), it can detect a new user state (e.g., a new grip position) (916), and / or it can detect a new actual device configuration (e.g., a new orientation, folding angle, or shape) (918).

[0163] As an example, the UI operation engine may continuously monitor the grip position by examining data (referred to as "grip data") generated by one or more sensors (referred to as "grip position sensors") integrated into the user device. Examples of grip position sensors include touch screen support components (e.g., touch-sensitive elements and pressure-sensitive elements), proximity sensors, ambient light sensors, gyroscopes, accelerometers, etc. In some embodiments, sensing of the grip position is assisted by active generation of stimuli such as vibrations (e.g., generated by motors or piezoelectric elements). Additionally or alternatively, techniques including optical sensors (e.g., cameras) facing the user may be employed to determine the position of the hand based on, for example, corneal reflexes. The grip data may describe various characteristics of the grip position, including the position of the palm relative to the user device and its display, as well as the position of the thumb(s) relative to the user device and its display. Thus, the UI operation engine may obtain grip data generated by the grip position sensor(s) and then examine the grip data to determine an optimal grip position.

[0164] Next, consistent with the approach described above (e.g., with respect to FIG. 8), the UI operations engine can determine a new optimal grip position (920). For example, the UI operations engine may determine a new optimal grip position if there is a new or updated level of sensitive information for display (912) and / or if there is a new level of privacy risk in the surrounding environment (914). As an example, if the UI operations engine determines that the surrounding environment has become more crowded (thereby reducing privacy), the UI operations engine may determine that a new optimal grip position that provides further obscuration of the sensitive information may be needed.

[0165] The UI manipulation engine may then adjust the UI according to the new optimal grip configuration (930). As discussed further below with respect to FIG. 11, the adjustment may include adding a UI element(s) that indicates where the user should place the selected finger(s). In one or more embodiments, the UI adjustment is performed substantially in real-time as the user manipulates the display (e.g., by changing its orientation, folding angle, or shape). Similarly, the UI adjustment may be performed substantially in real-time as the user moves their hand relative to the user device. For example, the location of the sensitive information may be changed in response to a determination that the user has moved their hand to interact with (e.g., touch) the display. Steps 910-930 may be performed iteratively or recursively.

[0166] 10A is an example data flow 1000 for implementing a grip manipulation technique using some example components of the UI manipulation engine of FIG. 3. Data flow 1000 illustrates an example of how data can flow between various blocks in a UI manipulation system (e.g., engine 300). Data flow 1000 is substantially similar to data flow 400 of FIG. 4A. However, here data regarding the characteristics of the user's hands are also obtained by optimal display configuration finder 330. As described above, these data can be used to determine which grip positions can be feasibly achieved by the user.

[0167] Meanwhile, FIG. 10B is another example data flow 1002 for implementing a grip manipulation technique using some example components of the UI manipulation engine of FIG. 3. As mentioned above, the grip position sensor(s) 910 may be responsible for generating grip data from which the position of the hand may be inferred. Examples of grip position sensors include touch screen support components (e.g., touch-sensitive elements and pressure-sensitive elements), proximity sensors, ambient light sensors, gyroscopes, accelerometers, etc. Additionally or alternatively, techniques including optical sensors (e.g., cameras) facing the user may be employed to determine the position of the hand based on, for example, corneal reflex. Using the grip data, the grip position finder 920 may determine the actual grip position of the user. In other words, the grip position finder 920 may determine the actual position of the hand based on an analysis of the grip data generated by the grip position sensor(s) 910. Information regarding the actual grip position may be provided to an auxiliary UI element generator 370, which may be configured to generate a UI element(s) for implementing a grip manipulation technique based on the optimal display configuration determined by the optimal display configuration finder 330 and the actual grip position determined by the grip position finder 920.

[0168] 11A-D show an example implementation of the grip manipulation technique. First, a UI manipulation engine (e.g., engine 300; FIG. 3) running on the user device 1100 can establish the locations of nearby individuals 1102a-c, as described above with respect to FIGS. 8-9. Such an approach allows the UI manipulation to identify visual paths that must be blocked to prevent these individuals from observing the display of the user device 1100. More specifically, the UI manipulation engine can calculate which direction(s) must be blocked to ensure privacy, as shown in FIG. 11B.

[0169] The UI operations engine can then identify an appropriate UI element(s) to present on the display. For example, the UI operations engine may determine where a visual guidance indicator should be presented on the display based on the calculated direction(s). In FIG. 11C, for example, the UI operations engine causes the presentation of visual guidance indicator 1104 in the upper right corner of the display. In some embodiments, visual guidance indicator 1104 is one of multiple visual guidance indicators presented on the display. For example, the UI operations engine may determine that multiple visual guidance indicators are needed in a crowded environment. The multiple visual guidance indicators may correspond to selecting fingers on the same hand or different hands.

[0170] The visual guidance indicia 1104 may be accompanied by instructions for placing a selected finger on the visual guidance indicia. When a user places the selected finger 1106 on the visual guidance indicia 1104, the view of nearby individuals 1102a-c will be obstructed (thereby ensuring that sensitive information remains private). In some embodiments, the visual guidance indicia 1104 is designed to be substantially covered by the selected finger 1106, where, for example, the selected finger 1106 completely covers the visual guidance indicia 1104 while a portion of the display is visible.

[0171] Many user devices are designed to allow multi-touch functionality (or simply "multi-touch"). Multi-touch allows a touch-sensitive display to recognize the presence of multiple points touching the display at the same time. Thus, the user device 1100 may be able to detect multiple contact points when one finger is held in one area while another finger touches another area. To account for this, a portion of the touch-sensitive display may be defined as outside of a multi-touch target area in some embodiments. For example, the user device 1100 may be instructed not to detect touch events that occur within a fixed area of ​​the touch-sensitive display (e.g., visual guidance indicia 1104 and its surroundings), or the user device 1100 may be instructed not to use such touch events for determining a multi-touch operation. Thus, the touch-sensitive display may have at least one portion in which touch events are recognized for multi-touch purposes and at least one portion in which touch events are not recognized for multi-touch purposes, and a visual guidance indicia (or visual guidance indicia) may be located in those portion(s) in which touch events are not recognized for multi-touch purposes.

[0172] Note that the visual guidance indicators 1104 may also represent instructions to position the selected finger 1106 completely outside the boundaries of the touch-sensitive display. As an example, the visual guidance indicators 1104 shown in FIG. 11C may be represented as arrows indicating that the selected finger 1106 should be positioned along the top or side of the user device 1100.

[0173] At a high level, the UI element(s) generated by the UI manipulation engine generally serve one of two purposes. First, the user may need to interact with these UI element(s) to assume an optimal grip position required to activate a desired function. Second, the user may need to interact with these UI element(s) to perform a desired function. Examples of desired functions include confirming a payment, entering a password, and viewing financial or personal information. Thus, the user may not be able to perform a desired function until the optimal grip position is assumed.

[0174] In some embodiments, the UI element(s) represent existing elements required for a given function. For example, the UI operations engine may use an existing graphic labeled "Show Password" as the UI element, but its characteristics (e.g., position and size) may be adjusted to meet the requirements described above. In other embodiments, the UI element(s) represent new elements that meet the requirements described above. For example, the UI operations engine may overlay a fingerprint(s) indicating where the selected finger(s) should be placed onto the UI so that a given function can be completed.

[0175] Each UI element created by the UI operations engine has properties that determine its appearance, functionality, and location. Examples of such properties include dimensions, color, animation, and location. Similarly, these properties may specify the action, if any, required to cause activation of the corresponding UI element (e.g., press and hold, swipe, tap repeatedly). In some embodiments, these properties are influenced by fundamental characteristics of the underlying UI in which the UI element(s) are created / selected. For example, the color of the UI element(s) may be changed to match the color scheme of the underlying UI.

[0176] In some embodiments, aspects of the grip manipulation technique are controlled by a user of the user device 1100. For example, the user may be prompted to specify where nearby individuals 1102a-c are located (e.g., by tapping an edge of the display to indicate the location). As another example, the user may be able to influence the location, number, or arrangement of the visual guidance indicators (e.g., visual guidance indicators 1104). For example, the user may initially be prompted to place a selected finger on the visual guidance indicator 1104, but the user may be able to change the location of the visual guidance indicator 1104 by performing an action (e.g., tapping at least twice and then dragging). As discussed above, characteristics of the user's hand may be used to determine an appropriate location of the visual guidance indicators shown on the user device 1100. In some embodiments, the user may be prompted to complete a calibration process in which different arrangements of visual guidance indicators are shown on the display of the user device 1100. These arrangements may include different numbers of visual guidance indicators placed in various locations. Based on the speed and ease with which the user can position a selected finger on one or more visual guidance elements included in each configuration, the UI manipulation engine may learn the most appropriate configuration for the user. [Computer System and Device Architecture]

[0177] 12 is a block diagram illustrating an example of a computing system 1200 in which at least some operations described herein may be implemented, such as some components of the computing system 1200 that may be utilized to implement a computing device (e.g., user device 130, 230 of FIGS. 1A and 2A) that includes a UI operations engine (e.g., UI operations engine 300 of FIG. 3).

[0178] Computing system 1200 may include one or more central processing units (also referred to as "processors") 1202, main memory 1206, non-volatile memory 1210, network adapter 1212 (e.g., a network interface), video display 1218, input / output devices 1220, control devices 1222 (e.g., keyboard and pointing devices), drive unit 1224 including storage medium 1226, and signal generator 1230, communicatively coupled to a bus 1216. Bus 1216 is illustrated as an abstraction representing one or more physical buses and / or point-to-point connections connected by appropriate bridges, adapters, or controllers. Thus, the bus 1216 may include a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or industry standard architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal serial Bus (USB), an I2C (IIC) bus, or the Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (also known as "Firewire"), etc.

[0179] Computing system 1200 may share a similar computer processor architecture as that of a personal computer, a tablet computer, a mobile phone, a gaming console, a music player, a wearable electronic device (e.g., a watch or fitness tracker), a network-connected ("smart") device (e.g., a television or home assistant appliance), a virtual / augmented reality system (e.g., a head-mounted display), or another electronic device that can execute (sequentially or otherwise) a set of instructions that specify action(s) to be performed by computing system 1200.

[0180] Although main memory 1206, non-volatile memory 1210, and storage medium 1226 (also referred to as a "machine-readable medium") are depicted as a single medium, the terms "machine-readable medium" and "storage medium" should be interpreted to include a single medium or multiple media (e.g., a centralized / distributed database and / or associated caches and servers) that store one or more sets of instructions 1228. Additionally, the terms "machine-readable medium" and "storage medium" should be interpreted to include any medium that can store, encode, or carry a set of instructions for execution by computing system 1200.

[0181] Generally, the routines executed to implement the embodiments of the present disclosure may be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions (collectively referred to as a "computer program"). A computer program typically comprises one or more instructions (e.g., instructions 1204, 1208, 1228) stored at various times in various memories and storage devices within a computing device. When read and executed by one or more processors 1202, the instruction(s) cause the computing system 1200 to perform operations to implement elements related to various aspects of the present disclosure.

[0182] Additionally, while the embodiments have been described in the context of a fully functional computing device, those skilled in the art will appreciate that various embodiments can be distributed as program products in a variety of forms, and the present disclosure applies regardless of the particular type of machine or computer-readable medium used to actually accomplish the distribution.

[0183] Further examples of machine-readable recording media, machine-readable media, or computer-readable media include volatile and non-volatile memory devices 1210, recordable type media such as floppy disks and other removable disks, hard disk drives, optical disks (e.g., compact disk read-only memory (CD-ROM), digital versatile disk (DVD)), and transmission type media such as digital and analog communications links.

[0184] Network adapter 1212 enables computing system 1200 to broker data within network 1214 with entities external to computing system 1200 through any communications protocol supported by computing system 1200 and the external entity. Network adapter 1212 may include a network adapter card, a wireless network interface card, a router, an access point, a wireless router, a switch, a multi-layer switch, a protocol converter, a gateway, a bridge, a bridge router, a hub, a digital media receiver, and / or a repeater.

[0185] Network adapter 1212 may include a firewall that determines and / or manages access / proxy permissions to data in a computer network and tracks various trust levels between different machines and / or applications. A firewall may be any number of modules having any combination of hardware and / or software components that can enforce a given set of access rights between a particular set of machines and / or applications (e.g., regulate the flow of traffic and resource sharing between these entities). A firewall may further manage access to and / or have access control lists that detail permissions including the right of individuals, machines, and / or applications to access and manipulate objects, and the circumstances under which the right stands.

[0186] The techniques presented herein may be implemented in programmable circuitry (e.g., one or more microprocessors), software and / or firmware, special purpose hardwired (i.e., non-programmable) circuitry, or a combination of such forms. The special purpose circuitry may be in the form of one or more application specific integrated circuits (ASICs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), etc. [remarks]

[0187] The foregoing description of various embodiments of the claimed subject matter has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precise forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments have been selected and described in order to best explain the principles of the invention and its practical application so that others skilled in the relevant art can appreciate the claimed subject matter, various embodiments, and various modifications that are suitable for the particular application contemplated.

[0188] Although the detailed description describes certain embodiments and the best mode contemplated, no matter how detailed the detailed description appears, the technology can be implemented in many ways. An embodiment may vary considerably in its implementation details while being encompassed in this specification. A specific term used when describing a particular feature or aspect of various embodiments should not be interpreted as meaning to be redefined herein to be limited to any particular characteristic, feature, or aspect of the technology with which the term is associated. In general, the terms used in the following claims should not be interpreted to limit the technology to the specific embodiments disclosed herein unless those terms are expressly defined herein. Thus, the actual scope of the technology encompasses not only the disclosed embodiments, but also all equivalent ways of carrying out or implementing the embodiments.

[0189] The language used herein has been selected primarily for ease of reading and explanation. It has not been selected to define or enclose subject matter. Thus, it is intended that the scope of the technology be limited not by this detailed description, but rather by any claims that issue in an application based hereon. Thus, the disclosure of various embodiments is intended to be illustrative, but not limiting, of the scope of the technology, which is defined in the following claims. [Industrial Applicability]

[0190] This disclosure is applicable to data security.

Claims

1. 1. A method for privately displaying information, the method comprising: receiving instructions to display confidential information on a mobile device, the mobile device including a foldable display; determining a form factor for how the sensitive information will be displayed, the form factor being less than a total area of ​​the foldable display; generating a user interface for display on the foldable display based on the determined form factor, the user interface displaying the sensitive information in the form factor but visually distorted to a user unless the foldable display is manipulated to a selected angle; Detecting an eye position of the user from an eye detector on the mobile device facing the user; and adjusting the user interface in response to the detected position of the user's eyes; A method for providing the above.

2. The method of claim 1 , wherein the user interface is adjusted such that a new viewing angle for the user interface reflects the detected position of the eyes of the user.

3. receiving a current orientation state of the mobile device from one or more orientation sensors on the mobile device; the user interface is further adjusted in response to the received current orientation state of the mobile device. The method of claim 1.

4. A method for privately displaying information, the method comprising: receiving instructions to display confidential information on a mobile device, the mobile device including a foldable display; determining a form factor for how the sensitive information will be displayed, the form factor being less than a total area of ​​the foldable display; generating a user interface to be displayed on the foldable display based on the determined form factor, the user interface displaying the sensitive information in the form factor but visually distorted to a user unless the foldable display is manipulated to a selected angle; and generating, based on the selected angle, a visual guidance indicia that guides the user to manipulate the foldable display to the selected angle; A method for providing the above.

5. The method of claim 4 , wherein the visual guidance indicia are visible to the user as a selected geometric shape only when the foldable display is manipulated to the selected angle.

6. A method for privately displaying information, the method comprising: receiving instructions to display confidential information on a mobile device, the mobile device including a foldable display; determining a form factor for how the sensitive information will be displayed, the form factor being less than a total area of ​​the foldable display; generating a user interface for display on the foldable display based on the determined form factor, the user interface displaying the sensitive information in the form factor but visually distorted to a user unless the foldable display is manipulated to a selected angle; Obtaining a current folding angle of the foldable display; and generating additional guidance to guide the user to manipulate the foldable display to the selected angle based on a difference between the current folding angle and the selected angle; A method for providing the above.

7. The method of claim 1 , wherein the user interface is collectively displayed by multiple sections of the foldable display.

8. The method of claim 7 , wherein the foldable display is a bi-fold display.

9. further comprising identifying a level of sensitivity of the sensitive information; The method of claim 1 , wherein the form factor is further determined based on and inversely related to the level of confidentiality.

10. determining a level of privacy risk in an environment surrounding the mobile device based on readings from one or more sensors on the mobile device; The method of claim 1 , wherein the form factor is further determined based on the identified level of privacy risk.

11. The method of claim 10 , wherein the form factor is inversely correlated to the level of the identified privacy risk.

12. determining a folding angle at which the user interface will appear distorted to the user unless a display section of the mobile device is folded to the folding angle; The method of claim 1 , wherein the user interface is further generated based on the determined folding angle.

13. The method of claim 12 , wherein the user interface is deformed symmetrically about an axis along a hinge of the foldable display.

14. 7. The method of claim 1, further comprising: based on the determined form factor, modifying an original layout of the sensitive information to an alternative layout suitable for a form factor that is smaller compared to the overall area of ​​the foldable display.

15. 15. The method of claim 14, wherein the original layout is for a display form factor designated for a desktop computer and the alternative layout is for a display form factor designated for a mobile device.

16. obtaining a current folding angle of the foldable display; updating the form factor based on the current folding angle; and adjusting the user interface to conform to the updated form factor; The method of claim 1 , further comprising:

17. 17. The method of claim 16, wherein the obtaining, updating, and adjusting steps are performed substantially in real-time as the user manipulates the foldable display.

18. 1. A method for privately displaying information, the method comprising: receiving an instruction to display confidential information on a mobile device, the mobile device including a flexible display; determining a form factor for how the sensitive information will be displayed, the form factor being less than a full area of ​​the flexible display; generating a user interface for display on the flexible display based on the determined form factor, the user interface displaying the sensitive information in the form factor but visually distorted to a user unless the flexible display is manipulated into a selected shape; Detecting the position of the user's eyes from a user-facing eye detector on the mobile device; and adjusting the user interface in response to the detected position of the user's eyes; A method for providing the above.

19. A method for privately displaying information, the method comprising: receiving an instruction to display confidential information on a mobile device, the mobile device including a flexible display; determining a form factor for how the sensitive information will be displayed, the form factor being less than a full area of ​​the flexible display; generating a user interface for display on the flexible display based on the determined form factor, the user interface displaying the sensitive information in the form factor but visually distorted to a user unless the flexible display is manipulated into a selected shape; and generating, based on the selected shape, a visual guidance indicia that guides the user to manipulate the flexible display into the selected shape; A method for providing the above.

20. A method for privately displaying information, the method comprising: receiving an instruction to display confidential information on a mobile device, the mobile device including a flexible display; determining a form factor for how the sensitive information will be displayed, the form factor being less than a full area of ​​the flexible display; generating a user interface for display on the flexible display based on the determined form factor, the user interface displaying the sensitive information in the form factor but visually distorted to a user unless the flexible display is manipulated into a selected shape; Obtaining a current shape of the flexible display; and generating additional guidance to guide the user to manipulate the flexible display into the selected shape based on a difference between the current shape and the selected shape; A method for providing the above.

21. A method for privately displaying information, the method comprising: receiving an instruction to display confidential information on a mobile device, the mobile device including a flexible display; determining a form factor for how the sensitive information will be displayed, the form factor being less than a full area of ​​the flexible display; generating a user interface for display on the flexible display based on the determined form factor, the user interface displaying the sensitive information in the form factor but visually distorted to a user unless the flexible display is manipulated into a selected shape; and generating, based on the selected shape, visual guidance indicia that guides the user to grasp the flexible display in a manner that occludes a portion of the flexible display; A method for providing the above.

22. further comprising identifying a level of sensitivity of the sensitive information; 21. The method of claim 18, wherein the form factor is further determined based on, and inversely related to, the level of confidentiality.

23. determining a level of privacy risk in an environment surrounding the mobile device based on readings from one or more sensors on the mobile device; The method of claim 18 , wherein the form factor is further determined based on the identified level of privacy risk.

24. 21. The method of any of claims 18 to 20, further comprising determining a fold angle and a fold line such that unless a display section of the mobile device is folded at and along the fold angle, the user interface will appear distorted to the user.

25. obtaining a current shape of the flexible display; updating the form factor based on the current shape; and adjusting the user interface to conform to the updated form factor; 21. The method of claim 18, further comprising:

Images