User interface manipulation in foldable screen devices

Abstract

User interface (UI) manipulation techniques are disclosed that can allow a user device to cause sensitive information displayed on a flexible, foldable, or otherwise reconfigurable display to be hidden or obscured from a bystander while maintaining or improving its exclusive accessibility to the primary user. Examples of the disclosed techniques can manipulate the UI in a manner that causes it to be largely visible only as expected when the user device is physically configured to a particular fold angle (and / or viewing angle). In some examples, the UI can be customized so that it elevates a particular device configuration (e.g., fold angle) that can provide an optimal security configuration for the user's current surrounding environment. The UI displayed on the display screen can change form factor (e.g., expected size as seen by the user) to accommodate the current surrounding environment and / or sensitivity of the content being displayed.

Description

Technical Field

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 008,473, filed April 10, 2020, entitled "User Interface Manipulation in a Foldable Screen Device," and U.S. Provisional Application No. 62 / 991,553, filed March 18, 2020, entitled "Foldable Phone UI Manipulation for Privacy." Each of these applications is incorporated herein by reference in its entirety. Technical Field

[0004] This disclosure generally relates to data security, and more specifically, to protecting sensitive information displayed on mobile end-user devices from attempts by bystanders to misappropriate such information. Background Technology

[0005] With the widespread use of computers and portable electronic devices, the preferred mode of information presentation has long since shifted from paper to electronic formats. Typically, these electronic devices are equipped with displays (e.g., liquid crystal display (LCD) screens) that present visual information to human users.

[0006] In many situations, such as when financial or business transactions are involved, sensitive information such as Social Security numbers or bank account numbers may be displayed on the screen. This technological state has created vulnerabilities for unethical bystanders who can steal users' sensitive personal and financial information by viewing user information from the side of the device and taking psychological or physical photographs. Attached Figure Description

[0007] One or more embodiments of this disclosure are shown by way of example and not limitation in the various figures of the accompanying drawings, and similar reference numerals in the drawings indicate similar elements.

[0008]

Figure 1A

[0009]

Figure 1B

[0010]

Figure 1C

[0011]

Figure 1D

[0012]

Figure 2A

[0013]

Figure 2B

[0014]

Figure 2C

[0015]

Figure 3

[0016]

Figure 4A

[0017]

Figure 4B

[0018]

Figure 4C

[0019]

Figure 5

[0020]

Figure 6

[0021]

Figure 7A

[0022]

Figure 7B

[0023]

Figure 7C

[0024]

Figure 7D

[0025]

Figure 7E

[0026]

Figure 7F

[0027]

Figure 8

[0028]

Figure 9

[0029]

Figure 10A

[0030]

Figure 10B

[0031]

Figure 11A

[0032]

Figure 11B

[0033]

Figure 11C

[0034]

Figure 11D

[0035]

Figure 12

[0036] References to "an embodiment," "one embodiment," etc., in this specification mean that a particular feature, function, structure, or characteristic described is included in at least one embodiment of this disclosure. The appearance of such phrases in this specification does not necessarily refer to the same embodiment. Furthermore, the mentioned embodiments are not necessarily mutually exclusive.

[0037] As mentioned above, financial and business transactions are increasingly taking place on computers and other portable electronic devices (such as mobile phones) rather than on paper. These devices are typically equipped with displays (e.g., liquid crystal display (LCD) screens) that present visual information to human users. In many cases, such as when financial or business transactions are involved, sensitive information like social security numbers or bank account numbers may be displayed on the screen. With smartphones gaining an increasingly larger share of the payments market, more and more people are frequently using their smartphones for banking applications. People may attempt to enter and read private data in highly insecure public places. This technological state creates vulnerabilities for malicious actors and agents who can physically observe a user's interactions with their device and steal sensitive personal and financial information from the user by viewing user information from the side of the device and taking psychological or physical photographs.

[0038] Meanwhile, significant progress has been made in the creation of flexible displays (e.g., those with fabric-based substrates) over the past decade, primarily driven by advancements in organic light-emitting diode (OLED) display technology. These flexible display devices allow for diverse configurations within a single device, such as allowing variable screen sizes to meet application needs and variable physical configurations. With improvements in flexible displays, they enable a variety of new physical configurations for user interfaces and interaction modes. The UI elements of foldable phones (e.g., one or more screens) can be quickly manipulated into different shapes or changed to different folding angles. In addition to flexible displays, devices with rigid displays but two or more screens (e.g., dual-foldable phones with two screens) have also appeared on the market, thanks to the increasing prevalence of even traditional displays and the continuous reduction in manufacturing costs.

[0039] Therefore, user interface (UI) manipulation techniques are described herein that allow user devices to conceal or obscure sensitive information displayed on flexible, foldable, or otherwise reconfigurable displays from onlookers while maintaining or enhancing its accessibility specifically for the primary user. More specifically, one or more embodiments of the disclosed techniques can manipulate the UI in such a way that it is largely visible only as intended when the user device is physically configured at a certain folding angle. In some examples, the UI can be customized to facilitate a device configuration (e.g., folding angle) that provides an optimal security configuration for the user's current environment. The UI displayed on the screen can change its shape factor (e.g., the expected size for the user to view) to adapt to the current environment (e.g., as the user moves from one environment to another, or as the environment around the user changes). Alternatively or additionally, the displayed UI can change its shape factor based on the sensitivity of the displayed content.

[0040] Through UI manipulation, the techniques described here can hide or obscure information displayed on a device in an intuitive way that maintains usability and readability for the user, thereby enabling an effective user interface system that helps users easily achieve optimal screen configuration in terms of security and prevents accidental data loss to bystanders.

[0041] In the following description, examples of foldable mobile devices are used to explain various aspects of these technologies for illustrative purposes only. However, it should be noted that the technologies described herein are not limited to foldable phones or any other particular type of device. Furthermore, even though the following description focuses on implementing various aspects of the disclosed technologies on a mobile phone, other electronic devices or systems (e.g., laptops or tablets) can adapt these technologies in a similar manner. Unless the context clearly indicates otherwise, the terms “foldable display” or “foldable screen” used interchangeably herein refer to any variation of a display whose display area is foldable, including at least two variations: (1) a display having at least one flexible screen (e.g., a flexible OLED display) configured to be foldable, and (2) a display having at least two parts, each of which has a screen, but the two parts are configured to function together (or collectively) as a single screen, and the two parts are mechanically coupled to each other by a folding mechanism. Some examples of foldable screens include bi-fold displays, tri-fold displays, etc.

[0042] In this specification, the term "user" generally refers to anyone physically operating the mobile device, while the term "bystander" generally refers to anyone physically close to the user but not physically operating the mobile device. Additionally, it should be noted that the UI manipulation techniques described herein should be distinguished from techniques that display information using only a smaller area than the full area (e.g., displaying by using fewer pixels, such as simply reducing a 1900×1200 pixel display to a 1680×1050 pixel display); as discussed herein, UI manipulation techniques utilize visual distortion to visually instruct the user to place the phone in a specific physical device configuration (e.g., a folded angle) before the user can properly view the UI. Therefore, in this disclosure, in the context of a display, the term "shape factor" generally refers to how large or small the display should appear visually to the user (i.e., from the user's perspective), rather than from the perspective of a third party (e.g., a bystander). Similarly, for the purposes of this discussion, unless explicitly stated otherwise in the context of the description, the terms "viewing" or "perceiving" are used from the perspective of the user of the device in question, and not from the other.

[0043] It should be noted that the accompanying drawings are merely examples for the purpose of illustrating the disclosed UI manipulation techniques; they are not drawn to scale.

[0044] Application Environment

[0045] Figure 1AAn environment 100 is shown, in which the foldable phone user interface described herein can be implemented. In environment 100, a user 110 is shown interacting with a foldable screen 120 on a foldable mobile device 130 in the presence of multiple bystanders 140. Because the bystanders 140 are present in environment 100, there is a high probability that the bystanders 140 may steal user information displayed on the user interface of the mobile device 130 (i.e., as would be displayed directly on the foldable screen 120 in a conventional manner). Figure 1A As shown, mobile device 130 is displaying user's private and sensitive information, which poses a risk to user privacy. Therefore, it is necessary to protect user 110's information from being stolen by bystander 140.

[0046] Therefore, as discussed in more detail below, the UI manipulation techniques disclosed herein can manipulate the UI in such a way that the UI becomes substantially visible to the user only when the device is configured / reconfigured by the user to a selected device configuration (e.g., folded at a certain angle, and / or viewed from a certain perspective in some embodiments). In other words, when the user device is physically configured to a specific folding angle and / or perspective, the disclosed UI system can manipulate the UI so that the resulting UI is visually presented from the user's perspective at an increased size and / or angle.

[0047] In one or more examples, the disclosed UI manipulation system is able to determine the privacy risk level of the user's surroundings by detecting the likelihood and potential number of bystanders present in the user's environment. Alternatively or additionally, the UI system is able to determine the sensitivity of private and sensitive information presented on the user's mobile device 130. Then, in some embodiments, based on the privacy risk level of the user's surroundings and / or the sensitivity of the private and sensitive information presented on the user's mobile device 130, the UI system is able to determine the shape factor of how the sensitive information will be displayed on the foldable screen 120, such that only the user 110 can obtain an undistorted view of the sensitive information. In these embodiments, the sensitive information may also remain distorted for the user 110 unless the foldable screen 120 is manipulated to a selected folding angle and / or a selected viewing angle. In other words, in some examples, the UI can be customized to facilitate a specific device configuration (e.g., folding angle and / or viewing angle) that provides an optimal security configuration for the user's current environment 100.

[0048] Folding angle

[0049] Figure 1B The folding angle 138 of a foldable phone (e.g., user equipment 130) is shown. Figure 1BAs shown, the folding angle 138 is the angle between the parts of a multi-part foldable mobile device (e.g., user device 130). More specifically, the folding angle 138 is formed by the angle between two axes: a first axis 132 extending along the surface of a first part (e.g., the left side) of the foldable phone 130, and a second axis 134 extending along the surface of a second part (e.g., the right side) of the foldable phone 130. The first and second parts of the foldable phone 130 are mechanically connected and fold along a hinge axis 136. Figure 1B As shown, the UI on device 130 is displayed by the left and right portions of the foldable display.

[0050] It should be noted that, for the sake of simplicity, the dual-folding display embodiments described herein focus on configurations that fold in the lateral direction (e.g., from left to right); however, the disclosed techniques can be similarly applied to those embodiments having configurations that fold in the longitudinal direction (e.g., from top to bottom). In some of those embodiments, the configuration can be interchangeable, for example, with manual or automatic screen rotation functionality, so that the configuration can better suit user preferences and current device orientation.

[0051] According to this embodiment, a folding angle 138 can be selected, and the UI is distorted accordingly, so that when the foldable screen of the foldable phone 130 (e.g., screen 120) is folded or manipulated to the selected folding angle (or desired folding angle), the user interface is optimally readable by user 110, but distorted to others in the environment (e.g., bystander 130). The selected folding angle can also be described as the optimal interior angle at which the display 120 is folded. For example, in the case of a double-folding display (e.g., single-hinge), the folding angle is the interior angle at which the display is folded.

[0052] In some embodiments, the desired folding angle can be determined based on the sensitivity of the private information to be displayed on the user interface and / or the presence of potential bystanders around the user, to blur the bystander's view of the information on the user's private screen. For example, when a user wants to access private or secure data, mobile phone camera data can be used to determine whether any potential bystanders are present around the user.

[0053] Once the optimal folding angle of a phone is determined, the user interface can be generated / distorted / customized to make it readable at that angle. The user folds the phone to a degree that makes the user interface readable from the user's position, preventing potential onlookers from peeking into the user's private information. For example, a user might want to check their bank account information while riding a bus. When the user accesses a software application installed on a foldable phone described here (e.g., a bank's proprietary app), the user interface presented to the user appears distorted. The user closes the phone to an even narrower angle until the user interface becomes visually readable from their perspective (e.g., edges and fonts do not appear warped / distorted), while preventing content (by closing the phone to a narrower angle) from being seen by potential onlookers around the user.

[0054] Perspective

[0055] Figure 1C An example foldable phone (e.g., user equipment 130) is shown with a viewing angle 148 relative to a user (e.g., user 110). Generally, the viewing angle is the angle from which the mobile device 130 faces the user 110. The viewing angle can also be understood as the position of the user 110's eye relative to the foldable device 130. Figure 1C As shown, in the example of the double-folding phone, the viewing angle 148 can be the angle between the first axis 142, which is perpendicular (i.e., 90°) to the unfolded edge of the device 130, and the line of sight of the user 110. Figure 1C The double-folding device folds around a centrally located hinge axis 146 (e.g., the same configuration as hinge axis 136).

[0056] It should be noted that, for convenience, such as Figure 1C As shown, the disclosure herein assumes that the gaze point (i.e., the point where the user's eyes are focused) is the center of the screen. The gaze point defines the starting edge from which the viewing angle 148 can be measured. While this definition of the viewing angle and / or the assumption of the gaze point may vary depending on the implementation (e.g., the gaze point may be at the top or bottom of the screen; or, in some implementations, the gaze point may be dynamically determined by a sensor), the UI manipulation techniques disclosed herein are similarly applicable regardless of any changes in the definition of the viewing angle and / or the location of the gaze point.

[0057] Static and dynamic UI adjustments

[0058] When a foldable screen is folded to a desired folding angle and viewed from a desired perspective, the UI generated by the UI manipulation techniques described here generally has an effect visible only to the user (i.e., from the user's perspective), meaning there is no visual distortion or deformation. Thus, depending on the embodiment, the user device may have the option to dynamically or statically adjust UI distortion. For the purposes of this discussion, the term "static UI manipulation" means that the UI is merely manipulated / distorted (e.g., as discussed below regarding...). Figure 2A The way the UI is described is for viewing at the optimal fold angle and / or from the optimal viewing angle; it is "static" in the sense that the UI presentation does not change based on the actual fold angle of the device or the actual position of the user's eye / gaze. In contrast, the term "dynamic UI manipulation" means that the UI presentation changes and receives adjustments based on the actual fold angle of the device and / or the actual position of the user's eye / gaze.

[0059] More specifically, such as Figure 1D As shown in Table 150, there are four different combinations of UI adjustment configurations: (A) both folding angle UI adjustment and viewing angle UI adjustment are static; (B) static folding angle UI adjustment and dynamic viewing angle UI adjustment; (C) dynamic folding angle UI adjustment and static viewing angle UI adjustment; and (D) both folding angle UI adjustment and viewing angle UI adjustment are dynamic. For the embodiment implementing static folding angle adjustment, the generated UI distortion does not need to take into account the actual folding angle of the device; instead, the UI is generated based on the selected optimal folding angle, determined by the user to reconfigure the device (e.g., by folding the screen) to the optimal folding angle in order to view the correct UI. Similarly, for the embodiment implementing static viewing angle adjustment, the generated UI distortion does not need to take into account the user's eye position / gaze point; instead, the UI is generated based on the selected optimal viewing angle, determined by the user to adjust the device to a grip position / orientation, and / or to reposition the user's eyes relative to the device so that the device is viewed from the optimal viewing angle in order for the user to view the correct UI.

[0060] In contrast, for the embodiment implementing dynamic folding angle adjustment, UI distortion is dynamically generated and adjusted taking into account the actual folding angle of the device; that is, the UI can deform as the actual folding angle changes. Similarly, for the embodiment implementing dynamic viewing angle adjustment, the generated UI distortion does indeed take into account the user's eye position / gaze point. In other words, the UI is dynamically generated and adjusted based on the user's actual eye position / gaze point. With dynamic viewing angle adjustment, the user does not need to adjust their grip position / orientation, etc., and can view the correct UI from virtually any viewpoint available to the user.

[0061] For simplicity, unless the context otherwise requires, the following description of UI manipulation techniques will focus primarily on configurations employing static folding angle UI adjustment and dynamic viewing angle UI adjustment, i.e., configuration (B). It is observed here that configuration (B) may be preferred in many field applications due to at least two advantages: first, the static folding angle forces the user to change the folding angle, thus helping to protect sensitive data from onlookers; second, the dynamic viewing angle alters UI distortion based on the user's eye position, providing convenience and increasing usability. In comparison, configuration (A) may be preferred in low-cost and / or low-power implementations, where additional sensor systems such as user eye-tracking systems may be prohibitively expensive and / or power-intensive. On the other hand, configuration (D) relatively offers the most seamless user experience because the UI is visible to the user regardless of their current folding angle or gaze point; however, it is less secure than the configuration with a static folding angle because users are less inclined to make any adjustments (e.g., fold the phone more narrowly) if the UI visibility remains constant for the user.

[0062] Although configuration (B) is preferred, the disclosed techniques can also be similarly applied to configurations (A), (C), and (D). Specifically, in the following description discussing the determination and generation of the optimal folding angle, the same or similar considerations (e.g., information about the inside and / or outside of the user device) can be applied to determine and generate the optimal viewing angle (e.g., for those implementations with static viewing angle adjustment). In some embodiments, the user may be given the option to choose between different configurations.

[0063] Furthermore, it should be noted that in some embodiments, UI distortion may be “updated” based on a determined level of sensitivity to user privacy information and / or a determined level of privacy risk. While such “updates” to UI distortion can generally be understood as “dynamic” in some sense, since these updates are made in response to changes in the sensitivity of privacy risks in the surrounding environment or in either or both of the displayed data, this type of argument becomes clear in the context and is not confused with the general argument for dynamic UI manipulation in response to the user’s gaze point and / or folding angle (i.e., parameters within the user’s control).

[0064] User interface manipulation for sensitive data protection

[0065] Figure 2A An example of how the foldable phone user interface described here can be implemented for privacy and sensitive data protection is shown. Figure 2A The image shows a foldable phone 230 with a bi-fold display 220 that folds along a hinge axis 236. Figure 2AExample UIs 201-208 are also shown, illustrating how to manipulate the foldable phone 230 in four different combinations for four different example privacy scenarios (i.e., the first group is example UIs 201-202, the second group is example UIs 203-204, and so on) for different folding operations in the following four different example privacy scenarios: (1) private space, (2) relatively safe space, (3) public but quiet space, and (4) busy public space. In the following text, the privacy scenarios and their corresponding grip angles are for discussion purposes only; depending on the implementation, more or fewer scenarios may be used, and different grip angles may be used.

[0066] Example UIs 201, 203, 205, and 207 illustrate the UI presented to the user (e.g., user 110) in different privacy scenarios, and example UIs 202, 204, 206, and 208 illustrate the actual UI displayed on the bi-fold screen 220. Specifically, the actual UIs in example UIs 202, 204, 206, and 208 represent how the user's private / sensitive information can be laid out on the user interface of a phone with a bi-fold display (e.g., bi-fold display 220). In other words, the "actual UI" represents exactly what the user interface looks like to the user in each scenario when the foldable phone 230 is fully open. In contrast, the "user-seen UI" represents what the user's private / sensitive information looks like on the user interface of the phone 230 when the bi-fold display 220 is folded at the optimal folding angle (and, in some embodiments, viewed from the optimal viewing angle) for each scenario.

[0067] Taking the first privacy scenario—private space—as an example, Example UI 202 shows the actual UI presentation of a user's private / sensitive information on the phone 230 when the dual-fold display 220 of the foldable phone 230 is fully open. In comparison, Example UI 201 shows how the actual UI 202 visually presents itself to the user when both parts of the dual-fold display 220 (i.e., the left and right sides) are fully open and occupy the entire area of ​​the foldable display 220. It's important to understand that users generally prefer to keep the foldable phone 230 fully open in their private spaces, such as at home, as this makes full use of the display area of ​​the foldable display 220. It should be noted that in the fully open state, the actual UI (i.e., Example UI 202) is the same as the UI presented to the user from their optimal viewing angle (i.e., Example UI 201).

[0068] In the second example privacy scenario—an office cubicle—because it is a location that can generally be categorized as relatively safe but not as safe as a user's home, it may not be advisable for the user to keep phone 230 fully open, as there is still a small possibility that a bystander might be watching the user's phone. In this case, keeping phone 230 slightly folded but at a wide angle may be sufficient to protect the user's private / sensitive information from being stolen by a bystander. Example UI 203 shows the actual UI presentation of the user's private / sensitive information on phone 230 in a scenario where the user is in a relatively safe space. In such an environment, the optimal view for the user can be achieved when both parts of the dual-fold display 220 are folded to a wide angle (e.g., 135° or greater). By comparison, the UI as seen by the user (e.g., UI 203) shows what the actual UI 204 looks like to the user when both parts of the dual-fold display 220 are folded to the optimal folding angle for a relatively (but not completely) safe space. It should also be noted here that UI 203 as seen by the user refers to what the actual UI 204 looks like to the user when viewing the foldable phone 230 from the optimal viewing angle (see above regarding...). Figures 1C-1D (Description of UI 203). The UI 203 seen by the user is a user interface that, in a relatively safe space, appears smaller than the entire area of ​​the display at the optimal folding angle (e.g., compared to UI 201) but is not distorted to the user. At the same time, the same user interface appears distorted to an observer (e.g., UI 204).

[0069] Similarly, in locations generally categorized as public yet still quiet spaces—such as a third privacy scenario, a park—keeping the phone 230 folded at a narrow angle may be desirable for the user, especially compared to the second scenario where the user is in a safer place like an office. This is because there is a greater likelihood of bystanders watching the user's phone compared to a safer place like an office. In this situation, keeping the phone folded at a narrow angle protects the user's private / sensitive information from being intercepted by bystanders. Example UI 206 is the actual UI presentation of the user's private / sensitive information on the phone 230 in a scenario where the user is in a public yet still quiet space. In such an environment, the optimal view for the user is achieved when both parts of the dual-fold display 220 are folded at a narrow angle (e.g., 135°–75°). User-seen UI 205 shows what the actual UI 206 looks like to the user when the foldable phone 230 is folded to the optimal folding angle for a public yet still quiet space. In this way, even if the display area perceived by the user becomes smaller than the entire area of ​​the foldable display 220 (e.g., compared to UI 201), the UI 205 seen by the user remains undistorted at the optimal folding angle. However, the same user interface becomes more obscured to an observer due to its narrower folding angle and appears more distorted to the observer compared to the second scenario.

[0070] Finally, in locations generally categorized as busy public spaces—such as the fourth privacy scenario, a bus or subway—folding the phone 230 to a narrower angle than in previous scenarios (e.g., a park) may be preferable for the user. This is because there is a greater likelihood of bystanders staring at the user's phone on a bus compared to a park. In this situation, keeping the phone 230 folded at a very narrow angle can protect the user's private / sensitive information from being stolen by bystanders. Example UI 208 is an actual UI representation of the user's private / sensitive information on the phone 230 in a scenario where the user is in a busy public space. In such an environment, the optimal view for the user is achieved when both parts of the dual-fold display 220 are folded to the narrowest angle among all four scenarios (e.g., 75° or less). User-seen UI 207 shows what the actual UI 208 looks like to the user when the foldable phone 230 is folded to the optimal folding angle for a busy public space. Figure 2AAs shown, even though the user-perceived display area is the smallest of the four (compared to UIs 201, 203, and 205), UI 207 as seen by the user remains undistorted at the optimal folding angle. However, compared to the first three scenarios, example UI 208 is the most concealed to an observer due to its narrowest folding angle and appears the most distorted to an observer compared to all previous scenarios. It should be noted that in embodiments where the foldable display is configured to fold at or along the center (e.g., a double-fold display configuration), the user interfaces implementing this technology (e.g., UIs 204, 206, and 208) can deform symmetrically about an axis (e.g., axis 236) along the hinge of the foldable display (e.g., display 220).

[0071] In this way, the disclosed UI manipulation / distortion technology provides an intuitive user interface system where the content of the user interface may be visually distorted in the actual UI representation, and appears undistorted and readable only when the display (e.g., display 220) is folded to a specific angle. This enables the protection of the user's private and sensitive information from theft by a bystander (e.g., bystander 140), as the bystander can only see the distorted user interface display, such as those examples shown as the actual UI representation (e.g., example UIs 204, 206, and 208). Furthermore, bystanders are further prevented from stealing the user's privacy and sensitive information if the user folds or closes the display at a specific folding angle to obstruct the bystander's view of the readable UI (e.g., those examples shown as the UI representation seen by the user (e.g., example UIs 203, 205, and 207)).

[0072] Figure 2B Examples of visual guide markers 250 and text guides 252 presented in the actual UI and the UI seen by the user are shown. As discussed here, it is possible to generate a UI to be displayed on a foldable or flexible display based on a selected folding angle, so that the UI will only appear visually normal to the user when the foldable phone's screen is folded to the selected folding angle. The UI generated from this technique is intuitive to the user, at least because when the phone is not in the selected configuration (e.g., folded to the selected folding angle), the content will look abnormal (e.g., distorted or crooked).

[0073] Nevertheless, to further enable users to intuitively configure and / or direct calls, and to reduce reliance on distortion of content displayed to guide users, the generated UI may additionally include visual guidance markers (e.g., guidance marker 250) that guide the user to manipulate the foldable display to a selected folding angle. In some implementations, visual guidance marker 250 may be displayed at the top of the display or in a location that does not obstruct normal content in the user's UI. Visual guidance marker 250 may be a selected geometry, such as a circle, square, triangle, rhombus, etc. By implementing the disclosed UI manipulation technique, the visual guidance marker will be presented to the user as the selected geometry (e.g., as shown in the image) only when the foldable display is manipulated to the optimal folding angle. Figure 2B The "UI as seen by the user" representation shows a circle. For example, when the selected geometry is a circle, the actual UI may appear to the user in a shape other than a circle when the foldable display is not manipulated to the optimal folding angle (e.g., as shown in the image). Figure 2B The “actual UI” refers to the tilted ellipse shown in the diagram. In other words, the geometry on the intuitive user interface only appears circular when the foldable display is manipulated to the optimal folding angle. In some embodiments, the actual UI may include additional on-screen prompts indicating to the user what the selected geometry is (it may also be part of text guidance 252, discussed below). The user can then adjust the phone to a narrower angle until the ellipse appears circular. From the user's perspective, the user interface becomes readable and visually normal, but the content may appear obscured or visually distorted to bystanders around the user.

[0074] Additionally or alternatively, the generated UI may include visual guidance markers that guide the user based on information about... Figures 8-11D The discussion concerns grip manipulation techniques for manipulating a person's or her grip. Therefore, in one or more embodiments, the generated UI may include visual guide markers for UI manipulation techniques and visual guide markers for grip manipulation techniques. However, the visual guide markers may be generated independently of each other. The visual guide markers for grip manipulation techniques may be generated independently of UI manipulation techniques (i.e., without visual distortion) or in conjunction with UI manipulation techniques (i.e., with visual distortion).

[0075] Furthermore, or alternatively, one or more embodiments of the UI manipulation system herein may include text guidance (e.g., text guidance 252) on the user interface to instruct the user to change the current folding angle to the optimal folding angle. For example, text guidance 252 may instruct the user using words such as “open the foldable display further” or “close the foldable display further.” In some variations, the text guidance may be displayed as an instruction in a currently readable font (i.e., in a non-distorted manner), such as “turn off the phone until the oval becomes a perfect circle.”

[0076] In this way, visual guidance 250 and / or textual guidance 252 can instruct the user to fold the phone precisely so that the user interface is readable from the user's position, thus preventing potential onlookers from seeing the user's private information. As a practical example, a user wants to check their bank account information while riding a bus. When the user accesses the information on their foldable phone, the phone presents a distorted-looking UI. At the top is an oval with an instruction in currently readable font: "Please fold the phone to a narrower angle until the black oval becomes a perfect circle (and the UI looks straight and horizontal)." The user folds the phone to a narrower angle for the phone until the oval becomes a circle. After this adjustment, the user interface is now fully readable from the user's position, but its content is now obscured from potential onlookers.

[0077] Furthermore, in some examples, the UI manipulation system may require the user's device to be in an optimal configuration before displaying sensitive information to the user on the UI. For example, the UI manipulation system may be configured to display only secondary UI elements and actively occlude, blur, cover, otherwise obstruct, or simply prevent sensitive data from being displayed on the user's device screen unless / until the foldable display is manipulated to an optimal device configuration (e.g., optimal folding angle). For example, the UI may only display guidance information to the user (e.g., visual markers 250 and / or text guidance 252) until the user follows the guidance instructions and reconfigures or manipulates the foldable phone to the selected optimal folding angle. Only then will the UI manipulation system display sensitive information on the UI (e.g., as described above).

[0078] Figure 2CAdditional, optional implementation details of some examples of foldable phone UI manipulation techniques are shown. As discussed herein, because at least several examples of the described UI manipulation techniques have the visual effect of reducing or shrinking the display area of ​​the foldable display perceived by the user (e.g., comparing UI 201 with UIs 203, 205, and 207), in some implementations, the content of the display can be automatically adjusted based on a determined effective shape factor of the foldable display. Specifically, in various embodiments, the original layout of sensitive information can be altered or rearranged into an alternative layout based on the determined shape factor, such that the alternative layout is adapted to a shape factor smaller than the entire area of ​​the foldable display. For example, the original layout (e.g., one designed for a larger display, such as a desktop / tablet display) can be changed into an alternative layout designed for mobile devices with smaller or smaller screens (e.g., different font sizes, graphics, and / or navigation arrangements).

[0079] Figure 2C Two sample UIs, 270 and 272, are shown, with sample UI 270 illustrating the original layout of the content to be displayed when the screen is fully open. When the device implementing the UI manipulation technology is folded, the displayed layout can be automatically changed to an alternative layout, as demonstrated by sample UI 272. Additionally or alternatively, additional UI elements such as scrollbars 274 can be added to the displayed UI to help users navigate content within the simulated, scaled-down screen size.

[0080] Example system components and data flows

[0081] Figure 3 A functional block diagram of an example UI manipulation engine 300 is shown, which can be used to implement the foldable phone UI manipulation technology described here.

[0082] The example UI manipulation engine 300 can be implemented and run on the user's mobile device (e.g., mobile device 130, 230). Figure 3 As shown, the UI operation engine 300 may include a UI sensitivity extraction system 310, an ambient environment state assessment system 320, an optimal display configuration discoverer 330, an actual display configuration discoverer 340, a user state assessment system 350, a UI distortion generator 360, and an auxiliary UI element generator 370. It should be noted that... Figure 3The components shown are merely illustrative; for simplicity, certain well-known components such as the central processing unit (CPU), memory module, screen display(s), and communication circuitry are not shown. It should be noted that one or more systems, subsystems, components, and / or modules described herein can be implemented as or include software applications; however, any suitable combination of these systems / subsystems / modules can be implemented as hardware or firmware components to achieve the same or similar functionality. Similarly, Figure 3 The example components shown are for the convenience of the discussion in this article; depending on the actual implementation, more or fewer components may be used, and components may be combined or split.

[0083] Depending on the implementation, the UI manipulation engine 300 can be activated when it receives an instruction to display sensitive information (e.g., on a foldable display of a mobile device). For example, an application programming interface (API) can be implemented for the engine 300, allowing it to receive software function calls (e.g., from software applications) as instructions. In variations, the engine 300 can be implemented as part of a feature set provided by an operating system (OS), or implemented in a system software library, etc. In additional or alternative embodiments, the engine 300 can be standalone software residing in memory and intercepting / detecting the sensitive information to be displayed.

[0084] Upon receiving an indication that sensitive information exists to be displayed, the UI manipulation engine 300 determines the optimal display configuration for that sensitive information. To do so, the engine 300 considers one or more factors, including those based on external and / or internal information. More specifically, according to the UI manipulation techniques described, there are generally two types of information that influence the determination of how much the perceived display should be scaled down (i.e., its shape factor): (1) those information fragments inside the phone, and (2) those outside the phone. The former (i.e., internal information) can be extracted by the UI sensitivity extraction system 310, and the latter (i.e., external information) can be extracted by the surrounding environment state assessment system 320.

[0085] A. UI Sensitivity Extraction System

[0086] Figure 4A It is for use Figure 3 Example data flow 400 is provided, using sample components of the UI manipulation engine to implement foldable phone UI manipulation technology. Data flow 400 illustrates an example of how data flows between various blocks within the UI manipulation system (e.g., engine 300). See also... Figure 3 and Figure 4A Now, let's explain the data flow 400 together with the function block diagram of the UI manipulation engine 300.

[0087] The UI sensitivity extraction system 310 can determine the privacy sensitivity of data displayed on a user's device and output UI sensitivity data. In other words, the UI sensitivity extraction system 310 can determine the sensitivity level of information to be presented on a user's mobile device.

[0088] In some examples, UI sensitivity data can describe various sensitivity levels of different features currently displayed on the screen. Examples of high-sensitivity items may include password input fields, bank account information, credit card information, etc. Depending on the implementation, medium sensitivity may include personal information such as Social Security numbers, birthdays, home addresses, or phone numbers. The UI sensitivity extraction system 310 can also identify low-sensitivity items, such as public domain information, generic website text, or images. In several embodiments, users may have the option to adjust or edit these items according to their own preferences. Alternatively or concurrently, UI sensitivity data can describe the overall sensitivity rating of the screen that matches the highest sensitivity feature on the page.

[0089] According to some embodiments, the UI sensitivity extraction system 310 may obtain UI sensitivity data from local and / or remote resources. Examples of such resources may include lookup tables or databases in which sensitivity ratings for common software applications / components / functions may be recorded. For example, one or more resources may be identified as highly sensitive when: the software mobile application is a banking application; the application window displays a password input screen; a webpage providing a specific function (e.g., viewing a bank statement) is triggered; or a UI element contains a password input field. As an additional or alternative embodiment, the UI sensitivity extraction system 310 may obtain UI sensitivity data through metadata, such as message metadata related to who the sender is, metadata related to the nature of the message, the urgency of the message, and / or the address of the recipient (e.g., distinguishing sensitivity based on work or personal email). Furthermore, in some variations of the API implemented for the UI manipulation engine 300, the UI sensitivity extraction system 310 may receive UI sensitivity data from the API (e.g., via one or more parameters in a function call).

[0090] In one or more embodiments, the UI sensitivity extraction system 310 can obtain UI sensitivity data through content analysis. In some examples, the UI sensitivity extraction system 310 can determine the sensitivity level based on the presence of keywords (e.g., “highly confidential”) in sent or received messages, or by analyzing graphic / photographic content (e.g., by employing known image processing and recognition algorithms). Some embodiments of the UI sensitivity extraction system 310 can also generate UI sensitivity data by using machine learning techniques (e.g., supervised machine learning). For example, the UI sensitivity extraction system 310 can first be trained with a set of training data whose sensitivity is labeled to build a UI sensitivity determination data model. Examples of training data may include actual content (e.g., text, documents, images, or videos) and their corresponding sensitivity levels. In this way, the UI sensitivity extraction system 310 can dynamically generate sensitivity ratings for sensitive UI content.

[0091] Furthermore, some variants of the UI sensitivity extraction system 310 can generate UI sensitivity data through user sentiment analysis, for example, by inferring potential sensitivity from changes in the user's emotions. For instance, if a user behaves more discreetly, changes facial expressions, or displays emotions while operating the application, the UI sensitivity extraction system 310 can infer that the application's sensitivity level is higher than that of typical applications. The determination of these inferred sensitivity levels can be based on, for example, gesture recognition, facial expression recognition, and / or electroencephalography (EEG).

[0092] B. Surrounding Environment Status Assessment System

[0093] The ambient environment status assessment system 320 includes a set of system hardware, firmware, and / or software components that can be used to determine and analyze the privacy status of a user's ambient environment. For example... Figure 3 As shown, the ambient environment state assessment system 320 includes three example components: an ambient environment state sensor 322, an ambient environment state data preprocessor 324, and an ambient environment state discoverer 326.

[0094] More specifically, an ambient environment state assessment system 320 can be used, for example, based on readings from an ambient environment state sensor 322, to identify the level of privacy risk in the mobile device's surrounding environment. An optimal display configuration discoverer 330 and a UI distortion generator 360 can use the identified privacy risk level to determine the optimal shape factor for the user's private / sensitive information to be displayed on the mobile device. In some embodiments, the shape factor for the user's private / sensitive information to be displayed on the mobile device can be negatively correlated with the identified privacy risk level. That is, the higher the identified privacy risk level, the smaller the shape factor of the UI seen by the user should be. For example, as mentioned above regarding... Figure 2AThe explanation can be divided into four privacy risk levels: private space, relatively safe space, public but quiet space, and busy public space. The busier the surrounding environment, the smaller the UI should be perceived by the user.

[0095] The ambient environment state sensor 322 may include one or more sensors that can be used to obtain information about the user's surrounding environment (i.e., "ambient environment data"). In various examples, ambient environment data may include visual images or recordings of the user's location, specific geographic location information (e.g., Global Positioning System (GPS) coordinates), audio information (e.g., background sound data and / or voice data from the surrounding environment), device connectivity information (e.g., information about devices used to detect people near the user, such as connectivity data), and / or other suitable data that can be used to determine the presence and status of people around the user. Examples of sensors that can be used to obtain suitable ambient environment data include sensors such as: rear and / or front cameras capable of generating light feeds from the user's location, or other light-based (e.g., infrared) imaging technologies; GPS sensors capable of generating satellite positioning information; microphones or audio sensors capable of generating audio signals collected from the user's location; and / or wireless network transceivers (e.g., Bluetooth) capable of generating device signatures within a range (e.g., 10 meters, depending on the transceiver's range) of the user's mobile device. Depending on the embodiment, these sensors may be on the user's mobile device or may be connected to the device separately.

[0096] Specifically, according to one or more embodiments, the ambient environment state sensor 322 can then send ambient environment data to an ambient environment state data preprocessor 324, which can preprocess the ambient environment data (e.g., to improve the signal-to-noise ratio (SNR) to an appropriate quality) for subsequent analysis. Furthermore, the ambient environment state data preprocessor 324 can apply various applicable types of preprocessing to the ambient environment data based on the type of data collected. For example, in some embodiments, the ambient environment state data preprocessor 324 can preprocess optical data generated from optical sensors (e.g., cameras) in the ambient environment state sensor 322 to correct for scene lighting, focus, vibration, motion artifacts, reflections, and other features. Furthermore, in many such embodiments, the ambient environment state data preprocessor 324 can use machine vision techniques to preprocess the optical data to identify and label (e.g., by adding metadata or tags) key features, such as people or faces.

[0097] Furthermore, some implementations of the ambient environment state data preprocessor 324 can preprocess audio data to remove background noise (e.g., wind or road noise), amplify specific audio features (e.g., speech), and further improve audio quality by enhancing the SNR. Similar to the video preprocessing embodiments described above, in some embodiments, the ambient environment state data preprocessor 324 can preprocess audio data to add metadata or tags to key features, such as different speech or unintentionally heard phrases.

[0098] Furthermore, in some embodiments, the ambient environment state data preprocessor 324 can preprocess connectivity data to identify, count, and potentially locate unique device signatures near the user. For example, the ambient environment state data preprocessor 324 can utilize information included in Wi-Fi management frames, requests, beacons, and Ping to identify, count, and / or locate unique device signatures and / or geographic locations. In some examples, the ambient environment state data preprocessor 324 can also preprocess GPS / satellite data to identify certain qualitative attributes of the user's current location (e.g., store name, road name, or landmark). This GPS / satellite / location data can be combined with existing map services on the mobile phone (e.g., Google Maps). Depending on the implementation, the preprocessing of location data can leverage machine learning techniques (e.g., to filter and classify features).

[0099] Next, the ambient environment state discoverer 326 can assess the current state of the user's ambient environment (i.e., "ambient environment state") by analyzing the preprocessed ambient environment data generated by the ambient environment state data preprocessor 324. Subsequently, the ambient environment state discoverer 326 can determine the privacy risk level of the user's ambient environment based on the ambient environment state. In one or more implementations, the ambient environment state discoverer 326 can extract relevant information from the ambient environment data when determining the ambient environment state.

[0100] Specifically, in one or more examples, the ambient environment data may include the number of other people nearby (e.g., bystanders). In some of these examples, when determining the number of other people nearby, the ambient environment state discoverer 326 may perform a count of feature tags for different people added to the camera feed during preprocessing of the ambient environment state data preprocessor 324. In other embodiments, the ambient environment state discoverer 326 may determine the number of other people nearby by counting the number of nearby users using machine vision facial recognition technology. In variations, the number of other people nearby may be determined by counting the number of nearby devices detected via Bluetooth technology (or other suitable short-range wireless connectivity technology). Alternatively or additionally, the ambient environment state discoverer 326 may determine the number of bystanders by counting the number of unique human voices identified during ambient environment data preprocessing. Certain variations of the embodiments may provide a simple estimate of the total sound level in the ambient environment based on the variation in audio peak amplitude over a given time interval.

[0101] Furthermore, in some embodiments, the ambient environment data may include the spatial distribution and gaze direction of nearby people. In some of these examples, the ambient environment status discoverer 326 may detect this information using camera data and machine vision techniques, for example, to identify someone standing behind the user and facing the user device (which would be a security threat), in contrast to someone standing to the user's left or right but possibly with their back to the user device (which would be a lesser security threat). In many implementations, the ambient environment data may include known attributes associated with the user's location, such as predicting the busyness / publicity of the user's location based on a predictive assessment of the user's GPS coordinates. For example, the ambient environment status discoverer 326 may detect from the user device's GPS data that the user is in a public place, thus inferring that others may be nearby. In another example, the ambient environment status discoverer 326 may detect from GPS data that the user is in the grounds of a large state park, thus inferring that others are unlikely to be nearby. In some embodiments, the ambient environment status discoverer 326 may further associate the known attributes of the user's location with the network connectivity of the user device. For example, when a user device is connected to “home Wi-Fi” or “car Wi-Fi”, the ambient environment status discoverer 326 can determine that the privacy level is high (i.e., the risk level is low). Conversely, when a user device is connected to “public Wi-Fi” or “airport free Wi-Fi”, the ambient environment status discoverer 326 can determine that the privacy level is low (i.e., the risk level is high).

[0102] Based on ambient environment state data and through the above-described example, the ambient environment state discoverer 326 can generate an ambient environment state output indicating the likelihood that a user's data may be exposed while using the user's device. In other words, the ambient environment state output by the ambient environment state discoverer 326 can convey the level of privacy risk in the user's surrounding environment to other components of the UI manipulation engine 300. In some embodiments, the communication of the ambient environment state to subsequent system components can take the form of a specific description of several individuals who pose a data exposure risk to the user, including their location, gaze angle, distance from the user, and other information. In variations, the communication of the ambient environment state to subsequent system components can take the form of a general metric of user risk (e.g., "data exposure risk"), where the ambient environment state discoverer 326 can apply weights to various aspects of the analyzed ambient environment data to determine the likelihood that the user's data may be exposed at the current location. In some additional or alternative examples, the ambient environment state discoverer 326 can directly generate a rating or privacy risk level of the user's surrounding environment.

[0103] C. Optimal Display Configuration Finder

[0104] The optimal display configuration discoverer 330 can determine the optimal physical configuration (i.e., the "optimal display configuration") of a foldable or flexible screen to prevent it from being observed by other people nearby. Specifically, the optimal display configuration discoverer 330 can use (1) the sensitivity level of private and sensitive information presented on the user's mobile device, and / or (2) the privacy risk level of the user's surrounding environment to identify a physical configuration in which the likelihood of the screen content being seen by unethical bystanders is reduced, minimized, or even eliminated. In other words, the optimal display configuration discoverer 330 can use UI sensitivity data (e.g., generated by the UI sensitivity extraction system 310 as described above) and / or the surrounding environment state (e.g., generated by the surrounding environment state assessment system 320 as described above) to determine the optimal privacy configuration of the display (e.g., folding angle), for example, to prevent the screen content from being misused by bystanders.

[0105] Specifically, in some embodiments, the optimal display configuration discoverer 330 can determine the optimal folding angle based on information internal to the user device (i.e., the sensitivity level of the user's private information determined by the UI sensitivity extraction system 310). In some other embodiments, the optimal display configuration discoverer 330 can determine the optimal folding angle based on information external to the user device (i.e., the privacy risk level determined by the ambient environment state assessment system 320). Additionally or alternatively, the optimal display configuration discoverer 330 can determine the optimal folding angle based on both the determined sensitivity level of the user's private information and the determined privacy risk level. For embodiments that implement static viewing angle adjustment, the optimal display configuration discoverer 330 can also determine the optimal viewing angle.

[0106] Then, some embodiments of the optimal display configuration finder 330 can determine the optimal folding angle, i.e., the optimal inner angle at which the display is folded (see...). Figure 1B (As described above). For embodiments where the user device's display includes a single connector (e.g., a single hinged display or multiple displays), the optimal display configuration may be the optimal folding angle at which the display is folded. For example, an acute folding angle may provide greater viewing privacy (at the expense of a smaller perceived size of the display) and is optimal for areas with high privacy risks and / or highly sensitive content; in contrast, an obtuse folding angle may provide lower viewing privacy while providing a larger perceived size of the display and is best suited for areas with low privacy risks and / or low-sensitivity content.

[0107] The optimal display configuration can vary depending on the physical characteristics of the user device. For example, in some embodiments, the device may be a device with two or more folded sections designed to have a specific predetermined folding configuration. In such a device, the folding angle and rotation direction of the screen may be limited, so that for each folded section, the optimal display configuration may be a combination of the folding angle plus a certain rotation direction vector. Furthermore, in some examples, the device may be a device with a general flexible display structure. Such a device may be a single folding screen with dynamically configurable folding behavior, which is not limited to predetermined folding behavior. In such a device, the folding angle and rotation direction of the screen are less constrained. Thus, the optimal display configuration may include a specific selected shape, and this configuration may also include, for example, the direction in which the screen should fold and the position of the fold center (e.g., the fold line) on the screen surface. This combination of folding angle and fold line can, for example, be used by an auxiliary UI element generator 370 (discussed below) to guide the user to fold the display fabric into the selected shape. In this context, many examples of fold lines can be similar to the hinge axis described here for foldable (e.g., bi-fold) displays.

[0108] It should be noted that, in some examples, data from the accelerometer / gyroscope sensor built into the user device can be used to identify how the user is currently gripping the device, and with this data, the optimal display configuration discoverer 330 can determine whether the optimal folding angle can be physically achieved by folding one side or both sides of the device.

[0109] Once the optimal display configuration is determined, it can be output by the optimal display configuration discoverer 330. For purely static UI operation embodiments (i.e., those embodiments where the UI does not deform / change with changes in the actual folding angle or the position of the user's eye), such as the configuration (A) described above, the optimal folding angle and / or viewing angle can be output to the UI distortion generator 360 for UI rendering and display.

[0110] D. Actual display configuration discoverer

[0111] Figure 4B It is for use Figure 3 Example data flow 402 is provided, using sample components of the UI manipulation engine to implement foldable phone UI manipulation technology. Data flow 402 illustrates an example of how data flows between various blocks within the UI manipulation system (e.g., engine 300). See also... Figure 3 and Figure 4B Now, let's explain data flow 402 together with the function block diagram of UI manipulation engine 300.

[0112] As previously described, several embodiments of the disclosed UI manipulation techniques employ dynamic UI manipulation, i.e., UI distortion changes or deforms when the user's actual gaze point changes (e.g., in configuration (B)) or when the actual folding angle of the user's device changes (e.g., in configuration (C)), or both (e.g., in configuration (D)). One or more system components (e.g., the actual display configuration discoverer 340 or the user state assessment system 350) are described below, upon which these embodiments rely as the basis for their dynamic UI adjustments.

[0113] The actual display configuration discoverer 340 can determine the current configuration (or "actual display configuration") of the user device. For example, the actual display configuration discoverer 340 can determine the actual display configuration using device data obtained from one or more display configuration sensors built into the device. Examples of information obtainable from these display configuration sensors may include: the current folding angle and / or folding orientation configuration, device position, and / or device orientation (e.g., tilt angle). In some examples, device orientation can be obtained from data from an accelerometer, gyroscope, or other suitable sensor on the user device. This data can be used to identify how the user is currently gripping the device and / or how the user should move the mobile device to achieve the optimal viewing angle (e.g., in embodiments where viewing angle adjustment is static).

[0114] In some embodiments, the display configuration sensor used by the actual display configuration discoverer 340 to collect device data may include a mechanical sensor. For example, in a dual-panel (e.g., two-part or double-folded) flexible display, the mechanical sensor may be incorporated into the hinge that engages the two panels. In some examples, the sensor may include a flexible electronic material that changes resistance when the hinge is closed and opened, thereby enabling the measurement of the screen's folding angle. The folding angle data can be used to identify how the foldable screen is currently folded and / or how the user should reconfigure the foldable screen to achieve an optimal folding angle (e.g., for embodiments where the folding angle is adjusted to a static position).

[0115] For typical flexible displays (e.g., fabric displays), some embodiments of a practical display configuration may include display distortion at points distributed across the surface of the display, with a density high enough to describe surface distortion across the entire surface. For example, in many such embodiments, the practical display configuration may include data representing a vector field for each point on the surface, the vector field having scalars of distortion magnitude and distortion angle.

[0116] Furthermore, some variations of the display configuration sensor may also include one or more cameras. For example, in a typical flexible display embodiment, optical data collected from a camera (which may be built into the device or from a third-party device) can be used by the actual display configuration finder 340 to create a normal map or vector field of the current device surface. In some additional or alternative embodiments, the actual display configuration finder 340 may also use one or more cameras to determine relative device orientation and estimate the spatial position of the device. Additionally, in some embodiments, the actual display configuration finder 340 is capable of utilizing camera data and image processing techniques to capture the actual display configuration in 3D, which can generate 3D model files, such as STL files.

[0117] In various implementations, the display configuration sensor may include a time-of-flight (ToF) sensor, which measures the distance to an object using the time it takes for light to travel to and from the object and reflect back. For example, the actual display configuration finder 340 may use a built-in ToF sensor to determine the device's position by measuring the distance between the device and a person or other physical feature. Alternatively, the display configuration sensor may include radar. For example, the actual display configuration finder 340 may use radar or other object or motion sensing technologies to determine spatial relationships between display elements or between display elements and the user.

[0118] E. User Status Assessment System

[0119] User status assessment system 350 can be a set of system components used to determine and analyze the physical status of a user. User status assessment system 350 may include example components such as user status sensor 352 and user status discoverer 354.

[0120] User state sensor 352 includes one or more sensors that user state assessment system 350 can use to obtain data about the user's state (or "user state"). More specifically, examples of user state that can be acquired by user state sensor 352 may include: user gaze point—the position on the screen that the user is looking at; head angle—the angle of the user's head relative to the display; viewing angle—the angle of the user's eyes relative to the display; and / or head orientation—the orientation of the user's entire body relative to the display. Depending on the implementation, user state sensor 352 may be a dedicated eye-tracking sensor system or a combination of sensors on the user device. Example components of user state sensor 352 may include, for example, one or more front-facing cameras, motion sensors (which may be on the user device or worn by the user, such as a smart bracelet), a suitable projector, and / or a time-of-flight sensor. For example, a projector may project specific light or light patterns (e.g., infrared or near-infrared light) onto the user's eyes, and a camera may capture images of the user's eyes and the pattern, which can then be applied using image processing and machine vision techniques to determine the location of the user's eyes and gaze point. In another example, a time-of-flight sensor may detect the distance of the user from the screen.

[0121] F.UI Distortion Generator

[0122] UI Distortion Generator 360 generates visual distortion in the user interface (also referred to here as "UI distortion," "UI warping," or "UI manipulation"). UI distortion manipulation alters the visibility of the UI so that the optimal view of the user interface can only be obtained from the user's current perspective under the optimal display configuration (e.g., at the optimal folding angle). From any other bystander's perspective, the display's visibility is reduced (due to the closed folding angle) and becomes blurred (due to visual distortion), making it less likely for a bystander to steal the user's private information from the user's device by peering at it.

[0123] Depending on the implementation, UI distortion can include combinations of different transformations and distortions. For example, UI distortion can include a series of transformations (e.g., translation, rotation, or tilt) to be applied to the user interface. According to several current embodiments, the UI distortion generator 360 can apply the distortion to the user interface (e.g., ... Figure 2A The applications discussed above (such as "optical illusion") utilize "optical illusion" to ensure that the UI and target information on the UI can only be read from a desired angle and / or distance. In some embodiments with "holographic display" hardware, the UI distortion generator 360 can generate a set of instructions, including, for example, which angles the pixel light of different pixels should be adjusted to, so that privacy-sensitive UI elements can only be viewed from certain angles and / or physical device configurations. (As discussed above...) Figure 2A The types and degrees of distortion discussed take into account the sensitivity level of the information being displayed. Transformations or distortions applied to highly sensitive UI content may be more extreme, making the UI content much less readable to people other than the target users (e.g., UI Example 207 compared to UI Example 203).

[0124] In some examples, the UI distortion generator 360 considers user state data and / or actual display configuration when determining and generating UI distortion. For example, using user state data (e.g., user gaze point), the UI distortion generator can adjust the viewing angle of the UI distortion so that it remains visible to the user as their eye position moves. Using the actual display configuration, several embodiments of the UI distortion generator 360 can also continuously modify the UI distortion in response to any change in the folding angle. In this way, the shape factor (e.g., size) of the perceived UI generated by the UI distortion can be updated and adjusted (e.g., in real-time or near real-time) in response to new actual display configurations and / or user state.

[0125] Furthermore, some embodiments of the UI distortion generator 360 can generate instructions for device hardware to compensate for the actual display configuration. In some implementations, the UI distortion generator 360 can generate instructions to increase the pixel brightness in certain portions of the UI element being viewed, for example, to compensate for visual degradation caused by viewing the screen at the outer edge of the view frustum. In other variations, the UI distortion generator 360 can generate instructions for recalibrating the device's 3D control gestures so that the gestures still function in the new UI under new aspect ratios, UI positioning, UI element distortion, and other conditions.

[0126] Furthermore, to improve user experience and usability, some embodiments of the UI distortion generator 360 can modify one or more features of the UI. For example, the UI distortion generator 360 can modify the size of user interface features. Alternatively or concurrently, the UI distortion generator 360 can modify the positioning of UI elements on the display. For example, the UI distortion generator 360 can move elements from the edge of the display to the center to reduce the amount of distortion required during extreme folding. In some embodiments, the UI distortion generator 360 can modify multiple user interface features, including, for example, color, brightness, or pixel distribution. In some examples, when the device is closed to a narrower folding angle (e.g., as mentioned above...), Figure 2C (As an example of the discussion), UI Distortion Generator 360 can rearrange the UI to a layout that is more suitable for smaller displays (i.e., as the user perceives it).

[0127] G. Auxiliary UI Element Generator

[0128] Figure 4C It is for use Figure 3 Example data flow 404 is provided, using sample components of the UI manipulation engine to implement foldable phone UI manipulation technology. Data flow 404 illustrates an example of how data flows between various blocks within the UI manipulation system (e.g., engine 300). See also... Figure 3 and Figure 4C Now, let's explain the data flow 404 together with the function block diagram of the UI manipulation engine 300.

[0129] The auxiliary UI element generator 370 can generate UI elements or features, and / or hardware-implemented features (e.g., stand-alone LED indicators), which can be used to guide the user to achieve the optimal display configuration. More specifically, in some examples, the auxiliary UI element generator 370 may selectively display one or more auxiliary UI elements (e.g., by using UI distortion from the UI distortion generator 360 and / or optimal display configuration parameters from the optimal display configuration finder 330) so that the characteristics of the auxiliary UI elements can change to indicate to the user when the user has correctly operated the folding screen and achieved the optimal display configuration (e.g., the optimal folding angle).

[0130] More specifically, in some embodiments, auxiliary UI elements may include size variations of shapes or patterns that are intuitively understandable to the user, such as distortion of the UI itself and / or additional visual guidance markers (e.g., selected geometry). For example, auxiliary UI elements may include distorted circles (or ellipses, e.g., circles) that only become perfectly round when the screen changes to an optimal display configuration. Figure 2B (Mark 250). In some examples, additional text guidance can be generated on the display, indicating to the user what the selected geometry is (e.g., a circle). To provide more helpful assistance to the user, one or more embodiments specify that the additional text guidance is displayed in an orientation that allows the user to see it even if the device is not configured (e.g., folded) to an optimal display configuration (e.g., at the optimal folding angle) (or prior to this)—such examples are shown as Figure 2B The text guidance in section 252. In some implementations, the content of the text guidance can be generated based on the difference between the actual display configuration (e.g., the current folding angle) and the optimal display configuration (e.g., the selected folding angle). For example, the text guidance could instruct the user to change the folding angle, i.e., to open or close more. In some examples, images can be used as an alternative to geometry.

[0131] As an additional embodiment, auxiliary UI elements may include hardware-implemented features, such as LED indicators when the display is in an optimal display configuration. In variations, holographic display indicators may be included, which can present the correct color or brightness when the display is in an optimal configuration. Furthermore, or alternatively, auxiliary UI elements may include patterns of pixels or illuminated dots that will only be rendered as a specific shape or image once the display is in an optimal configuration. For example, in a fabric-like display, when the fabric is manipulated into an optimal shape, pixels may only be rendered as machine-readable one-dimensional or two-dimensional barcodes (e.g., QR codes), thus serving as confirmation of achieving the optimal display shape.

[0132] Furthermore, in some variations, the auxiliary UI element generator 370 may display only auxiliary UI elements and actively occlude, blur, cover, otherwise obstruct, or simply prevent sensitive data from being displayed on the user device's screen, unless / until the foldable display is manipulated to an optimal device configuration (e.g., optimal folding angle). For example, the auxiliary UI element generator 370 may control the UI distortion generator 360 to occlude UI elements displaying sensitive data and only display guidance information to the user (e.g., the aforementioned visual markers 250, text guidance 252, etc.). Figure 2B (and / or LED indicators); then, the UI manipulation engine 300 only displays sensitive information on the UI (e.g., the information mentioned above) when the user follows the instructions and changes the foldable phone to the selected optimal folding angle. Figure 2A (Description method).

[0133] method

[0134] Figure 5 The diagram illustrates a flowchart of an example method 500 for implementing UI manipulation technology for a foldable phone. Method 500 can be implemented by a UI manipulation engine (e.g., engine 300). Figure 3 Various components of ) (e.g., components 310-370; Figure 3 This is implemented for user equipment (e.g., device 130, 230); Figure 1A , Figure 2A To mitigate UI distortion and rendering on foldable UI displays, sensitive and private information and data on foldable UI displays are protected from bystanders (theft). Method 500 also references... Figures 2A-2C and Figure 3 This is for your introduction.

[0135] First, the UI manipulation engine may receive (510) an instruction to display sensitive information on the display (e.g., display 220) of a mobile device (e.g., device 230). An example of such an instruction could be a software function call. In some examples, the mobile device may include a foldable display. The UI manipulation engine may then determine (520) the optimal display configuration for how the sensitive information is displayed. In several embodiments, the UI manipulation engine first identifies (522) the sensitivity level of the information to be displayed on the user's screen, and then determines a shape factor for how the sensitive information is displayed. Alternatively, the UI manipulation engine may determine (524) the privacy risks (or how busy) in the user's surrounding environment when determining the perceived shape factor of the UI. As mentioned above, the shape factor may generally include the perceived display size, i.e., the size of the display that the user sees when manipulating (e.g., folding) the device to an optimal device configuration (e.g., by folding a dual-foldable display to an optimal folding angle) after UI distortion techniques. Generally, the higher the sensitivity level of the data to be displayed, or the higher the privacy risk in the surrounding environment, the smaller the perceived shape factor (e.g., size) of the displayed UI becomes. In other words, the shape factor can be inversely correlated with the risk from the surrounding environment and / or the sensitivity of the data to be displayed. According to one or more embodiments of the disclosed UI manipulation techniques, the determined shape factor is smaller than the entire area of ​​the foldable display, for example, in privacy scenarios where bystanders may be present.

[0136] Next, the UI manipulation engine can generate (530) a UI to be displayed on the foldable display based on the determined shape factor. For example, the UI manipulation engine can receive (532) the actual phone configuration, and the UI displayed by the UI manipulation engine will show sensitive information in that shape factor (e.g., perceived size), but the UI will be visually distorted to the user except when the foldable display is manipulated to a selected physical phone configuration (e.g., a selected folding angle, and / or, in some embodiments, a selected viewing angle). Figure 2A As shown in the example UI, in a foldable display embodiment, the UI can be displayed jointly by multiple parts of the foldable display. In some embodiments, the foldable display can be a double-foldable display.

[0137] Specifically, in one or more embodiments with a foldable display, UI distortion is generated such that the user interface appears distorted to the user unless the display portion of the mobile device is folded to a folding angle. That is, after determining the optimal folding angle, the user interface appears distorted to the user unless the display portion of the mobile device is folded to that angle. In one or more examples, the UI deforms symmetrically about an axis along the hinge of the foldable display (e.g., as in UI examples 204, 206, and 208). In other examples, the UI distortion engine may, for example, receive the current orientation state of the mobile device from one or more orientation sensors on the mobile device and further adjust the UI in response to the received current orientation state of the mobile device.

[0138] Furthermore, in some embodiments, the UI manipulation engine can detect (534) the user's state, which may be, for example, the user's gaze point. Based on the user's state, the UI manipulation engine can (e.g., in those embodiments implementing dynamic viewpoint adjustment) determine the viewpoint, in which the user interface appears distorted to the user unless the mobile device is oriented relative to that viewpoint. In some of these examples, the UI manipulation engine can determine the position of the user's eyes and adjust the UI in response to the detected position of the user's eyes. In some implementations, the UI manipulation engine may employ a user-facing eye detector on the mobile device and further adjust the UI so that the new optimal viewpoint of the UI reflects the detected position of the user's eyes.

[0139] Figure 6 A flowchart illustrating another example method 600 for implementing UI manipulation techniques for a foldable phone is shown. Method 600 can be implemented by a UI manipulation engine (e.g., engine 300); Figure 3 Various components of ) (e.g., components 310-370; Figure 3 This is implemented to update user equipment (e.g., devices 130, 230); Figure 1A , Figure 2A UI distortion and rendering on foldable UI displays are addressed to protect sensitive, private information and data from bystanders (theft). Also refer to... Figures 2A-2C and Figure 3 Let me introduce method 600.

[0140] The UI manipulation engine may perform UI distortion updates upon receiving or acquiring (610) an update on the actual, current physical configuration of the user device (e.g., the current folding angle). More specifically, depending on the embodiment, the UI manipulation engine may detect one or more of the following new information fragments: it may detect (612) a new instruction to display sensitive data; it may detect (614) a new privacy risk level in the user device's surrounding environment; it may detect (616) a new user state (e.g., a new gaze point or a new eye position of the user); and / or it may detect (618) a new actual device configuration (e.g., a new actual folding angle or a new device orientation).

[0141] Then, consistent with the above discussion (e.g., regarding...) Figure 3 The UI manipulation engine can determine (620) a new optimal device configuration. In some examples, when there is a new or updated level of sensitive data to be displayed (612), and / or when there is a new privacy risk level in the surrounding environment (614), the UI manipulation engine can determine (620) a new optimal device configuration (e.g., a new optimal folding angle and / or, in some embodiments, a new optimal viewing angle). Furthermore, for those embodiments that implement dynamic viewing angle adjustment, when a new user state (e.g., a new gaze point) is detected (616), the UI manipulation engine needs to determine (620) a new optimal device configuration (e.g., a new viewing angle) for UI distortion generation. Similarly, for those embodiments that implement dynamic folding angle adjustment, when a new actual device configuration (e.g., a new actual folding angle) is detected (618), the UI manipulation engine needs to determine (620) a new optimal device configuration (e.g., a new folding angle) for UI distortion generation.

[0142] Subsequently, the UI manipulation engine adjusts (630) the UI based on the updated, new optimal device configuration. In one or more implementations, the UI distortion update is performed substantially in real time as the user manipulates the folded display. In one or more examples, steps selected from 610-630 may be performed repeatedly or recursively.

[0143] Work Example

[0144] Figure 7A This illustrates how to use foldable screen devices (e.g., Figure 1A Equipment 130; Figure 2A The device 230) is a double-folding display (e.g., Figure 1A The monitor is 120; Figure 2A Generated on display 220 (e.g., via UI distortion generator 360); Figure 3A more detailed example of distorted UI. More specifically, after determining the UI shape factor for how to display sensitive information (e.g., as mentioned above regarding...). Figure 3 The actual UI to be displayed on a foldable display is generated based on a determined shape factor. To determine how the UI will be distorted and displayed, the following provides a method for... Figure 7A The equations and explanations for example methods to calculate UI transformations under the above input parameters are provided.

[0145] like Figure 7A As shown, the mobile device 730 includes a foldable display 720 that folds along a central axis 736 (e.g., a hinge). In the following example calculations, it is assumed that the distance (X1) from the central axis 736 to the edge of the foldable display 720 is known. Furthermore, it is assumed that the distance (X2) from the central axis 736 to the user's eye can be detected and / or estimated (e.g., based on sensors that can be mounted on the device 730, such as an eye tracker and / or a ToF sensor). Finally, Figure 7A In the example shown, the folding angle is represented by (α) and the viewing angle is represented by (β).

[0146] Let (Y1) be the obtained length of the edge of the manipulated UI, and (Y2) be the obtained length of the center of the manipulated UI (e.g., ...). Figure 7A As shown). Let (Y0) be the initial length of the edge of the original, undistorted UI (as shown). Figure 7B (As shown). The relationship between Y0, Y1, and Y2 can be represented by the following transformation:

[0147] [Mathematical Expression 1]

[0148]

[0149] T1 is the transformation caused by the change in the folding angle α, and is a function of α, X1, and X2. T2 is the transformation caused by the change in the viewing angle β, and is a function of β, X1, and X2. It should be noted that, unlike the folding angle α, the viewing angle β can have three components because this angle can vary in three dimensions; therefore, T2 can be expressed as:

[0150] [Mathematical Expression 2]

[0151] T2 = T 2xy +T 2yz +T 2zx Equation (2)

[0152] In other words, there exists a transformation for rotation in each of the three orthogonal planes. Furthermore, it should be noted that when α = 180° and β = 0°, Y0 = Y1 = Y2.

[0153] Figure 7C It shows that due to Figure 7A The example calculation involves the change in the folding angle α. Specifically, the example calculation method for T1 is provided below.

[0154] Continue to refer to Figure 7C and Figure 7A If we assume that the viewing angle β is constant, then Figure 7C Point P1 (at the center of the display, where the central axis or hinge is located) is assumed to be static relative to the original plane of the device. Assuming that the thickness of the device is negligible compared to ΔX2, a transformation T1 can be applied to compensate for the increased perceived change in the height of the UI feature at point P2 (at the edge of the display) from the viewer's perspective at a distance X2.

[0155] In this coordinate system, the size of the UI feature at the central folding axis (P1) (e.g., axis 736) of the display remains unchanged, so Y2 = Y0, and there is no need to apply T1 to the UI feature at the central axis.

[0156] To determine the UI feature at point P2 (e.g., at an initial height Y0) at a distance X1 from the origin, a transformation T1 can be applied to compensate for the perceived increase in size due to the feature being closer to the viewer by an amount ΔX2. Therefore, a transformation (assuming a height of Y0) can be applied to the feature at P1 to obtain a smaller height Y1. This transformation can be expressed as:

[0157] [Mathematical Expression 3]

[0158]

[0159] ,in

[0160] [Mathematical Expression 4]

[0161]

[0162] In this way, based on Figure 7C In the example coordinate system shown, transformation T1 can be represented as:

[0163] [Mathematical Expression 5]

[0164]

[0165] Figure 7D It shows that due to Figure 7A The example calculation involves the change in viewpoint β. Specifically, an example method for the calculation in T2 is provided below.

[0166] Continue to refer to Figure 7D and Figure 7AIf we assume the folding angle α is constant, then Figure 7D Point P1 (at the center of the display, where the gaze point is located) is assumed to be static relative to the original plane of the device. Assuming that the thickness of the device is negligible compared to ΔX2, a transformation T2 can be applied to compensate for the increased perceived change in the height of the UI feature at point P2 (at the edge of the display) from the viewer's perspective at a distance X2.

[0167] The viewpoint β in three-dimensional (3D) space has three components, each with respect to rotation in the xy, yz, and zx planes, respectively. Each can be considered independently, but can be calculated using similar equations.

[0168] In the coordinate system described above, the size of the UI feature at the central axis (P1) remains unchanged so that Y2 = Y0, and there is no need to apply T2 to the UI feature at the center.

[0169] For a UI feature at point P2 (e.g., with an initial height Y0) at a distance X1 from the origin, a transformation T2 can be applied to compensate for the perceived increase in size due to the feature being closer to the viewer by an amount ΔX2. Therefore, a transformation can be applied to a feature at P1 (assuming a height of Y0) to obtain a smaller height Y1:

[0170] [Mathematical Expression 6]

[0171]

[0172] ,in

[0173] [Mathematical Expression 7]

[0174] ΔX2=X1sin(β xy Equation (7)

[0175] Therefore, based on Figure 7D The example coordinate system in the example can transform T 2xy Represented as:

[0176] [Mathematical Expression 8]

[0177]

[0178] Similar transformations can also be applied to other components of β: β yz and β zx The transformed component T is obtained. 2yz and T 2zx They are then used to represent transformation T2 (using equation (2)).

[0179] Figure 7EThis demonstrates an example of how a UI can adjust in response to changes in viewpoint to compensate for angular distortion (e.g., tilt) in the actual generated UI. First, as... Figure 7E As shown, example UI 701 illustrates what UI elements should visually appear to a user (e.g., by...). Figure 3 (The optimal display configuration is determined by the 330). Example UI 701 can be displayed on a flat portion of the display, for example, which could be one of the display sections that together form the foldable display discussed here. For the sake of simplicity, the following explanation will focus on angular distortion adjustment of a single section; however, the same technique can be similarly applied to multiple sections of the foldable display. Assume UI 701 is displayed on the yz plane, and the x-axis extends to the area shown. Figure 7E On the page, as shown in the figure, UI 701 includes a corner marked by a dot and should visually form a right angle (e.g., 90°). The following discussion will use this corner to explain how an example angle distortion adjustment is performed in response to a change in viewing angle β. Assume the angle formed by this corner is (γ).

[0180] Figure 7E Further examples of UI 702, 703, 704, and 705 are shown. Examples UI 702-703 illustrate the view of the display portion from an acute angle β (i.e., less than 90°), while UI 704-705 illustrate the view of the display portion from an obtuse angle β (i.e., greater than 90°). It is observed here that, without any compensation for the change in viewing angle, when viewing the display portion from an obtuse angle β, the corner angle γ becomes an acute angle (unlike the right angle in UI 701), as shown in example UI 702. Similarly, without any compensation for the change in viewing angle, when viewing the display portion from an acute angle β, the corner angle γ becomes an obtuse angle (unlike the right angle in UI 701), as shown in example UI 704.

[0181] Therefore, according to at least some embodiments (e.g., those implementing dynamic viewpoint UI adjustments, as described above), the UI can receive angle distortion adjustments (e.g., changes in tilt angles, such as at the corners of box UI elements) to account for perceived changes in tilt angles due to changes in the user's viewpoint. Viewpoint UI adjustments can compensate for angle distortions caused by changes in viewpoint, as shown in example UIs 703 and 705, for example. In other words, some embodiments disclosed herein may employ a mechanism for determining how much these tilt angles in the UI need to be adjusted based on changes in viewpoint β.

[0182] Specifically, in some examples, the corner angle γ can vary with the viewing angle β based on the hyperbolic tangent function (also known as the "tanh()" function), according to the user's perception. That is, in some embodiments, the relationship between the displayed corner angle γ and the viewing angle β can be described using a formula based on tanh(), for example:

[0183] [Mathematical Expression 9]

[0184] γ=a tanh(-bβ+c)+d Equation (9)

[0185] Figure 7F An example relationship between the amount of angular distortion adjustment and the change in viewing angle is shown, more specifically, equation (9). Figure 7F As shown, the angle on the UI can be increased or decreased by an amount inversely proportional to tanh(β). It should be noted that when the viewing angle β is 90°, the actual corner angle γ displayed on the UI is also 90°, that is, when the viewing angle β is vertical, there is no need to adjust the corner angle γ.

[0186] It should be noted that, in actual implementation, one or more parameters in equation (9) (i.e., “a”, “b”, “c”, and / or “d”) can be optimized or tuned based on the actual device design and application environment (e.g., specific display characteristics, display size, and / or sensor inputs (e.g., how far and / or where the user’s eyes are)). Figure 7F In the illustrated embodiment, a = 1.5, b = 2, c = π, and d = 1. In some implementations, parameters can be actively adjusted on a per-user basis to better suit different user habits and individual characteristics (e.g., grip angle or distance).

[0187] Furthermore, it should be noted that the tanh() function discussed above is merely an example, and in one or more embodiments, functions other than tanh() can be used to achieve similar effects. For example, in some variations, the "sigmoid" function (whose plot is similar to "tanh") can be used. Further, some examples can use straight lines, or combinations of multiple straight lines with different inclinations, to approximate the visual effect described here. It is observed that a suitable function can be characterized by plotting it as follows: (1) intersecting the origin at 90° (i.e., when β is 90°, γ is also 90°); (2) being symmetrical about the origin. Depending on the embodiment, it can also be characterized by (3) an increase in β leading to a decrease in γ, and vice versa (i.e., β and γ are negatively correlated). In some embodiments, when β is close to 0°, γ is close to 180°, and vice versa.

[0188] In the aforementioned approach, the disclosed UI manipulation technology enables the concealment or obscuring of sensitive information to be displayed on flexible, foldable, or otherwise reconfigurable displays from onlookers, while maintaining or improving the UI's accessibility to its primary users. This not only prevents user data loss and maintains UI usability and readability, but also allows for the dynamic customization of configurable devices to provide the optimal security configuration tailored to the user's current environment. Furthermore, through auxiliary UI elements, an intuitive UI system can be implemented that helps users easily achieve the best screen configuration for security.

[0189] It should be noted that, at least in some embodiments, the described UI manipulation techniques can be implemented based on one or more existing software development kits (SDKs). SDKs can help perform transformations of the UI given input parameters (e.g., size, folding angle, or viewpoint) (e.g., as described above). Generally, the type of transformation upon which the described UI techniques can be based can be referred to as perspective transformation, which can be a subcategory of SDKs related to geometric image transformations. An example of such an SDK with functionality for perspective transformation is OpenCV. TM These SDKs offer numerous functions that can accept the dimensions of the input image (e.g., the original, undistorted length of the UI edges, such as...). Figure 7B The inputs (Y0 in the diagram) describe a perspective reference (e.g., a viewpoint or object plane), and the desired output. In many cases, these functions can be used, programmed, or otherwise instructed to "straighten" images on a plane and present them on a viewer-facing plane. In some embodiments of this disclosure, the desired output may come from a "pre-distortion" graph that takes into account changes in viewpoint and / or folding angle.

[0190] Grip operation for sensitive data protection

[0191] As described above, one way to prevent nearby bystanders from seeing sensitive information is to display a visual guide marker on the user device's foldable display, prompting the user to manipulate the folding angle. The visual guide marker can be designed so that it only appears to be the selected geometry (e.g., a circle) when the foldable display is not manipulated to the optimal folding angle. When the foldable display is not manipulated to the optimal folding angle, the visual guide marker may appear to be another shape (e.g., a tilted ellipse).

[0192] Another way to prevent nearby bystanders from seeing sensitive information is to display visual guidance markers that prompt the user to adopt a grip position that blocks the view of nearby bystanders. The appropriate grip position can be determined based on the optimal arrangement of the hands and fingers to reduce the risk of data loss. The risk of data loss can be based on the angle at which identified bystanders are located around the user's device, and consequently, on which parts of the display are currently visible to those bystanders.

[0193] For example, a selected geometry (e.g., a circle) and instructions for placing a selected finger (e.g., index finger) on the selected geometry can be displayed on a foldable display of a user device. In such an embodiment, the position of the selected geometry can be based on the folding angle and the position of a nearby bystander. Although embodiments can be described in the context of user devices with foldable displays, these features are similarly applicable to user devices with flexible or otherwise reconfigurable displays. Therefore, the process described below can be employed by user devices with inflexible or non-foldable reconfigurable displays to prevent the loss of sensitive information in a seemingly natural manner.

[0194] As further discussed below, data regarding potential security and privacy risks to the user's device's surrounding environment and / or the sensitivity of information displayed on the user's device can be used to determine the optimal grip position to avoid information loss to nearby individuals. The optimal grip configuration can be used to generate one or more UI elements to encourage or assist the user in adopting that configuration. For example, data about the current grip position can be used to determine which UI elements should be generated to prompt the user to move their grip to the optimal position. The current grip position can be determined based on readings generated, for example, by touch-sensitive elements, pressure-sensitive elements, proximity sensors, ambient light sensors, etc.

[0195] Figure 8 A flowchart illustrating an example method 800 for implementing grip manipulation techniques is shown. Method 800 can be implemented by a UI manipulation engine (e.g., engine 300). Figure 3 Various components of ) (e.g., components 310-370; Figure 3 This is achieved by protecting sensitive information displayed on a user device from nearby individuals (stealing). At a higher level, method 800 can be described as an algorithmic approach used to determine hand gestures that will optimally prevent accidental disclosure of sensitive information displayed on the user device.

[0196] First, the UI manipulation engine may receive (810) an instruction regarding the display of sensitive information on the user device's display. An example of such an instruction could be a software function call. As noted above, the user device may include a foldable display, a flexible display, or other reconfigurable display. The UI manipulation engine may then determine (820) the optimal display configuration (also known as the "optimal grip position") regarding how the sensitive information is displayed. In several embodiments, the UI manipulation engine identifies (822) the sensitivity level of the information to be displayed and then determines a shape factor regarding how the sensitive information is displayed. Additionally or alternatively, the UI manipulation engine may determine (824) privacy risks in the user device's surrounding environment when determining the UI's shape factor. Generally, the higher the sensitivity level of the data to be displayed, or the higher the privacy risk in the surrounding environment, the smaller the UI's shape factor. That is, the shape factor may be negatively correlated with the risks from the surrounding environment and / or the sensitivity of the information.

[0197] Furthermore, the UI manipulation engine can determine (830) the optimal grip configuration for obscuring sensitive information. The optimal grip configuration (also known as the "optimal grip position") can be based on the ambient state discoverer (e.g., ambient state discoverer 326); Figure 3 The ambient state detector can determine the location of nearby individuals. For example, a machine vision facial recognition technology can be used to determine the number of nearby individuals, as the goal is to block their gaze from the display. If the user device's display is flexible or foldable, the UI manipulation engine can further determine the optimal grip position based on the current shape or folding angle. Therefore, the UI manipulation engine can use data about the surrounding environment and / or information sensitivity to determine the display's privacy requirements and identify the optimal grip position that obscures sensitive information displayed on it.

[0198] In some embodiments, the optimal grip position also takes into account the user's body size and capabilities (collectively, "hand attributes"). Examples of hand attributes include hand size, number and palm size (e.g., measurements in width and length) and flexibility. These hand attributes can be manually entered by the user, or they can be determined by algorithms based on past interactions with the mobile device. Alternatively, these hand attributes can be estimated for a given user based on demographic averages. This estimate can be used directly by the algorithm, or the averages can be further refined using the methods described above.

[0199] As mentioned above, the optimal grip position can be determined based on the location of nearby individuals whose view of the display will be blurred. More specifically, the UI manipulation engine can determine the optimal grip position (e.g., determined based on ambient data) to block the gaze of these nearby individuals and prevent them from seeing sensitive parts of the display (e.g., determined based on UI sensitivity data). As an example, the UI manipulation engine can determine the viewing angle of each nearby individual and then determine which viewing paths need to be blocked to protect sensitive information. The UI manipulation engine can then generate one or more grip patterns that (i) block the viewing paths and (ii) are feasible to implement given the user's hand attributes.

[0200] Next, the UI manipulation engine can generate (840) a UI to be displayed on the monitor based on the optimal display configuration and optimal grip configuration. For example, the UI manipulation engine can receive (842) data about the actual configuration of the user's device and the location of (one or more) nearby individuals, and then cause the display of (one or more) UI elements to indicate where the user should place his or her hands so that at least a portion of the monitor displaying sensitive information is blurred. Thus, depending on the nature of the monitor, the UI manipulation engine can obtain data indicating orientation, folding angle, or shape. Figures 10A-10B As shown in the example UI, one or more UI elements can be displayed to indicate where the selected finger should be placed to ensure the hand is in the optimal grip position.

[0201] Figure 9 A flowchart illustrating an example method 900 for implementing a UI-driven grip manipulation technique is shown. Method 900 can be implemented by a UI manipulation engine (e.g., engine 300). Figure 3 Various components of ) (e.g., components 310-370; Figure 3 This is achieved by protecting sensitive information displayed on user devices from being stolen by nearby individuals.

[0202] The UI manipulation engine can perform UI updates upon receiving (910) an update about the actual, current physical configuration of the user device. For example, the UI manipulation engine can detect one or more of the following new pieces of information: it can detect (912) new indications of sensitive data to be displayed; it can detect (914) new privacy risk levels in the user device’s surrounding environment; it can detect (916) new user states (e.g., new grip positions); and / or it can detect (918) new actual device configurations (e.g., new orientation, folding angle, or shape).

[0203] As an example, a UI manipulation engine can continuously monitor the grip position by examining data (referred to as "grip data") generated by one or more sensors (referred to as "grip position sensors") built into the user device. Examples of grip position sensors include touchscreen support components (e.g., touch-sensitive and pressure-sensitive elements), proximity sensors, ambient light sensors, gyroscopes, accelerometers, and the like. In some embodiments, sensing the grip position is aided by actively generating stimuli such as vibrations (e.g., generated by a motor or piezoelectric element). Alternatively or concurrently, techniques involving user-facing optical sensors (e.g., cameras) can be employed to determine the hand's position, for example, based on corneal reflection. The grip data can describe various characteristics of the grip position, including the position of the palm relative to the user device and its display, and the position of (one or more) thumbs relative to the user device and its display. Therefore, the UI manipulation engine can acquire the grip data generated by (one or more) grip position sensors and then examine the grip data to determine the optimal grip position.

[0204] Then, consistent with the above approach (e.g., regarding...) Figure 8 The UI manipulation engine can determine (920) a new optimal grip position. For example, the UI manipulation engine can determine a new optimal grip position when there is a new or updated level of sensitive information to be displayed (912) and / or when there is a new level of privacy risk in the surrounding environment (914). As an example, if the UI manipulation engine determines that the surrounding environment is becoming more crowded (thus reducing privacy), the UI manipulation engine can determine that a new optimal grip position that further obscures the sensitive information may be necessary.

[0205] Afterward, the UI manipulation engine can adjust the (930) UI according to the new optimal grip configuration. See below for details. Figure 11A-11D Further discussion suggests that adjustments may include adding UI elements that indicate where the user will place one or more selection fingers. In one or more implementations, UI adjustments are performed substantially in real time as the user manipulates the display (e.g., by changing its orientation, folding angle, or shape). Similarly, UI adjustments can be performed substantially in real time as the user moves his or her hand relative to the user device. For example, the location of sensitive information may change in response to determining that the user has moved his or her hand to interact with the display (e.g., touch). Steps 910-930 may be performed repeatedly or recursively.

[0206] Figure 10A It is for use Figure 3 Example components of the UI manipulation engine in the example demonstrate how data flows between various blocks within a UI manipulation system (e.g., engine 300). Data flow 1000 is essentially similar to... Figure 4A Data stream 400. However, here, data regarding the user's hand attributes is also obtained by the optimal display configuration discoverer 330. As mentioned above, this data can be used to determine which grip positions are actually feasible for the user.

[0207] at the same time, Figure 10B It is for use Figure 3 Another example data flow 1002 for implementing grip manipulation techniques is shown in the UI manipulation engine example components. As described above, one or more grip position sensors 910 can be responsible for generating grip data from which the position of the hand can be inferred. Examples of grip position sensors include touchscreen support components (e.g., touch-sensitive elements and pressure-sensitive elements), proximity sensors, ambient light sensors, gyroscopes, accelerometers, etc. Alternatively or additionally, techniques involving user-facing optical sensors (e.g., cameras) can be employed to determine the hand position, for example, based on corneal reflection. Using the grip data, grip position discoverer 920 can determine the user's actual grip position. In other words, grip position discoverer 920 is able to determine the actual position of the hand based on the analysis of the grip data generated by one or more grip position sensors 910. Information about the actual grip position can be provided to auxiliary UI element generator 370, which can be configured to generate one or more UI elements to implement grip manipulation techniques based on the optimal display configuration determined by optimal display configuration discoverer 330 and the actual grip position determined by grip position discoverer 920.

[0208] Figures 11A-11D An example implementation of the grip manipulation technology is shown. Initially, a UI manipulation engine (e.g., engine 300) is executed on user device 1100; Figure 3 This can establish the location of nearby individuals 1102a-c, as mentioned above. Figures 8-9 As discussed, this approach enables UI manipulation to identify visual paths that must be blocked to prevent individuals from observing the display of user device 1100. More specifically, the UI manipulation engine can calculate which directions must be blocked to ensure privacy, such as... Figure 11B As shown.

[0209] The UI manipulation engine can then identify the appropriate UI elements(s) to be displayed on the screen. For example, the UI manipulation engine can determine the location on the screen where visual guide markers should be displayed based on calculated orientation(s). Figure 11CIn this embodiment, the UI manipulation engine has enabled the presentation of a visual guide marker 1104 in the upper right corner of the display. In some embodiments, the visual guide marker 1104 is one of multiple visual guide markers displayed on the display. For example, the UI manipulation engine may determine that multiple visual guide markers are needed in a crowded environment. The multiple visual guide markers may correspond to selected fingers on the same hand or different hands.

[0210] The visual guide marker 1104 may be accompanied by an instruction to place a selected finger on the visual guide marker. When the user places the selected finger 1106 on the visual guide marker 1104, the line of sight of nearby individuals 1102a-c will be blocked (thus ensuring that sensitive information remains private). In some embodiments, the visual guide marker 1104 is designed such that it can be substantially covered by the selected finger 1106. Here, for example, the selected finger 1106 has completely covered the visual guide marker 1104, while a portion of the display remains visible.

[0211] Many user devices have been designed to allow multi-touch functionality (or simply "multi-touch"). Multi-touch enables a touch-sensitive display to recognize the simultaneous presence of more than one point of contact with the display. Therefore, when one finger is held in one area while another finger touches another area, user device 1100 may be able to detect multiple points of contact. With this in mind, in some embodiments, a portion of the touch-sensitive display may be defined outside the multi-touch target area. For example, user device 1100 may be instructed not to detect touch events occurring within a fixed area of ​​the touch-sensitive display (e.g., visual guide marker 1104 and its surroundings), or user device 1100 may be instructed not to use such touch events to determine multi-touch behavior. Therefore, the touch-sensitive display may have at least one portion in which touch events are not recognized for multi-touch purposes, and visual guide markers (or multiple visual guide markers) may be located in areas where touch events are not recognized for multi-touch purposes.

[0212] It should be noted that the visual guidance mark 1104 can also represent an instruction to place the selected finger 1106 completely outside the boundary of the touch-sensitive display. As an example, Figure 11C The visual guide mark 1104 shown can be presented as an arrow indicating that the finger 1106 should be positioned along the top or side of the user device 1100.

[0213] At a high level, the UI elements(s) generated by the UI operation engine generally serve one of two purposes. First, the user may need to interact with these UI elements(s)(s) to achieve the optimal grip position necessary to activate the desired function. Second, the user may need to interact with these UI elements(s)(s)(s)(s)) to perform a desired function. Examples of desired functions include confirming a payment, entering a password, and viewing financial or personal details. Therefore, the user may not be able to perform the desired function until the optimal grip position has been achieved.

[0214] In some embodiments, one or more UI elements represent existing elements required for a given function. For example, a UI manipulation engine may use an existing graphic labeled "Show Password" as a UI element, but may adjust its attributes (e.g., position and size) to meet the above requirements. In other embodiments, one or more UI elements represent new elements that satisfy the above requirements. For example, a UI manipulation engine may cause one or more fingerprints indicating where the selected finger(s) should be placed to cover the UI, thereby completing the given function.

[0215] Each UI element created by the UI action engine has attributes that govern its appearance, functionality, and position. Examples of such attributes include size, color, animation, and position. Similarly, these attributes can specify what action, if any, is needed to activate the corresponding UI element (e.g., press, swipe, repeated tap). In some embodiments, these attributes are influenced by the underlying characteristics of the underlying UI for which the UI elements(one or more) were created / selected. For example, the color of one or more UI elements(s) can be changed to match the color scheme of the underlying UI.

[0216] In some embodiments, aspects of the grip manipulation technique are controlled by the user of user device 1100. For example, the user may be prompted to specify the location of nearby individuals 1102a-c (e.g., by tapping the edge of the display to indicate the location). As another example, the user may be able to influence the position, number, or arrangement of visual guide markers (e.g., visual guide marker 1104). For example, while the user may initially be prompted to place a selected finger on visual guide marker 1104, the user may be able to change the position of visual guide marker 1104 by performing some action (e.g., tapping at least twice and then dragging). As described above, the user's hand attributes can be used to determine the appropriate position of the visual guide markers displayed on user device 1100. In some embodiments, the user may be prompted to complete a calibration process in which different arrangements of visual guide markers are displayed on the display of user device 1100. These arrangements may include different numbers of visual guide markers located in various positions. Based on the speed and ease with which the user can position one or more selected fingers on one or more visual guides included in each arrangement, the UI manipulation engine can learn the arrangement that best suits the user.

[0217] Computer system and device architecture

[0218] Figure 12 This is a block diagram illustrating an example of a computing system 1200, in which at least some of the operations described herein can be implemented. For example, some components of the computing system 1200 are used to implement, for example, a UI manipulation engine (e.g., Figure 3 The computing device (e.g., UI manipulation engine 300) Figure 1A and Figure 2A User equipment 130, 230).

[0219] The computing system 1200 may include one or more central processing units (also referred to as “processors”) 1202, main memory 1206, non-volatile memory 1210, network adapter 1212 (e.g., network interface), video display 1218, input / output devices 1220, control devices 1222 (e.g., keyboard and pointing devices), drive unit 1224 including storage medium 1226, and signal generation device 1230, all communicatively connected to bus 1216. Bus 1216 is shown as an abstract concept representing one or more physical buses and / or point-to-point connections connected by appropriate bridges, adapters, or controllers. Therefore, bus 1216 may include a system bus, a peripheral component interconnect (PCI) bus or a high-speed PCI (PCI-Express) bus, an HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (I2C) bus, or an IEEE standard 1394 bus (commonly referred to as “FireWire”).

[0220] The computing system 1200 may share a similar computer processor architecture with personal computers, tablet computers, mobile phones, game consoles, music players, wearable electronic devices (e.g., watches or fitness trackers), network-connected (“smart”) devices (e.g., televisions or home assistant devices), virtual / augmented reality systems (e.g., head-mounted displays), or other electronic devices capable of executing a set of instructions (which specify one or more actions to be taken by the computing system 1200) (sequentially or otherwise).

[0221] Although main memory 1206, non-volatile memory 1210, and storage medium 1226 (also referred to as "machine-readable medium") are shown as a single medium, the terms "machine-readable storage medium" and "storage medium" should be understood to include a single medium or multiple media (e.g., a centralized / distributed database and / or associated caches and servers) storing one or more sets of instructions 1228. The terms "machine-readable medium" and "storage medium" should also be understood to include any medium capable of storing, encoding, or carrying a set of instructions for execution by computing system 1200.

[0222] Generally, routines executed to implement embodiments of the present disclosure may be implemented as part of an operating system or a particular application, component, program, object, module, or sequence of instructions (collectively, a “computer program”). A computer program typically includes one or more instructions (e.g., instructions 1204, 1208, 1228) that are disposed at different times in different memories and storage devices in a computing device. When read and executed by one or more processors 1202, the instructions(s) cause the computing system 1200 to perform operations to execute elements relating to various aspects of the present disclosure.

[0223] Furthermore, although embodiments have been described in the context of a full-featured computing device, those skilled in the art will understand that various embodiments can be distributed as a program product in various forms. This disclosure applies regardless of the specific type of machine or computer-readable medium used for the actual implementation of the distribution.

[0224] Further examples of machine-readable storage media, machine-readable media, or computer-readable media include recordable media such as volatile and non-volatile memory devices 1210, floppy disks and other removable disks, hard disk drives, optical disks (e.g., optical disc read-only memories (CD-ROMs), digital multifunction optical discs (DVDs)), and also transmission media such as digital and analog communication links.

[0225] Network adapter 1212 enables computing system 1200 to mediate data in network 1214 with entities outside computing system 1200 via any communication protocol supported by computing system 1200 and external entities. Network adapter 1212 may include network adapter cards, wireless network interface cards, routers, access points, wireless routers, switches, multilayer switches, protocol converters, gateways, bridges, bridging routers, hubs, digital media receivers, and / or repeaters.

[0226] Network adapter 1212 may include a firewall that manages and / or administers permissions for access to / proxy data within the computer network and tracks different trust levels between different machines and / or applications. The firewall may be any number of modules with any combination of hardware and / or software components capable of enforcing a predetermined set of access rights between a specific group of machines and applications, and between machines and / or applications (e.g., monitoring traffic and resource sharing between these entities). The firewall may additionally manage and / or access access control lists that detail permissions, including the access and operational rights of individuals, machines, and / or applications to objects, and the conditions under which those rights are established.

[0227] The techniques described herein can be implemented through programmable circuits (e.g., one or more microprocessors), software and / or firmware, dedicated hardwired (i.e., non-programmable) circuits, or a combination of these forms. Dedicated circuits can take the form of one or more application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or field-programmable gate arrays (FPGAs).

[0228] Remark

[0229] The above description of various embodiments of the claimed subject matter has been provided for illustrative and descriptive purposes. It is not intended to be exhaustive and is not intended to limit the claimed subject matter to the precise forms disclosed. Many modifications and variations will be apparent to those skilled in the art. These embodiments were chosen and described to best illustrate the principles of the invention and its practical application, thereby enabling those skilled in the art to understand the claimed subject matter, the various embodiments, and the various modifications suitable for the particular intended use.

[0230] While the detailed description depicts certain embodiments and contemplated best modes, the technology can be practiced in a variety of ways, however specific the detailed description may be. Embodiments may vary considerably in their implementation details, but are still covered by the specification. Specific terms used in describing certain features or aspects of the various embodiments should not be construed as implying that such terms are redefined herein as limited to any particular characteristic, feature, or aspect of the technology associated with that term. Generally, the terms used in the following claims should not be construed as limiting the technology to the specific embodiments disclosed herein, unless those terms are expressly defined herein. Therefore, the actual scope of the technology includes not only the disclosed embodiments but also all equivalent ways of practicing or implementing the embodiments.

[0231] The language used in this specification has been chosen primarily for readability and pedagogical purposes. It may not have been chosen to describe or limit the subject matter. Therefore, it is intended that the scope of this technology not be limited by this specific embodiment, but rather by any claims made based on the application herein. Thus, the disclosure of various embodiments is intended to illustrate, and not limit, the scope of this technology as set forth in the appended claims.

[0232] Industrial applicability

[0233] This disclosure can be applied to data security.

Claims

1. A method for privately displaying information, the method comprising: Receive an instruction to display sensitive information on a mobile device, wherein the mobile device includes a foldable display; Identify the sensitivity level of the sensitive information; Based on readings from one or more sensors on the mobile device, identify the level of privacy risk in the environment surrounding the mobile device; A shape factor is determined regarding how the sensitive information will be displayed, wherein the shape factor is determined based on the sensitivity level and the privacy risk level in a manner smaller than the entire area of ​​the foldable display; and A user interface to be displayed on the foldable display is generated based on a determined shape factor, wherein the user interface displays the sensitive information according to the shape factor. The user interface is generated based on a selected optimal folding angle, regardless of the actual folding angle of the foldable display, so that the user interface is visually undistorted to the user only when the foldable display is manipulated to the selected optimal folding angle, and visually distorted to the user when the foldable display is not manipulated to the selected optimal folding angle.

2. The method as described in claim 1, wherein, The user interface is displayed by multiple parts of the foldable display.

3. The method as described in claim 2, wherein, The foldable display is a bi-foldable display.

4. The method of claim 1, wherein, in, The shape factor is negatively correlated with the sensitivity level.

5. The method of claim 1, wherein, The shape factor is negatively correlated with the identified privacy risk level.

6. The method of claim 1, further comprising: The optimal folding angle is determined such that, if the display portion of the mobile device is not folded to the optimal folding angle, the user interface appears distorted to the user.

7. The method of claim 6, wherein, The user interface deforms symmetrically about the hinge of the foldable display.

8. The method of claim 1, further comprising: Determine the viewing angle; if the mobile device is not oriented relative to the user's viewing angle, the user interface will appear distorted to the user. The user interface is also generated based on the determined perspective.

9. The method of claim 1, further comprising: The position of the user's eyes is detected from the user-facing eye detector on the mobile device; and The user interface is adjusted in response to the detected position of the user's eyes.

10. The method of claim 9, wherein, The user interface is adjusted so that the new perspective for the user interface reflects the detected position of the user's eyes.

11. The method of claim 9, further comprising: Receive the current orientation state of the mobile device from one or more orientation sensors on the mobile device. The user interface is also adjusted in response to the received current orientation state of the mobile device.

12. The method of claim 1, further comprising: Based on the selected optimal folding angle, a visual guidance mark is generated to guide the user to manipulate the foldable display to the selected optimal folding angle.

13. The method of claim 12, wherein, The visual guide markers are presented to the user as the selected geometry only when the foldable display is manipulated to the selected optimal folding angle.

14. The method of claim 1, further comprising: Obtain the current folding angle of the foldable display; and Based on the difference between the current folding angle and the selected optimal folding angle, additional guidance is generated to instruct the user to manipulate the foldable display to the selected optimal folding angle.

15. The method of claim 1, wherein, The sensitive information is not displayed in the user interface if the foldable display is not manipulated to the selected optimal folding angle.

16. The method of claim 1, further comprising: Based on the determined shape factor, the original layout of the sensitive information is changed to an alternative layout, which is adapted to a shape factor smaller than the entire area of ​​the foldable display.

17. The method of claim 16, wherein, The original layout is used to specify the display shape factor for a desktop computer, and the alternative layout is used to specify the display shape factor for a mobile device.

18. The method of claim 1, further comprising: Obtain the current folding angle of the foldable display; Update the shape factor based on the current folding angle; and The user interface was adjusted to conform to the updated shape factor.

19. The method of claim 18, wherein, The acquisition, update, and adjustment steps are performed substantially in real time as the user manipulates the foldable display.

20. A method for privately displaying information, the method comprising: Receive an instruction to display sensitive information on a mobile device, wherein the mobile device includes a flexible display; Identify the sensitivity level of the sensitive information; Based on readings from one or more sensors on the mobile device, identify the level of privacy risk in the environment surrounding the mobile device; A shape factor is determined regarding how the sensitive information will be displayed, wherein the shape factor is determined based on the sensitivity level and privacy risk level in a manner smaller than the entire area of ​​the flexible display; and A user interface to be displayed on the flexible display is generated based on a determined shape factor, wherein the user interface displays the sensitive information according to the shape factor. The user interface is generated based on a selected optimal folding angle, without considering the actual folding angle of the flexible display, so that the user interface is visually undistorted to the user only when the flexible display is manipulated to the selected optimal folding angle, and visually distorted to the user when the flexible display is not manipulated to the selected optimal folding angle.

21. The method of claim 20, further comprising: in, The shape factor is negatively correlated with the sensitivity level.

22. The method of claim 20, further comprising: Determine the optimal folding angle and folding line so that the user interface appears distorted to the user if the display portion of the mobile device is not folded along the folding line to the optimal folding angle.

23. The method of claim 20, further comprising: The position of the user's eyes is detected from the user-facing eye detector on the mobile device; and The user interface is adjusted in response to the detected position of the user's eyes.

24. The method of claim 20, further comprising: Visual guidance markers are generated based on the selected shape, which guide the user to manipulate the flexible display to the selected shape.

25. The method of claim 20, further comprising: Obtain the current shape of the flexible display; and Based on the difference between the current shape and the selected shape, additional guidance is generated to instruct the user to manipulate the flexible display to the selected shape.

26. The method of claim 20, further comprising: Obtain the current shape of the flexible display; Update the shape factor based on the current shape; and The user interface was adjusted to conform to the updated shape factor.

27. The method of claim 20, further comprising: Based on the selected shape, a visual guidance mark is generated, which guides the user to hold the flexible display in a manner that partially obscures the flexible display.

Images