User interface operation in a foldable screen device

Abstract

To provide user interface (UI) manipulation technology that enables a user device to hide from bystanders, or make obscure, confidential information displayed on a display which is reconfigurable by flexible, foldable, or other methods on one hand, and enables maintaining or improving exclusive accessibility to main users on the other hand.SOLUTION: User interface (UI) manipulation techniques are disclosed that can allow a user device to hide and obscure sensitive information displayed on a flexible, foldable, or otherwise reconfigurable display from onlookers whilst maintaining or improving its accessibility exclusively to the primary user. Examples of the disclosed techniques can manipulate the UI in a way such that it is largely only viewable as intended when the user device is physically configured to a certain folding angle (and / or viewing angle). In some examples, the UI can be customized such that it promotes a certain device configuration (e.g., folding angle) that can provide an optimum security configuration for the user's current surroundings. The UI displayed on the display screen can change in a form factor (e.g., an intended size as viewed by the user) to adapt to the current surroundings and / or on the basis of the sensitivity in the displayed content.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims priority to U.S. Provisional Application No. 63 / 008,473, filed Apr. 10, 2020, entitled “User Interface Manipulation in a Foldable Screen Device,” and U.S. Provisional Application No. 62 / 991,553, filed Mar. 18, 2020, entitled “Foldable Phone UI Manipulation for Privacy”. Each of these applications is hereby incorporated by reference in its entirety.

[0002] (Technical Field) The present disclosure generally relates to data security, and more particularly to protecting confidential information displayed on a mobile end - user device from onlookers attempting to steal such information. )

Background Art

[0003] With the widespread use of computers and portable electronic devices, the preferred mode of information presentation has long since shifted from paper to electronic. Generally, such electronic devices include a display screen (e.g., a liquid crystal display (LCD) screen) for presenting visual information to a human user. In many cases, for example, when involved in finance or commercial transactions, confidential information such as a social security number or a bank account number may be displayed on the display screen. This state of the art creates a vulnerability where an unethical onlooker can view the user's information from the side of the device and steal confidential personal and financial information from the user by taking mental or actual photographs.

[0004] ​​​​​​​​​​ [Brief explanation of the drawing]

[0005] One or more embodiments of this disclosure are shown, by example, in the drawings of the accompanying drawings, and among them Therefore, similar references indicate similar elements.

[0006] [Figure 1A] Figure 1A shows an example environment in which a user interface (UI) for a foldable phone, as described here, may be implemented.

[0007] [Figure 1B] Figure 1B shows the folding angle of an example of a foldable telephone.

[0008] [Figure 1C] Figure 1C shows the viewing angle of an example of a foldable phone for a human user.

[0009] [Figure 1D] Figure 1D shows a table illustrating different configurations of the disclosed UI operation technology.

[0010] [Figure 2A] Figure 2A illustrates how a foldable phone UI like the one described here can be implemented for different privacy scenarios.

[0011] [Figure 2B] Figure 2B shows an example of visual guidance indicators and text guidance displayed in the actual UI, compared to the UI as seen from the user's perspective.

[0012] [Figure 2C] Figure 2C shows additional implementation details regarding some examples of the UI operation techniques for the foldable phones described.

[0013] [Figure 3]FIG. 3 shows a functional block diagram of an example UI operation engine that can be used to implement the UI operation technology of a foldable telephone.

[0014] [Figure 4A] FIG. 4A shows an example data flow diagram for implementing the UI operation technology of a foldable telephone with some exemplary components of the UI operation engine of FIG. 3.

[0015] [Figure 4B] FIG. 4B shows another example data flow diagram for implementing the UI operation technology of a foldable telephone with some exemplary components of the UI operation engine of FIG. 3.

[0016] [Figure 4C] FIG. 4C shows yet another example data flow diagram for implementing the UI operation technology of a foldable telephone with some exemplary components of the UI operation engine of FIG. 3.

[0017] [Figure 5] FIG. 5 shows a flowchart illustrating an example method for implementing the UI operation technology of a foldable telephone.

[0018] [Figure 6] FIG. 6 shows a flowchart illustrating another example method for implementing the UI operation technology of a foldable telephone.

[0019] [Figure 7A] FIG. 7A shows a more detailed example regarding a method by which a distorted UI can be generated in some embodiments of the UI operation technology of a foldable telephone.

[0020] [Figure 7B] FIG. 7B shows additional assumptions of the calculations in the example of FIG. 7A.

[0021] [Figure 7C]Figure 7C shows an example of a transformation calculation by changing the folding angle in the example shown in Figure 7A.

[0022] [Figure 7D] Figure 7D shows an example of a conversion calculation due to a change in the field of view in the example shown in Figure 7A.

[0023] [Figure 7E] Figure 7E shows an example of how the UI can adjust in response to changes in the viewing angle to compensate for angular distortion (e.g., how it is tilted) in the actual, generated UI.

[0024] [Figure 7F] Figure 7F shows an example relationship between the amount of angular distortion adjustment and the change in field of view.

[0025] [Figure 8] Figure 8 shows a flowchart illustrating an example of a method for implementing grip operation technology.

[0026] [Figure 9] Figure 9 shows a flowchart illustrating an example of a method for implementing UI-driven grip operation technology.

[0027] [Figure 10A] Figure 10A shows an example data flow for implementing grip manipulation technology using some of the illustrative components of the UI manipulation engine in Figure 3.

[0028] [Figure 10B] Figure 10B is another example data flow for implementing grip manipulation technology using some of the illustrative components of the UI manipulation engine in Figure 3.

[0029] [Figure 11A] Figure 11A illustrates how an individual in close proximity to a user device may be able to view sensitive information displayed by the user device.

[0030] [Figure 11B] Figure 11B illustrates how a UI operation engine may calculate which directions should be obscured to ensure privacy, based on the location of nearby individuals.

[0031] [Figure 11C] Figure 11C illustrates how the UI operation engine may generate the display of visual guidance indicators to assist the user in positioning their hand in the optimal grip position.

[0032] [Figure 11D] Figure 11D illustrates how sensitive information displayed by a user device can be concealed from nearby individuals when the hand is positioned in the optimal grip position.

[0033] [Figure 12] Figure 12 shows a high-level block diagram illustrating an example of a mobile system in which at least some of the operations related to the technologies described herein may be implemented. [Modes for carrying out the invention]

[0034] References to “one embodiment,” “one embodiment,” etc., in this specification refer to the specific embodiment described. The features, functions, structures, or characteristics of are included in at least one embodiment of this disclosure. This means that the occurrence of such phrases in this specification does not necessarily mean that all embodiments are the same. This does not necessarily mean that the embodiments mentioned are mutually exclusive.

[0035] As mentioned above, finance and commercial transactions are moving away from paper and towards computers and mobile phones. This is increasingly being done on other portable electronic devices. Generally, such electronic devices A display screen is a screen that presents visual information to a human user (for example, a liquid crystal display (LCD)). It is equipped with a screen. Often, for example, when it is related to finance or commercial transactions, social security Confidential information, such as phone numbers or bank account numbers, may be displayed on the screen. (Smartphone) As smartphones gain an increasingly larger share of the payment market, more and more people will be using them regularly. People use their phones for banking applications. In public places where security is low, there is a risk of attempting to input and read private data. No. The state of this technology is that it is not possible to physically observe the user's interaction with the user's device. It is possible, and by viewing user information from the side of the device, and the impression or reality By taking photos of users, malicious actors can steal sensitive personal and financial information from them. This creates vulnerabilities for individuals and agents with malicious intent.

[0036] On the other hand, over the past decade, there have been advancements, primarily in organic light-emitting diode (OLED) display technology. This allows for flexible displays (for example, using fabric-based substrates) Significant progress has been made in the fabrication of I. These flexible display devices are To accommodate application needs and variable physical configurations, a variable screen size It enables different configurations within a single device, such as flexible displays. As gameplay is improved, the user interface and interaction modes will be changed. A variety of new physical configurations become possible. UI for foldable (foldable) mobile phones. Elements (for example, screens) can be quickly manipulated into different shapes, or It can be changed to different folding angles. In addition to the flexible display, it can also be used with conventional displays. Even the game itself is becoming widespread, and manufacturing costs continue to fall. It has a rigid display, but not two or more. Devices with multiple screens (for example, flip phones with two screens) also appeared on the market. It is.

[0037] Therefore, what is presented here is that user devices are only suitable for primary users. While maintaining or improving accessibility, from a bystander's perspective, flexible, foldable To conceal sensitive information displayed on a display that can be reconfigured in other ways, and to make it undetectable. It is a user interface (UI) operation technology that makes things clear. More specifically, it is disclosed One or more embodiments of the technology are such that the user device is physically configured to a certain folding angle. In this case, the UI can be manipulated so that it can only be viewed in the way primarily intended. In some examples, the UI provides the optimal security configuration for the user's current surroundings. It can be customized to prompt specific device configurations (e.g., folding angle). The UI displayed on the screen is adapted to the current environment (for example, the user's environment) (When moving from one boundary to another, or when the environment around the user changes) form factor The size (for example, the intended size as seen by the user) can be changed. In addition, or alternatively, Alternatively, the displayed UI will be based on the level of secrecy of the displayed content. The element factor can be changed.

[0038] The UI operation ensures that the technologies presented here are easy to use and read for the user. The ability to hide or conceal information displayed on the device in an intuitive way that is maintained. This enables users to easily achieve a screen configuration that is optimal for security. Furthermore, an effective user interface that helps prevent unintended data loss for bystanders. The face system can be implemented.

[0039] The following explanation uses folding as an example only, to illustrate various aspects of the technology. Examples of foldable mobile devices are used. However, the technology presented here is foldable Applicability to mobile phones or any other specific types of devices is not limited. Please note the following. Also, the following explanation describes various aspects of the disclosed technology on a mobile phone. The focus is on implementation, but other electronic devices or systems (e.g., wrap) A top or tablet can also be adapted to this technology in a similar manner. The terms "display" or "foldable screen" are not more clearly defined by the context. As far as is concerned, any display whose display area is collapsible and is used interchangeably here This refers to a variation of which should include at least two variations: (1) a foldable structure A flexible screen has been created (for example, one flexible OLE A display having a D display screen, and (2) at least two cells A display having two sections, each of which has a screen. However, the two sections are designed to function together (or collectively) as one screen. It is constructed such that two sections are mechanically joined to each other via a folding mechanism. Spray. Some examples of foldable screens include bi-fold displays and tri-fold displays. Includes displays, etc.

[0040] In this specification, the term "user" generally refers to a person who physically operates a mobile device. The term "bystander" generally refers to someone who is physically near the user but is not using a mobile device. This refers to someone who is not physically manipulating the UI. Also, the UI manipulation techniques introduced here are (for example) A 1900x1200 pixel display can simply be reduced to a 1680x1050 pixel display. By using fewer pixels to display, so that it can be scaled down to fit on a screen, This is distinct from techniques that simply use an area smaller than the entire area to display information. As we will consider, UI manipulation techniques require specific skills before the user can see the UI correctly. To make it intuitive for the user to position the phone in relation to the physical device configuration (e.g., folding angle) Note that the visual transformations required are used. Thus, in this disclosure Furthermore, the term "form factor" in the context of displays generally refers to a display. (i) is directed not from a third party (for example, a bystander), but from the user (i.e., from the user's perspective). It refers to how large or small something should appear visually when viewed from a specific point. In that sense, for the purposes of this specification, unless otherwise specifically made clear in the context of the specification, the term "Being seen" or "being perceived" is not from another party, but from the user of the device in question. It is used in relation to the perspective of the -za.

[0041] The drawings presented here are merely examples to facilitate the examination of the disclosed UI operation techniques. Please note that these are not drawn to scale. [Application Environment]

[0042] Figure 1A shows the user interface of a foldable telephone like the one described here. The environment 100 that can be applied is shown. The environment 100 is illustrated with a large number of bystanders 140. In the presence of a foldable screen 120 on a foldable mobile device 130 User 110 is interacting with the other party. Because Bystander 140 is present in environment 100, mobile Displayed on the user interface of the device 130 (i.e., foldable in the conventional way) (Displayed directly on the screen 120) User information may be stolen by bystanders 140. There is a high possibility that it will not work. As shown in Figure 1A, the mobile device 130 is the user's Where there is a risk to privacy, displaying users' private and confidential information Therefore, to protect user 110's information from being stolen by bystander 140. It is necessary.

[0043] Therefore, as will be discussed in more detail below, the UI operation techniques disclosed herein are The device is folded by the user (for example, at a certain folding angle, and / or In some embodiments, the selected device configuration (viewed from a certain viewing angle) is configured / The UI can be manipulated in a way that is only effectively visible to the user when it has been reconfigured. In other words, the disclosed UI system is designed so that the user device can be folded at a specific angle. When physically configured to the field of view, the resulting UI is for user 110 From a viewpoint, to be visually visible at a size and / or angle that enhances security, U I can be manipulated.

[0044] In one or more examples, the disclosed UI operating system is a bystander around the user. By determining the possibility and potential number of such entities, the privacy of the user's surrounding environment can be assessed. The level of risk can be determined. In addition, or alternatively, the UI system can determine the user's mobile Determine the level of confidentiality of the private and confidential information presented on the device 130. Yes, it is possible. Next, in some embodiments, the level of privacy risk in the user's surrounding environment Private and confidential information presented on the user's and / or mobile device 130 Based on the level of confidentiality of the information, the UI system ensures that only user 110 can access confidential information. How confidential information can be viewed from a foldable screen 1 The form factor for displaying on 20 can be determined. In these embodiments, The foldable screen 120 has a selected folding angle and / or a selected viewing angle. Unless manipulated, confidential information could remain distorted even to user 110. In other words, in some examples, the UI is optimal for the user's current environment. Specific device configurations that can provide security configurations (e.g., folding angle and / or viewing angle) It can be customized to encourage (Nogaku). [Folding angle]

[0045] Figure 1B shows the folding angle 13 of a foldable phone (e.g., user device 130). This shows 8. As shown in Figure 1B, the folding angle 138 is the multi-section folding. At an angle between sections of a mobile device (e.g., user device 130) Yes. More specifically, the folding angle 138 is the same as the two axes of the foldable phone 130. A first axis 132 extending along the surface of the first section (for example, the left section), and and on the surface of the second section (for example, the right section) of the foldable phone 130 The angle between the second axis 134 extending along it is formed. The first and second sections are mechanically connected and fold along the hinge axis 136. As shown in Figure 1B, the UI on device 130 is a foldable display. The left and right sections of the ray are displayed collectively (together).

[0046] To simplify the discussion, the foldable display embodiment described here is lateral While the focus is on a folding configuration (for example, from left to right), the technology disclosed is for the vertical direction. For example, this can be similarly applied to those embodiments having a configuration that folds from top to bottom. Please note that in some of these embodiments, the configuration may be, for example, manual or automatic drawing The screen rotation function allows the configuration to better suit the user's preferences and the current device orientation. It may be possible to make it replaceable.

[0047] According to this embodiment, the folding angle 138 can be selected, and the UI is The foldable screen of the foldable phone 130 (for example, The Clean 120 folds to the selected folding angle (or desired folding angle). When operated, the user interface is readable by user 110, but the surrounding area It is distorted for the remaining people (e.g., bystander 130). The selected folding angle is, This can also be described as the optimal inner angle at which the display screen 120 folds. For example, if the display is a foldable display (e.g., single hinge), The folding angle is the inner angle at which the display folds.

[0048] In some embodiments, the desired folding angle is displayed in the user interface. Based on the degree of confidentiality of private information and / or the presence of potential bystanders around the user. This may be decided to block the user's view of private information on the screen. For example, If a user wants to access private or secure data, the user's surroundings Phone camera data may be used to determine the presence of any potential bystanders.

[0049] Once the optimal folding angle of the phone is determined, the user interface will show that it is the optimal folding angle. It can be generated / distorted / customized so that it can be read from any angle. To make the interface readable at the user's position, the phone is folded to some extent, and hidden The perspective of an outsider prevents them from surveying the user's personal information. For example, a user can see the bus I want to see my bank account information while I'm on the train. Here's a foldable phone... The conversation was about installed software applications (for example, a bank's proprietary application). When a user accesses the system, the phone displays a distorted user interface. It is presented to the user. The user can visually read the user interface from the user's perspective. Make the phone narrower until it looks like (for example, the edges and fonts don't appear slanted / distorted). Close it to an angle, and in the meantime (by closing the phone to an even narrower angle), you can access the content. Prevent potential bystanders from around the area. [Viewing angle]

[0050] Figure 1C shows an example of a foldable telephone (e.g., user device 130) For example, it shows a field of view of 148 for user 110. Generally, the field of view is for user 110. This is the orientation angle of the mobile device 130 relative to the foldable device. The viewing angle is also the angle of the foldable device This can also be understood as the position of the user's eyes 110 relative to the vice 130. (See Figure 1C) As shown, in the example of a flip phone, the viewing angle 148 is the non-folding device 130 Between the first axis 142, which is perpendicular to the edge (i.e., 90°), and the line of sight of the user 110 The angle can be such that the folding device in Figure 1C is located in the center (for example, hinge axis 1). It can be folded around the hinge axis 146 (same configuration as 36).

[0051] As shown in Figure 1C, for convenience, the disclosure here is the point of focus (i.e., the user's eye) Please note that we are assuming the point of focus is the center of the screen. , define the starting edge where a field of view of 148 can be measured. This definition of the field of view and / or gaze The assumptions about the point may change depending on the implementation (for example, the point of focus may be at the top or bottom of the screen). (In some implementations, the point of focus may be dynamically determined by the sensor.) The UI manipulation techniques disclosed herein relate to the definition of the field of view and / or changes in the position of the point of focus. It is not applicable in any other way. [Dynamic UI adjustments in contrast to static ones]

[0052] The UI generated by the UI manipulation techniques here is generally a foldable screen. It is folded to the intended folding angle, and the screen is viewed from the intended viewing angle. Only when the user (i.e., the user) It has the effect of being able to see (from a particular viewpoint). Thus, depending on the embodiment, The device may have the option to dynamically or statically adjust UI distortion. For the purposes of this discussion, the term "static UI operation" refers to the optimal folding angle and / or to view from the optimal viewing angle, the UI (for example, as explained with respect to Figure 2A below) It is only manipulated / bent (by a certain method), and the UI rendering is the actual folding of the device. In the sense that it does not change based on the viewing angle or the actual position of the user's eyes / point of gaze. It means "static". In contrast, the term "dynamic UI operation" refers to UI rendering. Dulling is the actual folding angle of the device and / or the actual position / point of gaze of the user's eyes. This means that it changes and is adjusted based on its position.

[0053] More specifically, there are four different UI adjustment configurations, as shown in Table 150 in Figure 1D: (A) Both folding angle UI adjustment and viewing angle UI adjustment are static, (B) Static folding (C) Dynamic folding angle UI adjustment and static viewing angle UI adjustment The field of view UI adjustment, and (D) both the folding angle UI adjustment and the field of view UI adjustment are dynamic. There are four possible combinations. For embodiments that perform static folding angle adjustment, The resulting UI distortion does not need to take into account the actual folding angle of the device, rather The UI is generated based on the selected optimal folding angle, in order to see the correct UI. For example, reconfiguring the device to the optimal folding angle (by folding the screen) It depends on the user. Similarly, for embodiments that perform static field angle adjustment, the generated UI distortion does not require considering the user's eye position / point of focus; rather, the UI is about selection. It is generated based on the optimal field of view, so that the user can see the correct UI. Adjust the device to a holding position / orientation / etc. so that the device can be seen, and / or It is up to the user to direct their attention to the device.

[0054] In contrast, in embodiments that perform dynamic folding angle adjustment, UI distortion is, It is dynamically generated and adjusted by taking into account the actual folding angle of the device, i.e. The UI can deform as the actual folding angle changes. Similarly, dynamic viewing angle adjustment is possible. In the implemented embodiment, the generated UI distortion takes into account the user's eye position / point of gaze. In other words, the UI is dynamically generated and adjusted based on the user's actual eye position / point of gaze. It is done. With dynamic viewing angle adjustment, the user does not need to adjust the holding position / orientation / etc. Generally, the correct UI can be viewed from any available field of view for the user.

[0055] For the sake of simplification, unless otherwise clarified by the context, the following explanations of UI manipulation techniques are omitted. This configuration employs static folding angle UI adjustment and dynamic viewing angle UI adjustment, i.e. We will focus primarily on configuration (B). Here, configuration (B) is, firstly, a static folding angle The degree forces the user to change the folding angle, thus protecting confidential data from bystanders. Secondly, it is easy to change the UI distortion based on where the user's eyes are positioned. The dynamic field of view provides convenience and improves usability for the user. Even without these two advantages, it is recognized that it can be preferred in many field applications. In comparison, configuration (A) has extra sensors such as a user eye-tracking system. The system may be expensive and / or consume too much power, low cost This may be preferable in the implementation of low power. On the other hand, configuration (D) is user Regardless of the current folding angle or gaze point, the UI is visible to the user. As such, it provides the most seamless user experience relatively well, however, the UI looks If ease of use is the same for the user, the user will make some adjustments (for example, folding the phone more narrowly). Because you won't want to fold it, it is safer than a configuration with a static folding angle. It will likely decrease.

[0056] Although configuration (B) is preferred, the disclosed technology is based on configurations (A), (C), and The same applies to (D). Specifically, the determination and generation of the optimal folding angle will be considered. In the following explanation, (for example, for the implementation of those involving static field of view adjustments) Determining and generating an appropriate field of view requires the same or similar considerations (e.g., the inside of the user device and / (or information relating to external information) can be applied. In some embodiments, the user can apply different configurations. You may be given options to choose from among the results.

[0057] Furthermore, in some embodiments, UI distortion is used to determine the level of confidentiality of the user's personal information. Updates based on the determined level and / or level of privacy risk. It should be noted that this is possible. These "updates" of UI distortions are also, in a sense, generally "moving" It can be understood as "target," but it is a display date for secrecy or surrounding privacy risks. Because this type of examination is performed in response to a change in either or both of the following, in context The user's point of focus and / or folding angle (i.e., the user's control) is clearly defined. This should not be confused with a general consideration of dynamic UI operations that respond to parameters within the range of a troll. stomach. [User interface operations for protecting sensitive data]

[0058] Figure 2A shows the user interface of a foldable telephone like the one described here. Examples of how this can be implemented for privacy and confidential data protection are shown in Figure 2A. The diagram shows a bifold display 220 that folds around a hinge axis 236. This is a foldable mobile phone 230. Also shown in Figure 2A are four different examples. Explicit privacy scenarios - (1) private space, (2) relatively safe space, (3 ) public but quiet spaces, and (4) crowded public spaces, for different folding purposes Four sets of combinations of how the foldable mobile phone 230 can be operated. (i.e., the first set is illustrative UI201-202, and the second set is illustrative U) UI201-208 are examples showing I203-204, etc. So, privacy scenarios and their corresponding retention angles are merely for consideration. Depending on the implementation, more or fewer scenarios can be adopted, and different A holding angle may be adopted.

[0059] The example UIs 201, 203, 205, and 207 are for different privacy scenarios. The UI shown is what the user (for example, user 110) sees, and the example UIs 202 and 204 are shown. 206 and 208 show the actual UI displayed on the foldable screen 220. Specifically, the actual UI representations in the examples UI202, 204, 206, and 208. This means that the user's private / confidential information is stored on a foldable display (for example, a foldable display). How is it laid out on the user interface of a telephone with a display (220)? To show what you can get. In other words, the "actual UI" representation is that the foldable phone 230 is each How the user interface looks to the user when fully opened in Nario This accurately represents the UI as seen by the user. Ray 220 was folded at the optimal folding angle for each scenario (and so on). In some embodiments, when viewed from the optimal viewing angle, the user's private / confidential information How information is visually presented to the user on the user interface of the phone 230. This indicates that.

[0060] Let's take the first privacy scenario—private space—as an example. (Example UI20) 2 shows the foldable mobile phone 230 with its double-folded display 220 fully open. This represents the actual UI rendering of the user's private / confidential information on phone 230. For comparison, the UI visible to the user, example UI201, is a foldable display 22 The two sections of 0 (i.e., the left and right sections) are, that is, fully open, When the entire area of ​​the foldable display 220 is occupied, the actual UI 202 is used This shows how it looks visually to the 220-inch foldable display. Because it fully utilizes the display area, users typically use their private space, such as their home. It is understood that you would prefer to keep the foldable mobile phone 230 fully open. When fully open, the actual UI (i.e., example UI202) is the most suitable for the user. Note that this is the same UI as what the user sees at the appropriate viewing angle (i.e., example UI201). I want to be treated that way.

[0061] The second example privacy scenario—a small office room—is generally classified as relatively secure. It is a place where something could happen, but it is not as safe as the user's home, so bystanders looking at the user's phone. Since there's still a slight possibility, wouldn't it be wise for users to keep phone 230 fully open? It might be. In such cases, even if you fold the phone 230 slightly, keep it at a large angle. However, this is sufficient to protect users' private / confidential information from being stolen by bystanders. The example UI203 is for a scenario where the user is located in a relatively safe space. Therefore, the actual UI rendering of the user's private / confidential information on phone 230 is shown. In such an environment, the two sections of the foldable display 220 are large. When folded to a certain angle (for example, 135° or more), the optimal view for the user is achieved. It is possible. For comparison, the UI that the user sees, example UI203, is a bifold display. Two sections of I-220 concern the most relatively (but not completely) safe space. How does the actual UI204 look to the user when folded to the appropriate folding angle? This indicates that. Also, here, UI203 as seen from the user's perspective is a foldable mobile phone 2 How does the actual UI204 look to the user when viewed from the optimal viewing angle? Please also note that this represents (see the above explanation regarding Figure 1C-1D). User From this perspective, UI203 is at the optimal folding angle in a relatively safe space, (for example, U (Compared to the I201) It appears smaller than the entire display area, but the user does not perceive it as distorted. It is a user interface that does not exist. On the other hand, the same user interface appears distorted to an observer. It's visible (for example, UI204).

[0062] Similarly, a third privacy scenario is a park, a generally public space, but still quiet. In places that can be classified as private spaces, especially when users are in a safer environment such as an office, Compared to the second scenario where the phone was positioned, the user folded the phone 230 to a narrower angle. It might be wise to fold it up. This is compared to safer places like the office. This is because in a park, there is a high probability that bystanders will be watching the user's phone. In such cases, Folding a mobile phone at a narrow angle exposes the user's private / confidential information. It can be protected from being stolen by others. Example UI206 shows that the user is in a public place but still For scenarios where the user is located in a quiet space, the user's private information on phone 230 This is the actual UI rendering of confidential information. In such an environment, a folding display... The two sections of the Ray 220 are folded at a narrow angle (for example, 135°-75°). This allows for the achievement of the optimal view for the user. From the user's perspective, UI205 is collapsible. The 230 mobile phone, which can be used in public places, has an optimal folding angle for quiet spaces. This shows how the actual UI206 looks to the user when it is collapsed. In this way, the display area perceived by the user is (compared to, for example, UI201) Even if it becomes smaller than the entire area of ​​the foldable display 220, optimal folding In terms of angle, UI205 as seen by the user remains distortion-free for the user. However, However, the same user interface, due to its narrower folding angle, is more convenient for bystanders. In contrast, it becomes more hidden, and compared to the second scenario, it is even more distorted to bystanders. It looks like it's made of something.

[0063] Finally, the fourth privacy scenario is a generally crowded public space such as a bus or subway. In places that can be classified as intermediate, the user, compared to the previous scenario (e.g., a park), It might be wise to fold the phone 230 to an even narrower angle. This is in comparison to the park. In contrast, on a bus, bystanders are more likely to see the user's phone. In this case, folding the phone 230 at an even narrower angle would make it less likely to be stolen by a bystander. It can protect users' private / confidential information. Example UI208 shows the user in a crowded environment. Private / confidential information of phone users for scenarios when in public spaces This is the actual UI rendering. In this environment, the foldable display 220 is 2 One section has the narrowest angle (e.g., 75° or less) in all four scenarios. When collapsed, the optimal view for the user can be achieved. UI from the user's perspective. 207 is the optimal folding angle for the foldable mobile phone 230 in crowded public spaces. This shows how the actual UI208 looks to the user when it is folded. As shown in Figure 2A, the display area perceived by the user is (UI201, 203, and 2 (Compared to the 05) Despite being the smallest of all four, it has an optimal folding angle. Therefore, UI207, from the user's perspective, can remain distortion-free for the user. However, compared to the first three scenarios, the example UI208 is the narrowest collapse. Due to the angle, it is most hidden from bystanders, and compared to all previous scenarios, it is the most It appears distorted to the bystander. The foldable display is in the center or near the center. In embodiments configured to be folded (e.g., a bifold display configuration) This technology is implemented in user interfaces (e.g., UI204, 206, and 20 8) The hinge of a foldable display (e.g., display 220) (e.g., Note that the deformation may be symmetrical with respect to the axis along axis 236).

[0064] In this way, the disclosed UI manipulation / distortion techniques affect the content of the user interface. The 'tsu' is visually distorted in the actual UI representation, and the display (multiple options are possible) (for example) The display (220) is readable without distortion only when folded at a specific angle. It provides an intuitive user interface system that can be seen as such. bystanders as an actual UI representation (e.g., example UI204, 206, and 208) You can only see distorted user interface displays like the examples shown. Therefore, the user's privacy is protected from theft by bystanders (for example, bystander 140). This also enables the protection of confidential information. Furthermore, it improves the UI representation from the user's perspective (for example, the example U Readable UI as shown in examples I203, 205, and 207 When a user is looking at it, the user folds the display to a specific folding angle. By closing the screen and blocking the view of bystanders, bystanders are prevented from accessing the user's private and confidential information. This further hinders the theft of information.

[0065] Figure 2B shows the visual guidance displayed in the actual UI, in contrast to the UI as seen from the user's perspective. Examples of indicator 250 and text guidance 252 are shown. As discussed here, The UI displayed on a foldable or flexible display is selected for folding. It can be generated based on the angle, and as a result the UI is the screen of a foldable phone. It is only visually visible to the user as usual when folded to the selected folding angle. The UI generated from this technology is, at the very least, a phone with a selected configuration (for example, selected If it is not folded to the selected folding angle, otherwise the content is abnormal. For example, it may appear distorted or slanted, making it intuitive for the user.

[0066] Nevertheless, further enhancing the ability for users to intuitively configure and / or direct their phones To enable and reduce reliance on distortions in the displayed content to guide the user. To reduce this, the generated UI is designed so that the user can select a foldable display. Visual guidance indicators that guide you to the folding angle (for example, guidance indicator 25) 0) can be additionally included. In some implementations, the visual guidance index 250 is Display at the top of the display, or in a location that does not interfere with the user's normal content within the UI. The visual guidance indicators 250 may be selected from circles, squares, triangles, rhombuses, etc. It can be a geometric shape. With the implementation of the disclosed UI operation technology, the visual guidance indicators are The selected option is only available when the foldable display is operated to the optimal folding angle. Geometric shapes (for example, a circle as shown in the "UI as seen from the user's perspective" representation in Figure 2B) This will be visible to the user. For example, if the selected geometric shape is a circle, then folding When the display is not operated to the optimal folding angle, the actual UI will be a shape other than a circle. The user perceives it as a shape (for example, a tilted ellipse as shown in the "Actual UI" representation in Figure 2B). Yes, it is possible. In other words, the geometric shapes on an intuitive user interface are collapsible. The display appears as a circle only when it is operated to the optimal folding angle. In that embodiment, the actual UI shows the user what the selected geometric shape is. Additional hints on the display (which are part of the text guidance 252 described below) This may include (which is also possible). Next, the user narrows the phone until the ellipse looks like a circle. It can be adjusted to any angle. The user interface is easy to read from the user's perspective, and visually appealing. While it appears normal to the user, bystanders around the user would otherwise see that the content is blocked. It is either distorted or visually altered.

[0067] In addition, or alternatively, the generated UI is the grip considered with respect to Figure 8-11D. A visual guide that guides the user in manipulating their grip according to operating techniques. It can include an induction index. Therefore, in one or more embodiments, it is generated The UI includes visual guidance indicators for UI operation techniques and visuals for grip operation techniques. Guidance indicators may be included. However, visual guidance indicators are independent of each other. It may be generated by... Visual guidance indicators for grip operation techniques are UI operation techniques Independently (that is, without visual distortion), or in relation to UI manipulation techniques (i.e., It may be generated (with visual distortion).

[0068] In addition, or as an alternative, one or more embodiments of the UI operation system described herein are current Text guide to instruct the user to change the folding angle to the optimal folding angle. Dances (e.g., text guidance 252) may be included in the user interface. For example, text guidance 252 says, "Open the foldable display further." Alternatively, the user can be instructed with phrases like, "Close the foldable display further." In some variations, the text guidance might say, for example, "Call until the ellipse becomes a perfect circle." The instruction is to "close it" in a currently readable font (i.e., in a non-distorted way). It may be displayed as follows.

[0069] In this embodiment, the visual guidance 250 and / or text guidance 252 are By blocking the view of potential bystanders from the user's personal information, the user interface is viewed from the user's location. To make the phone face readable, the user is instructed to fold the phone just the right amount. It can be guided. As a practical example, a user might access their bank account information while on a bus. I want to see it. When a user accesses that information on a foldable phone, the phone appears distorted. The UI is presented to the user. There is an oval at the top, and currently it is written in an easy-to-read font. The instructions are: "Until the black ellipse becomes a perfect circle (and the UI appears to be in a straight line and horizontal)," "Fold your phone to a narrower folding angle." The user is told that the ellipse is Close the phone to a narrower angle relative to the phone until it appears as a circle. After adjustment, the user interface... The face became fully readable from the user's position, but the content on it remained hidden. Bystanders who remain passive are now blocked.

[0070] Furthermore, in some examples, the UI operating system displays sensitive information to the user on the UI. Before that can happen, it is possible to request that the user device be placed in an optimal configuration. For example, UI The optimal operating system is one that displays only auxiliary UI elements and has a foldable display. The user's configuration (e.g., optimal folding angle) will remain unchanged unless it is operated. Actively block, blur, mask, and hide sensitive data displayed on the Vice screen. If not, it may be configured to hinder or simply prohibit. For example, the UI may guide the user Follow the dance instructions and reconfigure the foldable phone to the selected optimal folding angle. Or until you operate it, guidance information (e.g., visual indicators 250 and / or text) will be displayed. Only guidance 252) can be displayed to the user. Only then can the UI operating system be displayed. Display sensitive information on the UI (for example, using the method described above).

[0071] Figure 2C shows additional arbitrary examples of UI operation techniques for foldable phones. Details of the setup are shown. At least several of the UI operation techniques introduced are considered here. That example is the display area of ​​a foldable display as perceived by the user (for example, Reduce or shrink UI201 (compared to UI203, 205, and 207). Because it has a visual effect, in some embodiments the content of the display is folded Based on the determined valid form factor for available displays, it will automatically adjust It can be rearranged. Specifically, in various embodiments, the original layout of the confidential information is determined It can be changed or rearranged to an alternative layout according to the specified form factor. The alternative layout is smaller compared to the entire area of ​​the foldable display. Suitable for non-factory applications. For example, the original layout (for example, it's desktop / tab). (May be designed for larger displays such as red displays) Or alternative layouts designed for mobile devices with smaller screens (e.g., It allows for different font sizes, graphics, and / or navigation placement. It may be changed to (it may have).

[0072] Figure 2C shows two illustrative UIs, UI270 and UI272, where the illustrative UI I270 is the original layout of the content that should be displayed when the screen is fully open. This shows that when a device implementing UI operation technology is folded, the displayed layout is , it can be automatically changed to the alternative layout shown by example UI272. Additionally, Alternatively, additional UI elements such as scrollbar 274 could be added to the displayed UI. It can be added to navigate content in a simulated, reduced screen size. Support users by providing assistance. [Example system components and data flow]

[0073] Figure 3 shows the UI operation technology used to implement the foldable phone described here. A functional block diagram of an example UI operation engine 300 is shown.

[0074] One example UI operation engine 300 is a user's mobile device (for example, a mobile device). It can be implemented on and run on (130,230). As shown in Figure 3, UI The operation engine 300 includes a UI confidentiality extraction system 310 and a surrounding state evaluation system 320. Optimal display configuration viewfinder 330, actual display configuration viewfinder 340, user This includes a state evaluation system 350, a UI distortion generation unit 360, and an auxiliary UI element generation unit 370. This is possible. The components shown in Figure 3 are merely examples, and are not representative of any well-known specific configuration. Elements, for example, central processing unit (CPU), memory module, screen display ( Note that, for the sake of simplicity, the (multiple) and communication circuits are not shown here. The one or more systems, subsystems, components, and / or modules described herein are software It may be implemented as a wearable application or may include a software application. Please note that these systems / subsystems / modules are obtained. Any suitable combination of hardware or files can achieve the same or similar functionality. It can be implemented as a firmware component. Similarly, the component shown as an example in Figure 3 is this This is shown for the purpose of facilitating consideration in the specification, and more or fewer components Elements may be used, and the components may be combined or separated depending on the actual implementation. good.

[0075] According to some implementations, the UI operation engine 300 is (for example, on a mobile device) We received instructions that confidential information would be displayed on a foldable display. Then it can start working. For example, Application Programming Interface (AP) I) Engine 300 receives instructions (for example, from a software application) Implemented for engine 300 so that it can receive software function calls. It is possible. In a modified example, the engine 300 is provided by the operating system (OS). It may be implemented as part of a set of functions, or in a system software library, etc. In additional or alternative embodiments, the engine 300 is a standalone that resides in memory. It can be software that intercepts / detects confidential information that is displayed.

[0076] After receiving instructions that there is confidential information to be displayed, the UI operation engine 300 , determine the optimal display configuration for that, and in order to do so, engine 300 This involves considering one or more factors, including those based on external and / or internal information. More specifically This is how small the perceived display screen is, according to the UI operation techniques introduced. Information that can influence the decision of what it should be (i.e., its form factor) is generally There are two categories: (1) information inside the phone, and (2) information outside the phone. Report. The former (i.e., internal information) is extracted by the UI secrecy extraction system 310. This can be done, and the latter (i.e., external information) is extracted by the surrounding state evaluation system 320. It is possible. [A.UI Confidentiality Extraction System]

[0077] Figure 4A shows a foldable phone with some illustrative components of the UI operation engine from Figure 3. This is an example of data flow 400 for implementing UI operation technology for the machine. Data flow 40 0 indicates that data is being collected between various blocks within the UI operating system (e.g., engine 300). An example of how it can flow is shown. Refer to Figures 3 and 4A simultaneously, and the data Flow 400 is described here along with the functional block diagram of the UI operation engine 300. .

[0078] The UI confidentiality extraction system 310 is responsible for the privacy of data displayed on the user device. It can determine the level of secrecy and output UI secrecy data. In other words, it is a UI secrecy extraction system. The TEM310 can determine the level of confidentiality of information presented on the user's mobile device. Cut.

[0079] In some examples, UI sensitivity data is a representation of the different features currently displayed on the screen. Various levels of secrecy can be described. Examples of high-secrecy items include password input boxes and banking information. This may include account information, credit card information, etc. Depending on the implementation, the level of confidentiality in the intermediate steps may be... This may include personal information such as membership number, date of birth, home address, or telephone number. The secrecy extraction system 310 uses public domain information, general website text, Alternatively, it can also identify items with lower levels of secrecy, such as images. In many embodiments, the user can identify this These items may have the option to be adjusted or edited to the user's own preferences. In addition, or alternatively, UI sensitivity data is matched to the most sensitive feature on the page. It is possible to describe an overall assessment of the screen's level of privacy.

[0080] According to some embodiments, the UI confidentiality extraction system 310 uses local resources and UI sensitivity data can be obtained from and / or remote resources. Examples of these resources are , secrets about commonly found software applications / components / features This may include a lookup table or database where degree ratings can be recorded. For example, If the software mobile application is a banking application, the application case When the selection window displays a password entry screen, certain functions (for example, bank transaction details) may be displayed. If a webpage providing detailed information is triggered, or if a UI element is password If it includes input fields, the resource(s) may be identified as highly sensitive. In one or an alternative embodiment, the UI secrecy extraction system 310 extracts metadata, for example, transmission Message metadata about who the believers are, metadata about the nature of the message , the urgency of the message, and / or the recipient's address (work or private email). UI sensitivity data can be obtained through (such as the distinction of sensitivity based on). Furthermore, the API In some variations implemented for the UI operation engine 300, the UI secrecy extraction system Tem310 is accessed from the API (for example, via one or more parameters of a function call) via the UI It can receive confidentiality data.

[0081] In one or more embodiments, the UI confidentiality extraction system 310 through content analysis UI secrecy data can be obtained. In a specific example, the UI secrecy extraction system 310 sends Keywords in a sent or received message (e.g., "highly confidential") Based on the existence of (fidential), or (for example, known image processing and recognition algorithms) By analyzing graphic / photo content using rhythm, the secret It is possible to determine the degree level. Several practical examples of the UI secrecy extraction system 310 The implementation method uses machine learning techniques (e.g., supervised machine learning) to improve UI confidentiality. It can also generate data. For example, the UI secrecy extraction system 310 can generate UI secrecy To establish a decision data model, the training data is labeled with its level of secrecy. The set can be used for initial training. An example of training data is shown in the actual context. The data (e.g., text, documents, images, or videos) and the corresponding level of confidentiality assessment. It may include the following: In this way, the UI secrecy extraction system 310 is on the fly It can generate an assessment of the confidentiality level of sensitive UI content.

[0082] Furthermore, several variations of the UI secrecy extraction system 310 are derived from changes in the user's mood. UI secrecy data obtained through user sentiment analysis, such as by inferring the likelihood of secrecy. It can generate data. For example, when a user interacts with an application, it can make the user more secretive. If the user acts according to their principles and changes their facial expression, mood, etc., the UI confidentiality extraction system 310 This infers that the application has a higher level of secrecy than typical applications. These inferential judgments of secrecy levels can be made, for example, in gesture recognition and facial recognition. , and / or based on electroencephalography. [B. Surrounding Condition Evaluation System]

[0083] The surrounding environment evaluation system 320 determines and analyzes the privacy status of the user's surroundings. System hardware, firmware, and / or software that may be used for this purpose. It includes a set of wear components. As shown in Figure 3, the peripheral state evaluation system 320 is Peripheral state sensor 322, peripheral state data preprocessor 324, and peripheral state fine It includes three example components, Da326.

[0084] More specifically, the peripheral state evaluation system 320, for example, uses the peripheral state sensor 322 to... Based on the readings, the level of privacy risks in the surrounding environment of mobile devices It can be used to identify Bell. This identified privacy risk level is optimal. It can be used by the display configuration finder 320 and the UI distortion generation unit 360. The optimal format for displaying a user's private / confidential information on their mobile device. Determine the element factor. In some embodiments, the elements displayed on the mobile device The form factor of the user's private / confidential information is identified as a privacy list. It can be inversely correlated with the level of the identified privacy risk. The higher the bell value, the smaller the UI form factor will appear to the user. For example... As explained in Figure 2A above, there are four levels of privacy risk - privacy A quiet space, a relatively safe space, a public but still quiet space, and a crowded public space - This allows the UI to be perceived by the user to become smaller, especially in crowded surroundings. It is likely.

[0085] The surrounding state sensor 324 collects information about the user's surroundings (i.e., "surrounding data"). It may include one or more sensors that can be used to acquire data. In various examples, surrounding Edge data includes visual images or records of the user's location, specific geographic location information (e.g., global (Global Positioning System (GPS) coordinates), audio information (e.g., background sound data and / or audio data from the surroundings), device connection information (e.g., connection data, etc., user information) Information for detecting the devices of people in the vicinity, and / or the presence of people around the user. This may include other appropriate data that can be used to determine the state. Applicable peripheral data To obtain the data, an example of a peripheral state sensor 324 is a rear and / or front camera, or Other light-based imaging (e.g., infrared) that can generate an optical feed from the user's position. Technology, GPS sensor capable of generating satellite position information, and audio signals collected from the user's location. A microphone or voice sensor that can be generated, and / or the user's mobile device. Signing devices within the vicinity (for example, 10 meters based on the transceiver's range) Wireless network transceivers that can be generated (e.g., Bluetooth®) It may include sensors such as the following. Depending on the embodiment, these sensors may be used by the user It may be integrated into a mobile device, or it may be connected to a device separately.

[0086] Specifically, according to one or more embodiments, the peripheral state sensor 322 then receives peripheral data The peripheral state data can be sent to the peripheral state data preprocessor 324 for subsequent analysis. The data can be preprocessed (for example, to improve the signal-to-noise ratio (SNR) to an appropriate quality). Furthermore, the peripheral state data preprocessor 324 processes the data according to the type of data collected. Various types of preprocessing can be applied to the surrounding data. For example, several implementations In this state, the peripheral state data preprocessor 324 uses the optical sensor in the peripheral state sensor 322 ( For example, optical data generated from a camera is preprocessed to determine the scene's illumination, focus, vibration, etc. Motion artifacts, reflections, and other features may be corrected. Furthermore, these actual In many implementations, the peripheral state data preprocessor 324 uses machine vision technology. Then, identify key features such as people or faces, and (for example, metadata or tags) By adding this information, optical data can be preprocessed for labeling.

[0087] Furthermore, some implementations of the peripheral state data preprocessor 324 have background noise (for example) For example, it removes wind or road noise, amplifies specific sound characteristics (e.g., voice), and reduces the signal-to-noise ratio (SNR). Audio data can be preprocessed to further improve audio quality by increasing R. Similar to the video preprocessing embodiments described above, in certain embodiments, peripheral state data preprocessing Ssa324 includes metadata or key features such as clear speech or heard phrases and words. The audio data may be preprocessed in order to add tags.

[0088] Furthermore, in certain embodiments, the peripheral state data preprocessor 324 provides connection data The data is preprocessed to identify, count, and potentially identify unique device signatures in the user's vicinity. The location can be determined. For example, the peripheral state data preprocessor 324 can determine the location of a unique device. To identify, count, and / or locate signatures and / or geographical locations Information contained in Wi-Fi management frames, requests, beacons, and pings. It can be used. The peripheral state data preprocessor 324, in some examples, GPS Preprocessing satellite data to reveal specific qualitative characteristics of the user's current location (e.g., store name). It is also possible to identify road names or landmarks using these GPS / satellite / location data. The data is considered in combination with existing phone map services (e.g., Google Maps). This is possible. Depending on the implementation, preprocessing of location data may involve (for example, filtering out features). Machine learning techniques can be used to analyze and classify data.

[0089] Next, the peripheral state finder 326 is generated by the peripheral state data preprocessor 324. By analyzing the pre-processed surrounding data, the current state of the user's surroundings (i.e., This allows for the evaluation of the "surrounding conditions." Subsequently, the surrounding conditions finder 326 evaluates the user's surroundings. Based on the edge state, the level of privacy risk around the user can be determined. In the above implementation, when determining the surrounding state, the surrounding state finder 326 can extract surrounding data or related information.

[0090] In particular, in one or more embodiments, the surrounding data can include the number of other people nearby (e.g., bystanders). In some of these examples, when determining the number of other people nearby, the surrounding state finder 326 can perform a count of the different people's feature tags added to the camera feed during preprocessing by the surrounding state data preprocessor 324. In other embodiments, the surrounding state finder 326 can determine the number of other people nearby by using machine vision face recognition technology to count the number of nearby users. In a variant, the number of other people nearby can be determined by counting the number of nearby devices detected via Bluetooth technology (or other suitable short-range wireless connection technology). Additionally, or alternatively, the surrounding state finder 326 can determine the number of bystanders by counting the number of distinct human voices identified during preprocessing of the surrounding data. A particular variant of the embodiment can provide a simple estimate of the total acoustic level in the vicinity based on changes in the audio peak amplitude over a given time interval in the surrounding data. In some of these examples, when determining the number of other people nearby, the surrounding state finder 326 can perform a count of the different people's feature tags added to the camera feed during preprocessing by the surrounding state data preprocessor 324. In particular, in one or more embodiments, the surrounding data can include the number of other people nearby (e.g., bystanders). In some of these examples, when determining the number of other people nearby, the surrounding state finder 326 can perform a count of the different people's feature tags added to the camera feed during preprocessing by the surrounding state data preprocessor 324. In other embodiments, the surrounding state finder 326 can determine the number of other people nearby by using machine vision face recognition technology to count the number of nearby users. In a variant, the number of other people nearby can be determined by counting the number of nearby devices detected via Bluetooth technology (or other suitable short-range wireless connection technology). Additionally, or alternatively, the surrounding state finder 326 can determine the number of bystanders by counting the number of distinct human voices identified during preprocessing of the surrounding data. A particular variant of the embodiment can provide a simple estimate of the total acoustic level in the vicinity based on changes in the audio peak amplitude over a given time interval in the surrounding data. In other embodiments, the surrounding state finder 326 can determine the number of other people nearby by using machine vision face recognition technology to count the number of nearby users. In a variant, the number of other people nearby can be determined by counting the number of nearby devices detected via Bluetooth technology (or other suitable short-range wireless connection technology). Additionally, or alternatively, the surrounding state finder 326 can determine the number of bystanders by counting the number of distinct human voices identified during preprocessing of the surrounding data. A particular variant of the embodiment can provide a simple estimate of the total acoustic level in the vicinity based on changes in the audio peak amplitude over a given time interval in the surrounding data. In a variant, the number of other people nearby can be determined by counting the number of nearby devices detected via Bluetooth technology (or other suitable short-range wireless connection technology). Additionally, or alternatively, the surrounding state finder 326 can determine the number of bystanders by counting the number of distinct human voices identified during preprocessing of the surrounding data. A particular variant of the embodiment can provide a simple estimate of the total acoustic level in the vicinity based on changes in the audio peak amplitude over a given time interval in the surrounding data. Additionally, or alternatively, the surrounding state finder 326 can determine the number of bystanders by counting the number of distinct human voices identified during preprocessing of the surrounding data. A particular variant of the embodiment can provide a simple estimate of the total acoustic level in the vicinity based on changes in the audio peak amplitude over a given time interval in the surrounding data. In a particular variant of the embodiment, based on changes in the audio peak amplitude over a given time interval in the surrounding data, a simple estimate of the total acoustic level in the vicinity can be provided. A particular variant of the embodiment can provide a simple estimate of the total acoustic level in the vicinity based on changes in the audio peak amplitude over a given time interval in the surrounding data.

[0091] Furthermore, in some embodiments, the surrounding data can include the spatial distribution and line-of-sight directions of nearby people. In some of these examples, the surrounding state finder 326 can, for example, contrast the situation where someone is standing to the left or right of the user but probably facing away from the user device (which can be a lesser security threat) with the situation where someone is standing behind the user and facing the user device (which can be a security threat). In some of these examples, the surrounding state finder 326 can, for example, contrast the situation where someone is standing to the left or right of the user but probably facing away from the user device (which can be a lesser security threat) with the situation where someone is standing behind the user and facing the user device (which can be a security threat). In some of these examples, the surrounding state finder 326 can, for example, contrast the situation where someone is standing to the left or right of the user but probably facing away from the user device (which can be a lesser security threat) with the situation where someone is standing behind the user and facing the user device (which can be a security threat). In some of these examples, the surrounding state finder 326 can, for example, contrast the situation where someone is standing to the left or right of the user but probably facing away from the user device (which can be a lesser security threat) with the situation where someone is standing behind the user and facing the user device (which can be a security threat). To identify , this information can be detected using camera data and machine vision techniques. In many implementations, the ambient data may include predicting how crowded / public the user's location is based on known characteristics related to the user's location, e.g., a predictive assessment of the user's GPS coordinates. For example, the ambient state finder 326 can detect from the user device's GPS data that the user is in a public place, and thereby infer that there is a high likelihood that other people are nearby. In another example, the ambient state finder 3 26 can discover from the GPS data that the user is in a field in a large state park, and thereby infer that there is a low likelihood that other people are nearby. In some embodiments, the ambient state finder 326 can further associate known properties of the user's location from the user device's network connection. For example, when the user device connects to "Home Wi-Fi" or "Car Wi-Fi", the ambient state finder 326 may determine a high privacy level (i.e., a low risk level), whereas, in contrast, when the user device connects to "Public Wi-Fi" or "Airport Free Wi-Fi", the ambient state finder 326 may determine a low privacy level (i.e., a high risk level). Based on the ambient state data and through the exemplary methods described above, the ambient state finder 3 26 can generate an ambient state output indicating the degree to which the user's data may be exposed when using the user device. In other words, from the ambient state finder 326

[0092] Based on the ambient state data and through the exemplary methods described above, the ambient state finder 3 26 can generate an ambient state output indicating the degree to which the user's data may be exposed when using the user device. In other words, from the ambient state finder 326 ​The surrounding state output indicates the level of privacy risk in the user's surroundings to the UI operation engine. This can be transmitted to other components of the 300. In some embodiments, it can be transmitted to subsequent system configurations. The transmission of surrounding conditions to the source is, among other information, a large amount of information that provides users with data exposure risks. This could be a specific description of the person, including their position, gaze angle, and distance from the user. In a modified example, the transmission of peripheral states to subsequent system components carries user risk (for example). This may take the form of a general-purpose metric relating to "data exposure risk" and peripheral state files. The 326 determines the possibility that user data may be exposed at the current location. Weighting can be applied to the characteristics of the analyzed peripheral data. Several additional or alternative examples So, the surrounding conditions finder 326 evaluates the privacy risks in the user's surroundings or You can generate bells directly. [C. Optimal Display Configuration Finder]

[0093] The optimal display configuration of the Finder 330 prevents observation by other people nearby. The optimal physical configuration for a foldable or flexible screen (i.e., "optimal display") The "play configuration" can be determined. Specifically, the Optimal Display Configuration Finder 330 allows you to determine the optimal display configuration. (1) The degree of confidentiality of private and sensitive information presented on the user's mobile device Using the level and / or (2) the level of privacy risk of the user's surrounding environment The probability that screen content will be seen by unscrupulous bystanders will be reduced and minimized, and It identifies physical configurations that may even be removed. In other words, it identifies the optimal display configuration. Data 330 is generated by the UI secrecy data (for example, by the UI secrecy extraction system 310 described above). (and / or the surrounding conditions (for example, by the surrounding condition evaluation system 320 described above) (The generated) can be used to prevent the content on the screen from being stolen by bystanders. Determine the optimal privacy configuration for the display (e.g., folding angle).

[0094] Specifically, in some embodiments, the optimal display configuration finder 330 is , information inside the user device, i.e., generated by the UI confidentiality extraction system 310 Based on the determined level of confidentiality of the user's private information, the optimal folding angle The degree can be determined. In some other embodiments, the optimal display configuration finder 330 This is information from outside the user device, i.e., generated by the peripheral state evaluation system 320. The optimal folding angle can be determined based on the determined level of privacy risk. Additionally, or alternatively, the optimal display configuration finder 330 allows the user to plan their plan. Determined levels of confidentiality and privacy risks for privatization information. Based on both, the optimal folding angle can be determined. Those perform static viewing angle adjustments. In this embodiment, the optimal display configuration finder 330 further determines the optimal viewing angle. can.

[0095] Next, several embodiments of the optimal display configuration finder 330 are optimally folded The angle, that is, the optimal internal angle at which the display folds, can be determined (Figure 1B and (See the related explanations above.) The user device's display is a single joint (for example) In those embodiments, including a single-hinge display or multiple displays, The optimal display configuration can be the optimal folding angle at which the display is folded. For example, an acute folding angle may provide higher display privacy (at the expense of a decrease in the perceived size of the display), and is optimal for high-privacy risk areas and / or high-secrecy content. An obtuse folding angle may provide lower display privacy while providing a larger perceived size of the display, and is optimal for low-privacy risk areas and / or low-secrecy content.

[0096] The optimal display configuration can vary depending on the physical characteristics of the user device. For example, in some embodiments, the device may be a device having two or more folding sections designed in a specific predetermined folding configuration. In such a device, the folding angle and rotation direction of the screen may be limited. Therefore, the optimal display configuration may be a combination of adding a vector of a specific rotation direction to the folding angle for each folding section.

[0097] ​This combination of folding angle and folding line is, for example, in the auxiliary UI element generation unit 370 (later (as described above) can be used, and the display fabric can be folded into the selected shape Guide the user to do so. In this context, many examples of collapsible lines are collapsible ( For example, a foldable display can be designed to have the same hinge axis as described here. can.

[0097] In some cases, the accelerometer / gyroscope sensor built into the user device Using this data, we can identify how the user is currently holding the device, and Through this, the optimal display configuration of the Finder 330 is achieved, and the optimal folding angle is achieved for the device. It is only possible to determine whether or not this can be physically achieved by folding one or both sides. Please be mindful of this.

[0098] After the optimal display configuration is determined, it is the optimal display configuration viewfinder 330. This can be output by: an embodiment of pure static UI operation (i.e., the UI is actually collapsible) (Things that deform / do not change with changes in angle or the user's eye position), for example, the above For configuration (A), the optimal folding angle and / or viewing angle is determined by the UI rendering. The output can be sent to the UI distortion generation unit 360 for display purposes. [D. Actual Display Configuration Finder]

[0099] Figure 4B is collapsible using some example components of the UI operation engine from Figure 3. This is an example data flow 402 for implementing UI operation techniques for a mobile phone. Row 402 is between various blocks within the UI operating system (e.g., Engine 300). This shows an example of how data can flow. Figures 3 and 4B are shown simultaneously. With reference to the data flow 402, the functional block diagram of the UI operation engine 300 and This will be explained next.

[0100] As previously mentioned, many embodiments of the disclosed UI manipulation technology are dynamic UI manipulation, That is, when the user's actual focus changes (for example, in configuration (B)), or When the actual folding angle of the user device changes (for example, in configuration (C)), Or both (for example, in configuration (D)) where the UI distortion changes or deforms. These embodiments rely on one or more of the following as the basis for these dynamic UI adjustments. System components (e.g., actual display configuration finder 340, or user state) The evaluation system (350) is described below.

[0101] The actual display configuration finder 340 shows the current configuration of the user device (or, "actual The actual display configuration can be determined. For example, the actual display configuration viewfinder 340 This refers to device data obtained from one or more display configuration sensors built into the device. These display configuration sensors can be used to determine the actual display configuration. Examples of information obtained include the current folding angle and / or folding direction configuration, and the device. This can include position and / or device orientation (e.g., tilt angle). In that example, the device orientation is determined by the accelerometer, gyroscope, or user device. This data can be obtained from other appropriate sensors installed on the vehicle. How the user is currently holding the device and / or how to achieve the optimal viewing angle How should users move their mobile devices to perform this action (for example, adjusting the field of view)? It can be used to identify the embodiment that is the target.

[0102] In some embodiments, a real display configuration file is used to collect device data. The display configuration sensors used by the 340 may include mechanical sensors. For example, a flexible display with two panels (e.g., two sections or folded in half) In gameplay, mechanical sensors may be incorporated into the hinges that join the two panels. In a typical example, the sensor is made of a flexible electronic material whose resistance changes as the hinge opens and closes. It can include, and as a result, it can measure the folding angle of the screen. Angle data shows how the foldable screen is currently folded, and / Alternatively, how the user can adjust the foldable screen to achieve the optimal folding angle It can be used to determine whether it should be reconfigured (for example, if the folding angle adjustment is static) (Regarding the embodiment described).

[0103] Generally speaking, flexible displays (for example, fabric displays) Some embodiments of actual display configurations describe surface distortion across the entire surface. To achieve this, the distortion of the display at multiple points across the surface of the display is sufficiently high. It may contain at a high density. For example, in many of these embodiments, the actual display configuration is For each point on the surface, the data represents a vector field having scalar quantities of the amplitude and angle of the distortion. It can include a tag.

[0104] Furthermore, some variations of the display configuration sensor may also include one or more cameras. Yes, it is possible. For example, in a typical embodiment of a flexible display, a camera (it is a digital display) Collected from (which can be built into the device or obtained from a third-party device) The optical data can be used by the actual display configuration viewfinder 340, and Create a normal map or vector field of the existing device surface. Some additional or alternative In an alternative embodiment, the camera(s) may also be a real display configuration viewfinder 340 It can be used to determine the relative orientation of a device and the spatial position of a device. It is estimated that. Furthermore, in some embodiments, the actual display configuration finder 340 is It can utilize camera data and image processing technology to capture the actual display configuration in 3D, This can result in 3D model files, such as STL files.

[0105] In various implementations, the display configuration sensor detects when light moves to an object, and the object The distance to the object is calculated using the time it takes for the object to reflect back from the target. It may include a time-of-flight (ToF) sensor for measurement. For example, a built-in ToF sensor is By performing a measurement of the distance between the device and a person or other physical features, the device To obtain the position, the actual display configuration finder 340 may be used. Additionally Alternatively, the display configuration sensor may include radar. For example, radar Or other objects or motion sensing technologies between display elements, or D To determine the spatial relationship between the display elements and the user, the actual display configuration file It can be used with the Nda340. [E. User State Evaluation System]

[0106] The user state evaluation system 350 is used to determine and analyze the user's physical state. It can be a combination of system components. The user state evaluation system 350 is This includes exemplary components such as a user state sensor 352 and a user state finder 354. It is possible.

[0107] The user state sensor 352 obtains data related to the user's state (or "user state"). It includes one or more sensors that may be used by the user state evaluation system 350 for this purpose. Specifically, an example of a user state that can be obtained by the user state sensor 352 is when the user The user's point of focus is the position on the screen where the user is looking, relative to the display. The angle of the user's head, the head angle, and the field of view, which is the angle of the user's eyes relative to the display. and / or, the orientation of the user's entire body relative to the display, including the orientation of the head. Yes, it is possible. Depending on the implementation, the user state sensor 352 may be a dedicated eye-tracking sensor system. This can be done, or it can be done using a combination of sensors installed in the user device. This is possible. Exemplary components of the user state sensor 352 include, for example, one or more front Camera, motion sensor (this is either on the user device or in a smart band) (Can be installed by the user as shown), suitable projector, and / or flight time It may include sensors. For example, a projector may have a specific light or light pattern (e.g., It can project infrared or near-infrared light (or similar) into the user's eyes, and the camera can project the pattern onto the user's eyes. It can capture images, and then image processing and machine vision technology are applied to the user's eye position. It may also be applied to determine the point of fixation. In another example, a time-of-flight sensor is used to screen It can detect the user's distance from the device. [F.UI Distortion Generation Unit]

[0108] The UI distortion generation unit 360 generates visual distortions in the user interface (here, "U It generates UI distortion (also called UI deformation or UI manipulation). From the user's current viewpoint in the optimal display configuration (e.g., optimal folding angle) To obtain the optimal view of the user interface, the visibility of the displayed UI is manipulated. To make the display visible from the viewpoint of any other bystander (closed folded angle) (due to) it decreases and (due to visual distortion) becomes unclear, and bystanders may miss it. This reduces the likelihood of stealing users' personal information from their devices.

[0109] Depending on the implementation, UI distortion can include different combinations of transformations and distortions. . For example, UI distortion is a series of transformations applied to the user interface (e.g., translation). It may include rotation or tilt. According to many embodiments, the UI distortion generation unit 36 0 means that the UI and the target information on the UI can only be read from a desired angle and / or distance. "Optical illusions" that can be expressed in Noh theater can be used in user interfaces (for example, those considered in Figure 2A). Applicable to certain implementations having "holographic display" hardware. Regarding the state, the UI distortion generation unit 360 determines, for example, which angle the pixel light of different pixels is at. It can generate a set of instructions that include how to adjust, especially for privacy-sensitive UI elements. It can only be seen from a specific angle and / or physical device configuration. (Referring to Figure 2A) As mentioned above, the type and degree of distortion depend on the level of confidentiality of the information being displayed. Consider this. Transformations or distortions applied to highly sensitive UI content may affect the UI content. To make it far less readable for anyone other than the target user (for example, UI example) Compared to example 203, UI example 207 can be more extreme.

[0110] In a specific example, when the UI distortion generation unit 360 determines and generates UI distortion, the user state data Consider the data and / or the actual display configuration. For example, user state data ( For example, using the user's gaze point, the UI distortion generation unit generates distortion as the user's eye position moves. When doing so, adjust the viewing angle of the UI distortion so that it remains easy for the user to see. Yes, it is possible. Using an actual display configuration, numerous embodiments of the UI distortion generation unit 360 are Furthermore, UI distortion can be continuously corrected in response to any change in the folding angle. And the perceived UI form factor (e.g., size) generated through UI distortion. ) responds to new actual display configurations and / or user states (e.g., real It can be updated and adjusted (in real time or near real time).

[0111] Furthermore, some embodiments of the UI distortion generation unit 360 complement the actual display configuration. It can generate instructions for the device hardware to compensate for this. In some implementations, U The distortion generation unit 360 compensates for, for example, visual degradation caused by viewing the screen at the outer edge of the viewing cone. To do this, a command is issued to increase the brightness of pixels in specific parts of the UI element that should be visible. It can generate. In other variations, the UI distortion generation unit 360 generates a new A in the new UI. Gestures can still function in areas such as aspect ratio, UI positioning, and distortion of UI elements. Alternatively, instructions may be generated for recalibrating the device's 3D control gestures. .

[0112] Furthermore, for improved user experience and usability, the UI distortion generation unit 360 Some embodiments can modify one or more features of the UI. For example, UI distortion generation unit 36 0 may modify the dimensions of the user interface features. In addition, or alternatively, UI The distortion generation unit 360 can correct the positioning of UI elements on the display. For example, UI The distortion generating unit 360 reduces the amount of distortion required for extreme folding of the display. Elements may be moved from the edges to the center. In some embodiments, the UI distortion generation unit 3 60 modifies many user interface features, such as color, brightness, or pixel distribution. It may be corrected. In some examples, the UI distortion generation unit 360 is (as described above with respect to Figure 2C) When a device (like the example) is folded to a narrower angle, a smaller display The UI can be rearranged to a layout suitable for (i.e., to be perceived by the user). ru. [G. Auxiliary UI element generation section]

[0113] Figure 4C is collapsible using some illustrative components of the UI operation engine from Figure 3. This is an example of a data flow 404 for implementing a capable telephone UI operation technique. Row 404 is between various blocks within the UI operating system (e.g., Engine 300). This shows an example of how data can flow. Figures 3 and 4C are shown simultaneously. With reference to the functional block diagram of the UI operation engine 300, data flow 404 is shown. This will be explained next.

[0114] The auxiliary UI element generation unit 370 guides the user to achieve the optimal display configuration. UI elements or functions that may be used for this purpose, and / or hardware enablement functions (e.g., st It can generate an LED indicator. More specifically, in some examples, The auxiliary UI element generation unit 370 generates an optimal display based on how the user correctly operates the folding screen. When a ray configuration (e.g., optimal folding angle) is achieved, the features of the auxiliary UI elements are conveyed to the user. To allow for changes in order to show, (for example, UI distortion from UI distortion generation unit 360 and / or This uses the optimal display configuration parameters from the optimal display configuration finder 330. You can choose to display one or more auxiliary UI elements.

[0115] More specifically, in some embodiments, auxiliary UI elements have a distortion effect on the UI itself. and / or additional visual guidance indicators (e.g., selected geometric shapes) that the user can use This may include changes in dimensions relative to an intuitively understandable shape or pattern. For example, auxiliary UI elements are distorted so that they become perfectly round only when the screen is changed to the optimal display configuration. It may include a circle (or ellipse, e.g., index 250 in Figure 2B). In some examples, select Additional text guides to show the user what the selected geometric shape is (e.g., a circle). An image may be generated on the display to provide more useful assistance to the user. In one or more embodiments, this additional text guidance ensures the device displays optimally. Without being configured (for example, at the optimal folding angle) (for example, folded) (or before that), provide that it is displayed in an orientation that the user can see - such An example is shown as text guidance 252 in Figure 2B. Content of the text guidance: In some implementations, the actual display configuration (e.g., current folding angle) and the most It can be generated based on the difference between the appropriate display configuration (e.g., selected folding angle) and the appropriate display configuration. For example, text guidance can change the folding angle, i.e., open it more. The user can be instructed to either open or close the window. In some examples, instead of geometric shapes, a picture can be used. The image can be used.

[0116] As an additional embodiment, the auxiliary UI element is configured such that the display is in the optimal display configuration. It may include hardware activation features such as LED indicators. This may include a holographic display indicator, which is a display It can display the correct color or brightness when it is in the optimal display configuration. In addition, or Alternatively, auxiliary UI elements take on a specific shape when the display is in its optimal display configuration. Alternatively, it may include a pattern of pixels that would only appear as an image, or points of illumination. For example, in a fabric display, the fabric is manipulated to the optimal shape. Only if the pixels are machine-readable 1D or 2D barcodes (e.g., QR codes (registration) It may appear as a trademark, and this confirms that the optimal display shape has been achieved. It functions as such.

[0117] Furthermore, in some variations, the auxiliary UI element generation unit 370 displays only the auxiliary UI elements. The display shows that a foldable display is the optimal device configuration (e.g., optimal folding angle) The screen of the user's device will not display sensitive data unless it is operated (to a certain degree). It can be actively blocked, blurred, masked, otherwise obstructed, or simply prohibited. For example, the auxiliary UI element generation unit 370 will block the UI for displaying confidential documents. The UI distortion generation unit 360 is controlled, and guidance information (for example, visual indicators 250) is also provided. , text guidance 252 (in Figure 2B), and / or the LED indicators mentioned above Only the (data) is shown to the user, and then the user follows the instructions and the foldable phone The UI operation engine 300 will only change to the selected optimal folding angle, (for example) (Regarding Figure 2A, confidential information is displayed on the UI using the method described above.) [Methodology]

[0118] Figure 5 shows an example method 500 for implementing UI operation technology for a foldable phone. A flowchart is shown. Method 500 is a confidential, on a foldable UI display. To protect private information and data from bystanders, user devices (e.g.) For example, for UI distortion and rendering on devices 130, 230 (Figures 1A, 2A). Various components of the UI operation engine (e.g., Engine 300; Figure 3) (e.g., configuration) This can be carried out by elements 310-370 (Figure 3). Method 500 is shown in Figure 2A. -2C and Figure 3 are presented with simultaneous reference.

[0119] First, the UI operation engine uses the display of the mobile device (for example, device 230). It can receive instructions to display confidential information on Ray (for example, display 220). 510). One example of such instruction may be a software function call. In this example, the mobile device may include a foldable display. Next, The UI operation engine optimizes the display to determine how sensitive information should be presented. The configuration can be determined (520). In many embodiments, the UI operation engine first, Identify the level of confidentiality of the information that should be displayed on the screen (522), and then, Determine the form factor for how confidential information should be displayed. In addition, Alternatively, the UI manipulation engine determines the form factor of the perceived UI. When determining this, the privacy risks in the surrounding environment of the user device (or how crowded) It is possible to determine whether (524) the form factor is, as mentioned above, generally Perceived display size, i.e., after UI distortion technology, the user (for example, two By folding the foldable display to the optimal folding angle, the device can be optimized for optimal use. The size of the display that the user sees when operating the device in a configuration (e.g., folding it). It may include the following: Generally, the more confidential the displayed data is, or the more confidential the surrounding environment is. The higher the privacy risk, the more the perceived form factor of the displayed UI (for example) The size will be smaller. In other words, the form factor will be smaller, and / or the risks from the surrounding environment will be smaller. This can be inversely correlated with the level of confidentiality of the displayed data. 1 of the disclosed UI operation techniques According to the above embodiments, the determined form factor may, for example, include the presence of bystanders. For privacy scenarios, it will be smaller than the entire area of ​​the foldable display. .

[0120] Next, the UI operation engine, based on the determined form factor, creates a collapsible It can generate UI that is displayed on the screen (530). For example, the UI operation engine is The actual phone configuration can be received (532), and the UI displayed by the UI operation engine is It displays sensitive information in terms of form factor (e.g., perceived size), but collapses. The available displays are selected for the physical phone configuration (e.g., selected folding angle and If not operated to / or one of the selected viewing angles in some embodiments, It is visually transformed in relation to the. As shown in the example UI in Figure 2A, it is collapsible. In the display embodiment, the UI is a foldable display consisting of multiple sections. It can be displayed collectively by [this method]. In certain implementations, a foldable display is folded in half. It can be used as a display.

[0121] Specifically, in one or more embodiments that include a foldable display, the mobile display The user interface is not available unless the display section of the vise is folded to its folding angle. UI distortion is generated so that the face appears distorted to the user. In other words, optimal folding After the folding angle is determined, the display section of the mobile device folds to the folding angle. Unless folded, the user interface will appear distorted to the user. In the example, the UI is deformed symmetrically about an axis along the hinge of the foldable display. (For example, similar to UI examples 204, 206, and 208). The UI distortion engine, for example, uses one or more compass sensors on a mobile device to... It can receive the current orientation status of the mobile device, and the current orientation status of the received mobile device The UI can be further adjusted in response to the current orientation.

[0122] Furthermore, in some embodiments, the UI operation engine is, for example, at the user's point of focus. Possible user states can be detected (534). Based on the user state, the UI operation engine , (for example, in those embodiments that perform dynamic field of view adjustment) mobile devices Unless the user interface is directed towards the user's field of view, the user interface will be distorted to the user. This allows you to determine the field of view. In some of these examples, the UI interaction engine determines the user The eye position can be determined, and the UI can be adjusted in response to the detected position of the user's eyes. In its implementation, the UI operation engine employs a user-facing eye detector on the mobile device. This allows for a new optimal field of view for the UI that reflects the detected position of the user's eyes. The UI can be further adjusted.

[0123] Figure 6 shows another example method 600 for implementing UI operation technology for a foldable phone. This is a flowchart. Method 600 is a confidential UI display on a foldable UI display. To protect private information and data from bystanders, user devices ( For example, further UI distortion and rendering on devices 130, 230 (Figures 1A, 2A) Various components of the UI operation engine for creating a new UI (e.g., Engine 300; Figure 3) (e.g.) For example, it can be carried out by components 310-370 (Figure 3). Method 600 This will be explained with simultaneous reference to Figures 2A-2C and 3.

[0124] The UI operation engine updates the actual current physical configuration of the user device (for example). When the current folding angle is received or obtained (610), the UI distortion can be updated. More specifically, depending on the embodiment, the UI operation engine may include the following new information. It can detect more than 1 of the following: It can detect new instructions for the displayed confidential data (6 12) This represents a new level of privacy risk in the environment surrounding user devices. (614) It can detect a new user state (for example, a new point of focus or a new user state). (616) can detect the new eye position of the sensor, and / or it can detect a new actual data The vice configuration (e.g., a new actual folding angle or a new device orientation) can be detected. ru (618).

[0125] Next, (for example, with respect to Figure 3) consistent with the above-described embodiment, the UI operation engine is newly The optimal device configuration can be determined (620). In some examples, sensitive information for displays can be determined. When there is a new level or update level of data (612), and / or surrounding When there is a new level of privacy risk (614), the UI operation engine will Optimal device configuration (e.g., a new optimal folding angle, and / or several implementations) The form factor allows for the determination of a new optimal field of view (620). Furthermore, dynamic field of view adjustment is implemented. For those that do so, if a new user state (for example, a new focus of attention) is detected, (616), the UI operation engine has a new optimal device configuration for generating UI distortion ( For example, it is necessary to determine the new viewing angle (620). Similarly, dynamic folding angle adjustment For those implementing the adjustment, the newly detected actual device configuration (for example, a new actual When there is a folding angle (618), the UI manipulation engine generates UI distortion. It is necessary to determine a new optimal device configuration (e.g., a new folding angle) (620) ru.

[0126] Subsequently, the UI operation engine adjusts the UI according to the updated, new optimal device configuration. (630). In implementations of 1 or more, UI distortion updates occur when the user manipulates the folded display. It is executed in virtually real time when it is performed. In one or more examples, step 610- The 630 selection steps can be performed repeatedly or recursively. [Example of operation]

[0127] Figure 7A shows a foldable screen device (e.g., device 130, Figure 1A; A bifold display (e.g., display 120, Figure 1A) of the vice 230 (Figure 2A). (220, Figure 2A) A distorted UI is generated (for example, by the UI distortion generation unit 360; Figure 3) Let's look at some more detailed examples of possible methods. More specifically, (for example, regarding Figure 3 as described above) (As shown above) the UI form factor for how confidential information should be displayed was determined. Later, based on the determined form factor, it will be displayed on a foldable display. The actual UI is generated as it should be. It determines how the UI should be distorted when displayed. Therefore, the following is provided, which calculates UI conversion given the above input parameters in Figure 7A. This is an example of a calculation method, including the equation and explanation.

[0128] As shown in Figure 7A, the mobile device 730 has a central axis 736 (e.g., hinge) It includes a foldable display 720 that folds in half around the center. In the following calculation example, The distance (X1) from the central axis 736 to the edge of the foldable display 720 is known. Let's assume that there is. Furthermore, the distance from the central axis 736 to the user's eye (X2) is (for example) , and sensors such as eye trackers and / or ToF sensors that may be mounted on device 730 (Based on) the assumption that it is possible to detect and / or estimate Finally, in the example shown in Figure 7A, the folding angle is represented by (α) and the field of view is It is represented by (β).

[0129] Let (Y1) be the result length of the edge of the manipulated UI, and the result of the center of the manipulated UI Let the length be (Y2) (as shown in Figure 7A). The initial edge of the original, undistorted UI. Let the length be (Y0) (as shown in Figure 7B). The relationship between Y0, Y1, and Y2 is as follows: This can be represented by the following transformation.

number

number

[0130] Figure 7C shows an example of a transformation calculation by changing the folding angle α in the example shown in Figure 7A. Specifically, the following shows an example of a calculation method for T1.

[0131] Continuing to refer to Figures 7C and 7A, assuming that the field of view β is constant, Figure 7C Point P1 (at the center of the display, where the central axis or hinge is located) is the original plane of the device. It is assumed that the device is stationary relative to the surface. The thickness of the device can be ignored compared to ΔX². Assuming that the transformation T1 is from the observer's viewpoint at distance X2 (to the edge of the display) Applied to compensate for the increased perceived change in the height of UI features at point P2. It is possible.

[0132] In this coordinate system, at the center of the display, the folding axis (P1) (for example, axis 736) Since the size of the UI features remains unchanged, Y2 = Y0, and T1 is on the central axis. It does not need to be applied to the UI features.

[0133] Determine the UI features at point P2, which is located at a distance X1 from the origin (for example, at an initial height Y0). Therefore, the transformation T1 is perceived as the size of a feature being close to the observer by a quantity ΔX2. It can be applied to compensate for the increase. Therefore, the transformation is applied to obtain a smaller height Y1. This can be applied to the features at P1 (assuming height Y0). Such a transformation is as follows: It can be expressed as follows.

number

number

[0134] Thus, based on the coordinate system of this example in Figure 7C, the transformation T1 can be expressed as follows: It is possible.

number

[0135] Figure 7D shows an example of the transformation calculation due to the change in field of view β in the example in Figure 7A. Specifically The following is an example of the calculation method for T2.

[0136] Continuing to refer to Figures 7D and 7A, assuming that the folding angle α is constant, Point P1 in Figure 7D (where the point of focus is located, at the center of the display) is the original plane of the device. It is assumed to be static. It is assumed that the device thickness is negligible compared to ΔX². Then, the transformation T2 is from the observer's viewpoint at distance X2 (at the edge of the display) (This can be applied to compensate for the increased perceived change in the height of UI features at point P2.) ru.

[0137] The field of view β in three-dimensional (3D) space has three components: the xy plane, the yz plane, and There is one rotation each in the zx plane. Each can be considered independently. However, a similar equation can be applied for computation.

[0138] In the coordinate system described above, Y2 = Y0, and T2 is applied to the UI feature at the center. To avoid the need to do so, the size of the UI features on the central axis (P1) remains unchanged. be.

[0139] For UI features at point P2 located at a distance X1 from the origin (for example, with an initial height Y0): The transformation T2 compensates for the perceived increase in size due to the feature being only a quantity ΔX2 closer to the observer. It can be applied to compensate. Therefore, the transformation is applied to obtain a smaller height Y1. This can be applied to the features at (assuming height Y0).

number

number

[0140] Therefore, based on the coordinate system shown in the example in Figure 7D, the transformation component T 2xy The table below shows It is possible

number

[0141] A similar transformation occurs with other components of β: β yz and β zx This may also apply to the conversion components. T 2yz and T 2zx These then lead to (using equation (2)) the transformation T2 It is used for this purpose.

[0142] Figure 7E shows the angular distortion in the actual, generated UI in response to changes in the field of view (for example) Here's an example of how the UI can adjust to compensate for the tilt (how it's tilted). This is shown. First, Figure 7E shows (for example, Optimal Display Configuration Finder 3 30. How UI elements (as determined by Figure 3) appear visually to the user. This is an example UI701 showing what should be done. The example UI701 shows the flat corner of the display. It can be displayed in the corner, for example, the foldable design considered here. It can be one section of a display section that forms a display together. To simplify the discussion, the following explanation focuses on adjusting the angular distortion of one section. Hitting the target, the same technology is equally applicable to multiple sections of a foldable display. It can be used. UI701 is displayed on the yz plane, and the x axis extends within the page shown in Figure 7E. Let us assume that. As shown in the figure, UI701 is indicated by a round dot, and visually This includes angles (corners) that would form a right angle (e.g., 90°) to the target. The following considerations are: Using this angle, how is angular distortion adjustment performed in response to changes in the field of view β? Let's explain whether this is possible. Assume that the angle formed by this angle is (γ).

[0143] Furthermore, Figure 7E shows illustrative UI702, 703, 704, and 705. The example UI702-703 shows the display section with a sharper viewing angle for the user. This shows the case where it is viewed from β (i.e., less than 90°), and UI704-705 is a display The emission is viewed by the user from an obtuse field of view β (i.e., greater than 90°). This illustrates the case where there is no compensation for changes in the field of view. The example UI702 shows... As shown, when the display section is viewed from an obtuse viewing angle β, the angle of the corner It can be observed that γ is an acute angle (in contrast to the right angle of UI701). Similarly, the field of view If there is no compensation for the change, the display sector will be affected, as shown in the example UI704. When the image is viewed from an acute field of view β, the angle γ is (in contrast to the right angle of UI701). ) It becomes an obtuse angle.

[0144] Therefore, at least some embodiments (for example, the dynamic field of view UI adjustment described above) According to what is being executed, the UI is designed to detect changes in the perceived tilt angle due to changes in the user's field of view. To take this into consideration, angle distortion adjustment (for example, the tilt at the corners of a box UI element) is necessary. It is possible to receive changes in the angle. With field of view UI adjustment, see example UI703 and 70 As shown in 5, angular distortion caused by changes in the viewing angle can be compensated for. In other words, Some embodiments disclosed herein relate these UI elements based on changes in the field of view β. A mechanism can be employed to determine how much the tilt angle needs to be adjusted.

[0145] Specifically, in some cases, from the user's perception, the angle γ is the hyperbolic tangent function ( This is based on the "tanh()" function and changes with the field of view β. Yes, it is possible. That is, in a particular embodiment, between the angle γ of the displayed angle and the field of view β The relationship can be described, for example, based on an expression that uses tanh().

number

[0146] Figure 7F shows an example of the relationship between the amount of angular distortion adjustment and the change in field of view, more specifically, by equation ( 9) is shown. As shown in Figure 7F, the angle on the UI is an amount inversely proportional to tanh(β). It can be increased or decreased. When the field of view β is 90°, the displayed corner on the actual UI is Angle γ is also 90°, meaning that when the field of view β is vertical, no adjustment of angle γ is necessary. Please take note of this.

[0147] In actual implementation, one or more parameters of equation (9) (i.e., "a", "b", "c") , and / or "d") are specific to the actual device design and application environment, for example, Display characteristics, display size, and / or sensor input (how much of the user's eyes are visible) It may be optimized or adjusted based on factors such as how far away it is and / or where it is located. Please note: In the embodiment shown in Figure 7F, a=1.5, b=2, c=π, and d=1 In some embodiments, the parameters are the user habits and personal characteristics of different people. It can be actively adjusted for each user to better suit their needs (for example, the holding angle or distance).

[0148] Furthermore, the tanh() function described above is merely an example, and in one or more embodiments... It should be noted that functions other than tanh() can be used to achieve a similar effect. For example, in one modified example, the "sigmoid" function (whose plot is similar to that of "tanh") is used. (which is) can be used. Furthermore, some examples are straight lines or lines with different slopes. Multiple combinations of straight lines can be used to approach the visual effect described here. Here, appropriately The function is such that (1) it crosses the origin at a 90° angle (that is, when β is 90°, γ is also 90°). (2) It is recognized that it can be characterized by a plot that is symmetric with respect to the origin. In some embodiments, (3) an increase in β leads to a decrease in γ, and vice versa. It is also characterized by the fact that (i.e., β and γ are inversely correlated) This is also acceptable. In some embodiments, when β is around 0°, γ is around 180°, Conversely, when β is around 180°, γ is around 0°.

[0149] In the embodiments described above, the disclosed UI operation technology is used for the UI for its primary users. While maintaining or improving accessibility, it can be flexible, foldable, or otherwise Concealing or obscuring confidential information displayed on a reconfigurable display from bystanders. This makes it possible. In this way, it prevents the loss of user data and improves the usability and readability of the UI. In addition to maintaining compatibility, the device configuration can be dynamically customized, It provides the optimal security configuration tailored to the current environment. Furthermore, auxiliary UI elements More intuitive to help users easily achieve screen configurations that are optimal for security. A UI system may be implemented.

[0150] Here, in at least some embodiments, the UI operation techniques described are input Transforming a given UI with parameters (e.g., dimensions, folding angle, or viewing angle) For example, one or more existing software developments that can help execute (as described above) It can be implemented based on the SDK. In general, the UI technologies introduced can be built upon. The transformation class may be a subcategory under the SDK related to geometric image transformations, such as perspective. These can be called perspective transformations. Functions available for perspective transformations. An example of such an SDK is OpenCV TM In these SDKs , the dimensions of the input image (for example, the original, undistorted edges of the UI, such as Y0 in Figure 7B) Input describing the length, perspective reference (e.g., field of view, or object plane), and There can be many functions that can take the desired output. In many cases, a function is 1 The images on two planes are "straightened" and then presented on a plane facing the observer. It may be used, programmed, or otherwise directed to do so. In one embodiment, the desired output takes into account the changes in the viewing angle and / or folding angle. It could be from "pre-distorted" graphics. [Grip operation for protecting sensitive data]

[0151] As mentioned above, one approach to shielding confidential information from nearby bystanders is the folding angle. Visual guidance indicators prompting the user to operate the device are displayed on the user's foldable device. The goal is to display the information on a screen. Visual guidance indicators are displayed on a foldable display. The selected geometric shape is only determined when the ray is manipulated to the selected folding angle. (For example, it may be designed to look like a circle.) Visual guidance indicators are collapsible. When the display is not operated to the optimal folding angle, it will switch to a different shape (for example, tilted). It may appear as an ellipse.

[0152] Another approach to shielding confidential information from nearby bystanders is to block the nearby bystanders' field of view. Visual guidance indicators that encourage users to adopt the correct grip position (grasp position) for the device. This can cause a display error. Proper grip positioning reduces the risk of data loss. Therefore, it may be determined based on the optimal placement of the hand and fingers. The risk of data loss is multiple. The determined angle at which the bystander is located around the user device, and therefore the display This could also be based on which parts are currently visible to those bystanders.

[0153] For example, the selected geometric shape (e.g., a circle) is the same as the selected hand (e.g., the left hand). Along with instructions to place the selected finger (e.g., index finger) on the chosen geometric shape, the user device It may also be displayed on a foldable display. In such embodiments, The position of the selected geometric shape is determined by the folding angle and the position of nearby bystanders (multiple bystanders possible). ) may also be based on the following. Embodiments include a user device having a foldable display While it can be explained in the context of [unclear], its features include flexible or otherwise reconfigurable [unclear] This is also applicable to user devices with displays. Therefore, it is explained below. The process is flexible in order to prevent the loss of confidential information in a seemingly natural way. Alternatively, it is adopted by user devices that have a reconfigurable display that is not foldable. It may also be used.

[0154] As will be further discussed below, potential security issues surrounding user devices and Data related to privacy risks and / or displayed by the user's device Data on the level of confidentiality of information provides the optimal grip to avoid information loss to nearby individuals. It can be used to determine the position. The optimal grip configuration is to adopt that configuration. It can be used to generate UI elements (or multiple elements) that encourage or support the user. For example, The current grip position data helps the user move their grip to the optimal position. This can be used to determine which UI element(s) should be generated. The grip position can be, for example, a touch-sensitive element, a pressure-sensitive element, or a proximity sensor. This can be determined based on readings generated by ambient light sensors, etc.

[0155] Figure 8 shows a flowchart illustrating an example method 800 for performing grip manipulation techniques. Method 800 protects sensitive information displayed by a user device from nearby individuals. Therefore, various components of the UI operation engine (e.g., Engine 300; Figure 3) (e.g., This can be carried out by components 310-370 (Figure 3). At a higher level, method 800 is, A hand that would best prevent the unintended disclosure of sensitive information displayed by the user device. This can be described as an algorithmic approach to determining posture.

[0156] First, the UI operation engine displays sensitive information on the user device's display. Instructions for this can be received (810). One example of such instructions is a software function call. It is possible. As mentioned above, the user device has a foldable display, flexible This may include a resizable display or a display that can be reconfigured in another way. Next, the UI The operation engine determines the optimal display configuration for how confidential information is displayed. The "optimal grip position" (also called the "optimal grip position") can be determined (820). In many embodiments, The UI operation engine identifies the level of secrecy of the information to be displayed (822), and then the machine Determine the form factor regarding how the dense information is displayed. In addition, or Alternatively, the UI manipulation engine determines the form factor for the UI based on the user Privacy risks in the surrounding environment of a device can be determined (824). Generally, The higher the level of confidentiality of the data being presented, or the higher the privacy risks in the surrounding environment. The UI form factor is small. In other words, the form factor is influenced by the surrounding environment. These risks and / or the level of confidentiality of the information may be inversely correlated.

[0157] Furthermore, the UI operation engine determines the optimal grip configuration to obscure sensitive information. It is possible (830). The optimal grip configuration (also called the "optimal grip position") is peripheral. Near It may also be based on the individual's position. For example, the peripheral viewfinder may be based on the visibility of the display. The purpose is to block the gaze of these individuals to prevent this, therefore machine vision The number of nearby individuals may be determined using facial recognition technology on the user device's display. If it is flexible or foldable, the UI operation engine will use its current shape or The optimal grip position may be further determined based on the folding angle. Therefore, UI operation The engine may use data relating to the surrounding environment and / or information confidentiality, To determine the privacy requirements for gameplay and to obscure the sensitive information presented therein. Identify the optimal grip position.

[0158] In some embodiments, the optimal grip position is determined by the user's physical dimensions and capabilities (assembly). This is further indicated by what is called "hand characteristics." Examples of hand characteristics include the hand, fingers, and This includes palm size (e.g., measured in terms of width and length) and flexibility. These hand characteristics may be manually entered by the user, or these hand characteristics may be mobile It may be algorithmically determined based on past interactions with the device. These hand characteristics are estimated for a given user based on demographic averages. This estimate may be used directly by the algorithm, or this mean may be The above approach may be further improved.

[0159] As mentioned above, the optimal grip position is the position where the display's field of view is obstructed by the nearby individual. It may be determined based on location. More specifically, the UI operation engine may (for example, around The optimal grip position to block the line of sight of these nearby individuals (as determined by edge data) The placement may be determined (for example, from UI privacy data) This prevents them from seeing the confidential parts. For example, the UI operation engine is near each You may determine an individual's field of view, and then which viewing path to use to protect confidential information. You may decide whether paths need to be blocked. Then the UI operation engine will (i) blocking the visual path, and (ii) feasible considering the characteristics of the user's hand, One or more grip patterns may be generated.

[0160] Next, the UI operation engine, based on the optimal display configuration and optimal grip configuration, It can generate the UI that will be displayed on the screen (840). For example, the UI operation engine is: Data regarding the actual configuration of the user device and the location(s) of nearby individuals(s). (842) may receive (842), then one of the displays on which at least confidential information is presented. You can display UI elements (or more) that indicate where the user should place their hand to hide a portion of the screen. Thus, depending on the nature of the display, the UI operation engine will adjust the orientation, folding angle, etc. Data indicating the degree or shape can be obtained. As shown in the example UI in Figure 10, the hand This shows where to place your selected fingers to ensure you are in the optimal grip position. The above UI elements may be shown.

[0161] Figure 9 shows a flowchart illustrating an example method 900 for implementing UI-driven grip operation technology. The chart is shown. Method 900 allows sensitive information displayed by a user device to be shared with nearby individuals. To protect against this, various components of the UI operation engine (e.g., engine 300; Figure 3) This can be carried out using elements (for example, elements 310-370; Figure 3).

[0162] The UI operation engine receives updates about the actual current physical configuration of the user device. Then (910), the UI update can be performed. For example, the UI operation engine will then execute the following new It may detect one or more of the information: it can detect new displays of sensitive data. (912) This represents a new privacy risk in the environment surrounding the user device. It can detect a new user state (for example, a new grip) (914) (916) can detect the position, and / or it can detect a new actual device configuration. (For example, a new orientation, folding angle, or shape) can be detected (918).

[0163] For example, the UI operation engine uses one or more sensors built into the user device (" Data generated by a "grip position sensor" (called "grip data") The grip position can be continuously monitored by examining the grip position sensor. Examples include touchscreen support components (e.g., touch-sensitive elements and pressure-sensitive elements). This includes IV elements, proximity sensors, ambient light sensors, gyroscopes, accelerometers, etc. In some embodiments, the grip position is sensed by vibration (for example, by a motor or piezoelectric element) It is supported by the active generation of stimuli such as (which are generated). In addition, or alternatively, for example, For example, to determine the position of the hand based on corneal reflection, an optical sensor facing the user (for example) Technology including a camera may be employed. Grip data is collected from the user device and The position of the palm relative to the display, and the user device and its display You may also describe various characteristics of the grip position, including the position of the thumb (or multiple thumbs) relative to the "I" position. Therefore, the UI operation engine generates grip position sensors (multiple sensors are possible) You can obtain grip data, and then examine the grip data to determine the optimal grip position. That's fine.

[0164] Next, consistent with the approach described above (for example, with respect to Figure 8), the UI operation engine is This allows for the determination of a new optimal grip position (920). For example, the UI operation engine is If there is a new or updated level of confidentiality for display purposes (912), and / Or, if there is a new level of privacy risk in the surrounding environment (914), This allows for the determination of the optimal grip position. For example, the UI operation engine can determine the surrounding environment to be more complex. If it is decided that the UI operation will become cluttered (and as a result, privacy will be reduced), The engine requires a new optimal grip position that will further obscure the confidential information. It can be decided that one will obtain it.

[0165] Subsequently, the UI manipulation engine can adjust the UI according to the new optimal grip configuration (9 30). As will be further discussed below with respect to Figure 11, the adjustment is made by the user's selected finger (multiple fingers). This may include adding UI elements (multiple elements are possible) that indicate where to place (one or more). In this context, UI adjustments are made when the user interacts with the display (for example, its orientation, folding). This is done in virtually real time (by changing the folding angle or shape). Similarly, UI adjustments are virtually real when users move their hands on the user device. It may be executed in time. For example, the location of sensitive information interacts with the display. For example, it may be modified in response to a decision by the user to move their hand to touch something. Steps 910-930 may be performed repeatedly or recursively.

[0166] Figure 10A shows gripping using some example components of the UI manipulation engine from Figure 3. This is data flow 1000, an example of implementing the operation technique. Data flow 1000 is How data is handled between various blocks within the UI operating system (e.g., Engine 300) This shows an example of how it can flow. Data flow 1000 is the data flow in Figure 4A. It is substantially similar to the Lowe 400. However, here we have to consider the characteristics of the user's hand. The data is also acquired by the optimal display configuration finder 330. As mentioned above. This data helps determine which grip positions are feasible for the user to achieve. It can be used for that purpose.

[0167] On the other hand, Figure 10B uses some example components of the UI operation engine from Figure 3 to... Data flow 1002 is another example for implementing the RIP manipulation technique. As mentioned above. The grip position sensor (multiple sensors are possible) 910 generates grip data that can be used to estimate the position of the hand. It may play a role in achieving this. An example of a grip position sensor is a touchscreen support component (for example) (e.g., touch-sensitive elements and pressure-sensitive elements), proximity sensors, ambient light sensors) This includes gyroscopes, accelerometers, etc. In addition, or alternatively, for example, corneal reflection To determine the hand position based on this, an optical sensor (e.g., a camera) facing the user is included. The following technology may be employed. Using grip data, the grip position finder 920 is It can determine the user's actual grip position. In other words, it can fine-tune the grip position. The DA920 divides the grip data generated by the grip position sensor (multiple sensors possible) 910. Based on the analysis, the actual position of the hand can be determined. Information regarding the actual grip position is supplementary. This may also be provided to the auxiliary UI element generation unit 370, which is the optimal display configuration finder. The optimal display configuration is determined by 330, and the grip position viewfinder 920 Based on the actual grip position determined, the UI is required to implement the grip operation technique. It can be configured to generate elements (or multiple elements).

[0168] Figures 11A-D show an example implementation of grip operation technology. First, user device 11 The UI operation engine that runs on 00 (for example, engine 300; Figure 3) is shown in Figures 8-9. As mentioned above, the location of nearby individuals 1102a-c can be established. Roach ensured that these individuals could not observe the display of user device 1100. Allow UI interactions to identify visual paths that must be blocked in order to proceed. More specifically, the UI operation engine ensures privacy, as shown in Figure 11B. It is possible to calculate which directions (multiple directions) must be blocked in order to achieve this.

[0169] Next, the UI manipulation engine selects appropriate UI elements (multiple elements are possible) to present on the display. ) can be identified. For example, the UI operation engine can determine the calculated direction(s), This can determine where visual guidance indicators should be displayed on the display. (Figure 11C) For example, the UI operation engine displays a visual guidance indicator 1104 in the upper right corner of the display. This is causing the presentation of. In some embodiments, the visual guidance indicator 1104 is It is one of several visual guidance indicators displayed on a screen. For example, UI The operating engine may determine that multiple visual guidance indicators are necessary in crowded environments. Multiple visual guidance indicators are available to help you select fingers on the same hand or different hands. You may do so.

[0170] The visual guidance index 1104 is used to position the selected finger on the visual guidance index. This may be accompanied by instructions. The user selects a finger 1106 on the visual guidance indicator 1104. If placed there, the field of view of nearby individuals 1102a-c would be obstructed (therefore, confidential information (Ensure the information remains confidential.) In some embodiments, visual guidance Indicator 1104 is designed to be substantially covered by the selected index 1106. Here For example, the selected finger 1106 can see part of the display, while the visual guide It completely covers the performance index 1104.

[0171] Many user devices allow multi-touch functionality (or simply "multi-touch"). It is designed to do so. Multitouch means that the touch-sensitive display can do so simultaneously. It makes it possible to recognize the presence of multiple points in contact with the display. The device 1100 allows one finger to be held in one area while another finger touches another area. At that time, multiple contact points may be detected. To take this into consideration, touch-sensitive displays In some embodiments, part of the gameplay is outside the multi-touch target area. It may be defined as follows. For example, user device 1100 is a touch-sensitive display Touches occurring within the fixed area of ​​Ray (e.g., visual guidance indicator 1104 and its vicinity) The user device 1100 may be instructed not to detect the event, or the user device 1100 may be I have been instructed not to use such touch events to determine the multitouch action. That's good too. Therefore, touch-sensitive displays can handle multiple touch events. At least one part recognized for the purpose of touch, and touch events are multitouch It may have at least one part that is not recognized for the purpose, and visual guidance indicators ( (or multiple visual guidance indicators) recognize touch events for the purpose of multitouch. They may be placed in those parts(s) that are not recognized.

[0172] The visual guidance index 1104 also provides a touch-sensitive indicator for the selected finger 1106. Note that this may represent instructions for positioning the spray completely outside its boundaries. Example: As shown in Figure 11C, the visual guidance indicator 1104 indicates that the selected finger 1106 is - Represented as an arrow indicating that it should be positioned along the top or side of the device 1100. It is possible.

[0173] At a high level, UI elements(s) generated by a UI manipulation engine are generally, It fulfills one of two purposes. Firstly, the user needs to activate the desired function. To find the optimal grip position, it is necessary to interact with these UI elements (multiple elements are possible). It is possible. Secondly, the user can use these UI elements(s) to perform the desired function. It may be necessary to interact with it. Examples of desired functions include payment confirmation, password entry, and access to financial or personal information. Therefore, until the optimal grip position is taken, The system may not be able to perform the desired function.

[0174] In some embodiments, UI elements(s) are required for a given function. It represents an existing element. For example, the UI operation engine will have "Show password" as a UI element. You may use existing labeled graphics, but their characteristics (e.g., position and The size may be adjusted to meet the requirements described above. In other embodiments, the UI An element(s) represents a new element that satisfies the requirements described above. For example, a UI operation engine. This system uses fingerprints (multiple fingers) to indicate where the selected finger(s) should be placed. It may be overlaid on a UI where the function can be completed.

[0175] Each UI element created by the UI manipulation engine determines its appearance, function, and position. It has properties such as dimensions, color, animation, etc. This includes the calling position. Similarly, these properties trigger the activation of the corresponding UI element. You can specify the necessary actions, if any (e.g., press and hold, swipe). (Tap, repeated taps). In some embodiments, these properties are UI elements ( The choice of which (multiple) is created / selected is influenced by the fundamental characteristics of the underlying UI. For example, The color of the UI element(s) will be changed to match the underlying UI's color scheme. obtain.

[0176] In some embodiments, the grip operation technique is used by the user device 1100 It is controlled by the system. For example, the user can (for example, to indicate location) the display's edge. (By tapping the ledge) Specify where nearby individuals 1102a-c are located. They may be prompted to do so. As another example, users may use visual guidance indicators (e.g., visual guides). It may be possible to influence the position, number, or arrangement of the lance indicator (1104). For example, the user is initially prompted to place their selected finger on the visual guidance indicator 1104. It is possible, but the user must perform a certain action (for example, tap at least twice and then drag). The position of the visual guidance index 1104 may be changed by performing the above. Thus, the characteristics of the user's hand are used to guide the visual guidance indicators shown on the user device 1100. It may be used to determine the appropriate position. In some embodiments, the user visually Different arrangements of the sensory guidance indicators are displayed on the user device 1100's display. You may be prompted to complete the calibration process. These arrangements are in various situations. It may include a different number of visual guidance indicators placed in each location. Based on the speed and ease with which a selected finger can be placed on one or more visual guidances, The UI operation engine may learn the most appropriate layout for the user. [Computer System and Device Architecture]

[0177] Figure 12 shows a computer in which at least some of the operations described herein may be implemented. This is a block diagram showing an example of the 1200 ng system. For example, the UI operation engine ( For example, a computing device including the UI operation engine 300 in Figure 3 (for example) The components used to implement the user devices 130 and 230 in Figures 1A and 2A These are some of the components of the Puteling System 1200.

[0178] The computing system 1200 is communicably connected to bus 1216, and has 1 or more The central processing unit (also called the "processor") above is 1202, the main memory is 1206, and Volatile memory 1210, network adapter 1212 (for example, network interface Face), video display 1218, input / output device 1220, control device 1222 (example) For example, a drive unit including a keyboard and pointing device, and a recording medium 1226. 1224 and signal generator 1230 may be included. Bus 1216 may include a suitable bridge, One or more physical buses and / or points connected by an adapter or controller It is illustrated as an abstract representation of a two-point connection. Therefore, bus 1216 This refers to the system bus, PCI (Peripheral Component Interconnect) bus, or PCI- Express bus, HyperTransport, or ISA (industry standard) architecture) bus, SCSI (Small Computer System Interface) bus, USB (Uni Versal serial bus), IIC (I2C) bus, or IEEE (Institute of Electrical Engineering) (and Electronics Engineers) Standard 1394 bus ("Firewire®") This may include (also called) and others.

[0179] Computing System 1200 is a personal computer, tablet computer Computers, mobile phones, game consoles, music players, wearable electronic devices (for example, time (e.g., a meter or fitness tracker), a network-connected ("smart") device (e.g., (e.g., a television or home assistant device), virtual / augmented reality systems (e.g., headset) (Mounted display) or performed by computing system 1200 A set of instructions that specify actions (or multiple actions) can be executed (sequentially or in other ways). It may share a computer processor architecture similar to that of electronic devices. stomach.

[0180] Main memory 1206, non-volatile memory 1210, and recording medium 1226 ("mechanically volatile" Although it is shown as a single medium, the term "machine-readable medium" (also called a "readable medium") is used. " and "recording medium" refer to a single medium or multiple media that store one or more sets of instructions 1228. A system (e.g., a centralized / distributed database and / or associated cache and server) It should be interpreted to include. Also, the terms "machine-readable medium" and "recording medium" should be interpreted to include. The computation system 1200 stores and encodes the instruction set for execution. This shall be interpreted as including any medium through which it can be carried.

[0181] Generally, routines performed to implement embodiments of the embodiments of this disclosure are operating systems. A system or a specific application, component, program, object, module A sequence of instructions (collectively called a "computer program") It can be implemented as a part. Computer programs are typically computing data One or more instructions set at various timings in various memories and storage devices within the device (e.g.) For example, it includes instructions 1204, 1208, and 1228. One or more processors 1202 When read and executed, the instruction(s) are sent to the computing system 1200. to perform actions to execute elements relating to various aspects of this disclosure.

[0182] Furthermore, the embodiments are described in the context of a fully functional computing device. However, those skilled in the art will see that various embodiments can be distributed as various forms of program products. You will understand that this disclosure is the machine used to actually implement the distribution. This applies regardless of the specific type of computer-readable medium.

[0183] Further examples of machine-readable recording media, machine-readable media, or computer-readable media include volatile Non-volatile memory devices 1210, floppy disks and other Removable disks, hard disk drives, optical discs (e.g., compact discs) Scred-only memory (CD-ROM), digital multi-purpose disc (DVD), etc. Recordable media, as well as transmission types such as digital and analog communication links. Including the medium.

[0184] The network adapter 1212 connects to the computing system 1200. Any communication program supported by the 1200 communication system and external entities Through Tocol, entities and networks that are outside the computing system 1200 Network adapter 121 enables the mediation of data within network 1214. 2 is a network adapter card, wireless network interface card, router, Access points, wireless routers, switches, multilayer switches, protocol converters, Gateways, bridges, bridge routers, hubs, digital media receivers, and / Alternatively, it can include repeat customers.

[0185] The network adapter 1212 provides access to data within the computer network. / Determine and / or manage proxy permissions and different machines and / or applications It may include a firewall that tracks various levels of trust between systems. The code is a combination of machine and application, machine and machine, and / or application features. A predetermined set of access rights can be enforced between a set of entities (for example, these entities) Hardware and / or software that regulate the flow of traffic and resource sharing between them. It can be any number of modules having any combination of fair components. Earwall further enables objects by individuals, machines, and / or applications This document details permissions, including access rights, operational rights, and the circumstances under which authorization is granted. Access to the access control list may be managed and / or may be held.

[0186] The technology described here involves programmable circuits (e.g., one or more microprocessors). ), software and / or firmware, special purpose hardwired (sun) It can be implemented by a circuit (which is not programmable), or a combination of such forms. A special purpose circuit is a programmable integrated circuit (ASIC) consisting of one or more application-specific integrated circuits. Forms such as logic devices (PLDs) and field-programmable gate arrays (FPGAs) It is possible. [remarks]

[0187] The foregoing description of various embodiments of the claimed subject matter is provided for illustrative and descriptive purposes only. Therefore, it must be comprehensive, or disclose the requested subject matter in the exact form it is presented. This is not intended to be limiting. Many modifications and variations will be obvious to those skilled in the art. The embodiments are selected and described in order to best illustrate the principle and practical applications of the present invention. Therefore, those skilled in the art will be able to understand the claimed subject matter, various embodiments, and intended You will be able to understand various modifications that are suitable for specific uses.

[0188] The detailed description illustrates specific embodiments and the intended best mode, but the detailed No matter how detailed the description may seem, this technology can be implemented in many ways. Embodiments are described herein. While encompassed by the same entity, the details of its implementation can vary considerably. Certain terms used when describing specific features or aspects are associated with the terms themselves. Redefined herein to be limited to any specific characteristics, features, or aspects of the technology being referred to. It should not be interpreted as meaning that it will be done. Generally, in the following claims The terms used herein are defined herein unless otherwise explicitly defined herein. The technology should not be construed as limiting it to the specific embodiments shown. Therefore, The actual scope of this technology is not limited to the disclosed embodiments, but also includes any implementation or implementation of the embodiments. It encompasses all equivalent methods.

[0189] The language used in this specification has been selected primarily for readability and clarity. Yes, it exists. It was not chosen to define or enclose the subject. Therefore, The scope of this technology is not defined by this detailed description, but rather by the application issued thereunder. It is intended to be limited by any claim. Therefore, various embodiments are open. The following illustration illustrates, but does not limit, the scope of the technology defined in the claims below. It is intended to be something other than that. [Industrial applicability]

[0190] This disclosure is applicable to data security.

Claims

[Claim 1] A program that displays information, the program is A process for displaying information on a mobile device, wherein the mobile device includes a foldable display. A process for determining a form factor relating to how the aforementioned information is displayed, wherein the form factor is determined based on the folding angle of the foldable display. A process for generating a user interface to be displayed on a foldable display based on a determined form factor, wherein the user interface displays the information in the form factor, but is visually altered to the user unless the foldable display is folded to a selected angle. A process for obtaining the change in the folding angle of the aforementioned foldable display, and A process for dynamically adjusting the user interface in response to a change in the folding angle, A program that runs on the computer built into a mobile device.

Images