Image display method

Abstract

To provide an image display method capable of displaying an image in relationship to an operation unit (physical button) formed in a part of a region on a surface of a display unit.SOLUTION: An information processing apparatus includes: a display unit having a display including a display screen on which an image including at least one operation image is displayed, and a touch panel for detecting an operation of a user touching the operation image; and a control unit which forms an operation unit to be touched by the user for operation in a part of a region on a surface of the display unit. An image display method includes: a determination step of determining, by the information processing apparatus, whether the operation unit is available or not based on available operation unit information regarding available operation unit and hardware profile information regarding the operation unit; and an operation unit control step. The operation unit control step includes deforming the partial region so that the operation unit is formed in the partial region when it is determined that the operation unit is available.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image display method. [Background technology]

[0002] A method is known in which, under control of a control unit, an operation unit (physical button) that a user can touch and operate is formed in a partial area on the surface side of a display unit including a display and a touch panel (or the partial area on which the physical button is formed is returned to its original state before the operation unit was formed) (see, for example, Patent Document 1 and Non-Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2011-508935 [Non-Patent Document 1] https: / / drive.google.com / file / d / 1--KfMTvkerE1dZxiXC6Uf-zRVMtbA6YY / view Summary of the Invention [Problem to be solved by the invention]

[0004] However, Patent Document 1 and Non-Patent Document 1 do not at all consider displaying images in relation to the operation unit (physical buttons) formed in a partial area on the surface side of the display unit, and there is room for improvement in this regard.

[0005] The present disclosure has been made to solve such problems, and provides an image display method that can display an image in relation to an operation unit (physical button) formed in a partial area on the surface side of the display unit. [Means for solving the problem]

[0006] The image display method according to the present disclosure is an information processing device including a display including a display screen on which an image including at least one operation image is displayed and a display unit equipped with a touch panel that detects a user's touch operation on the operation image, and a control unit that forms an operation unit that the user touches and operates in a partial area on the surface side of the display unit, the information processing device comprising: a determination step of determining whether the operation unit is available or not based on available operation unit information regarding an available operation unit and hardware profile information regarding the operation unit; an operation unit control step of deforming the partial area so that the operation unit is formed in the partial area or so that the partial area in which the operation unit is formed returns to its original state before the operation unit was formed; an image display step of displaying the image on the display screen with the operation image aligned to match the operation unit; and a contact operation detection step of detecting the user's touch operation on the operation unit, and the operation unit control step deforms the partial area so that the operation unit is formed in the partial area if it is determined that the operation unit is available. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to provide an image display method that can display an image in relation to an operation unit (physical button) formed in a partial area on the surface side of a display unit. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a schematic configuration diagram of an information processing device 1 that implements an image display method according to a first embodiment. [Figure 2] 2 is a cross-sectional view of an example of a display unit 11. FIG. [Figure 3] 1 is an example of a program stored in the storage unit 12. [Figure 4] 11A is a diagram illustrating a state in which an operation surface image 12a2 is displayed on a display 11b (display screen) in a state in which a button image b designated by button image designation information 12a3 is aligned to coincide with a physical button B. FIG. [Figure 5]10 is a flowchart of an operation example 1 (image display method) of the information processing device 1. [Figure 6] 10 is a flowchart of an operation example 2 (image display method) of the information processing device 1. [Figure 7] 11A is a diagram illustrating a state in which an operation surface image 12a2 is displayed on a display 11b (display screen) in a state in which a button image b designated by button image designation information 12a3 is aligned to coincide with a physical button B. FIG. [Figure 8] 10 is a flowchart of an operation example 3 (image display method) of the information processing device 1. [Figure 9] (a) An example of a physical button B formed in a partial area A on the front side of the display unit 11, used in operation example 3; (b) an example of a game image HT1 displayed in a "partial area" of the display 11b (display screen); and (c) an example of a game image HT2 displayed in the "entire area" of the display 11b (display screen). DETAILED DESCRIPTION OF THE INVENTION

[0009] (Embodiment 1) Hereinafter, an information processing device 1 that implements an image display method according to the first embodiment will be described with reference to the accompanying drawings. Corresponding components in each drawing are given the same reference numerals, and redundant description will be omitted.

[0010] FIG. 1 is a schematic diagram of an information processing device 1 that implements an image display method according to the first embodiment.

[0011] The image display method of the first embodiment is implemented in an information processing device 1. The information processing device 1 is, for example, a smartphone carried by a user or a navigation device mounted on a vehicle. The information processing device 1 has a function of forming a physical button B (an example of an operation unit of the present disclosure) that a user touches and operates in a partial area on the front side of the display unit 11 (physical button forming area 11a), and a function of returning the partial area on which the physical button B is formed to its original state before the physical button B was formed. Specifically, the information processing device 1 is configured as follows.

[0012] As shown in FIG. 1, the information processing device 1 includes a display unit 11, a storage unit 12, a control unit 13, and a memory .

[0013] The display unit 11 includes a physical button forming portion 11a, a display 11b, and a touch panel 11c.

[0014] FIG. 2 is a cross-sectional view of an example of the display unit 11. As shown in FIG.

[0015] As shown in FIG. 2, the physical button forming portion 11a and the touch panel 11c are arranged in an overlapping state on the display 11b. The display 11b includes a display screen on which an operation surface image 12a2 (see, for example, FIG. 3) including at least one operation image b is displayed. The display 11b (display screen) displays various images (e.g., operation surface images, game images) in addition to the operation surface image 12a2 including the operation image b. The user visually recognizes the various images displayed on the display 11b (display screen) through the physical button forming portion 11a and the touch panel 11c. The touch panel 11c detects the user's touch operations (e.g., tapping, sliding, flicking, swiping, pinching in / out, etc.). While a capacitive touch sensor is often used for the touch panel 11c, it may also be configured with a transparent, highly sensitive pressure sensor, or may be configured with both sensors. This makes it possible, for example, to simultaneously detect the position of a touch on the surface of the partial area A (electrostatic touch detection) and the depression (pressure detection). The display unit 11 may further include a haptic sheet (not shown) disposed below the display 11b to provide feedback for various touch operations.

[0016] A physical button B (for example, a convex button or a concave button; an example of an operation unit of the present disclosure) that the user touches is formed in a partial area A on the surface side (physical button forming area 11a) of the display unit 11. This is realized, for example, by the control unit 13 (physical button control unit 13a) controlling an actuator (not shown) to deform (raise or lower) the partial area A on the surface side (physical button forming area 11a) of the display unit 11.

[0017] For example, the actuator is a microfluidic pump (not shown) connected via a flow path (not shown) to a hollow portion S between a partial area A (a predetermined fixed area where physical button B is formed) on the surface side (physical button forming portion 11a) of the display unit 11 and the touch panel 11c.

[0018] Because the surface side (physical button forming portion 11a) of the display unit 11 is an elastically deformable layer portion, the control unit 13 (physical button control unit 13a) controls the actuator to move (supply) fluid to the cavity S via a flow path to increase the volume, thereby pushing up a partial area A of the surface side (physical button forming portion 11a) of the display unit 11 that corresponds to the cavity S. As a result, the partial area A of the surface side (physical button forming portion 11a) of the display unit 11 rises, and a physical button B that the user touches and operates is formed (see FIG. 2).

[0019] Conversely, when the control unit 13 (physical button control unit 13a) controls the actuator to move (discharge) the fluid from the hollow portion S to the outside through the flow path, thereby reducing the volume, a portion of area A on the surface side (physical button forming portion 11a) of the display unit 11 corresponding to the hollow portion S returns to its original state (e.g., a flat state) before the formation of the physical button B.

[0020] As described above, in accordance with control from the control unit 13 (physical button control unit 13a), a physical button B that the user touches is formed in a partial area A on the surface side (physical button forming area 11a) of the display unit 11 including the display 11b and the touch panel 11c (or the partial area A on which the physical button B is formed is returned to its original state before the physical button B was formed). As a means for realizing this, for example, JP2011-508935, Flat Panel Haptics (https: / / drive.google.com / file / d / 1--KfMTvkerE1dZxiXC6Uf-zRVMtbA6YY / view) can be used.

[0021] The shape (outer shape) of the physical button B is, for example, rectangular as shown in Fig. 1, but is not limited to this and may be circular, cross-shaped, or another shape. The number of physical buttons B may be one or more.

[0022] The storage unit 12 is a non-volatile storage unit such as an SSD or a ROM.

[0023] FIG. 3 shows an example of the programs stored in the storage unit 12.

[0024] As shown in FIG. 3, the storage unit 12 stores an application 12a (application software, hereinafter also referred to as app 12a), an OS 12b (operating system), and hardware profile information 12c.

[0025] The application 12a mainly includes a program 12a1, an operation screen image 12a2, button image designation information 12a3, and available physical button information 12a4.

[0026] The program 12a1 may be any program that has a function of displaying the operation surface image 12a2 on the display 11b (display screen). Specific examples of the program 12a1 include a game program (for example, a smartphone) and a navigation program (for example, a navigation device).

[0027] The operation surface image 12a2 includes an operation image b that the user touches and operates via the touch panel 11c. Specific examples of the operation surface image 12a2 include a remote control image, a game image, and a window image. Specific examples of the operation image b include a button image included in the remote control image, a button image (e.g., a cross-shaped button image) displayed together with the game image, and a button image included in the window image for closing the window image. Hereinafter, the operation image b will also be referred to as a button image b. Note that the operation image b is not limited to a button image, and may be a key image or a slider image.

[0028] The shape (outline) of the operation surface image 12a2 is, for example, rectangular as shown in Fig. 3, but is not limited thereto and may be circular or another shape. The size of the operation surface image 12a2 may be the same as that of the display 11b (display screen), or may be larger or smaller than that of the display 11b (display screen). The number of operation surface images 12a2 may be, for example, one, or may be a collection of multiple images as shown in Fig. 3.

[0029] On the other hand, the shape (external shape) of the button image b is, for example, a triangle or a circle as shown in FIG. 3, but is not limited thereto and may be a rectangle, a cross, or any other shape. The shape (external shape) of the button image b may correspond to the shape (external shape) of the physical button B, or may not correspond to the shape (external shape) of the physical button B. The size of the button image b may be the same as the size of the physical button B, or may be larger or smaller than the size of the physical button B. The number of button images b may be, for example, one or more.

[0030] The button image designation information 12a3 (an example of operation image designation information in the present disclosure) is information that designates a button image b to be aligned with a physical button B formed in a partial area A on the surface side (physical button forming portion 11a) of the display unit 11. The button image b designated by the button image designation information 12a3 is, for example, a button image corresponding to a main operation button or a button image corresponding to a most frequently operated button. The button image designated by the button image designation information 12a3 may be set by a user, or may be automatically set by the application 12a or the like by applying predetermined image processing such as image recognition to the operation surface image 12a2 or by adding additional data to the operation surface image 12a2.

[0031] Specific examples of the main operation button image or the most frequently operated button image specified by the button image specification information 12a3 are a play button image included in a remote control image, a cross button image displayed together with a game image, and a button image for closing a window image included in a window image.

[0032] The usable physical button information 12a4 (an example of usable operation unit information in the present disclosure) is information about physical buttons (physical buttons formed in a partial area on the front side (physical button forming area) of the display unit) that can be used by the application 12a. The usable physical button information 12a4 is used in the information processing device 1 in which the application 12a is installed to determine whether the application 12a can use a physical button B formed in a partial area A on the front side (physical button forming area 11a) of the display unit 11 of the information processing device 1.

[0033] For example, the available physical button information 12a4 includes information on the layout, operation, and attributes of physical buttons available to the application 12a (for example, metadata of UI elements written in html, css, XML format, etc.).

[0034] The arrangement is information representing, for example, the arrangement on the display of a physical button usable by the app 12a (such as an area where the physical button is formed). The arrangement may include, for example, information on a rough position on the screen (for example, right-side, center, or left-side; or, for example, top, middle, or bottom), upper / lower limit display sizes for which zooming in or out is permitted, etc. The operation is information representing, for example, an operation when a physical button usable by the app 12a is touch-operated. It may also include, for example, information on a margin width for the touch operation, touch sensitivity, etc. The attribute may include information representing the type of physical button usable by the app 12a (such as a cross button, an ABXY button, a stick-type button, or a slider-type button), and information specifying whether or not pressure-sensitive detection is enabled for the physical button usable by the app 12a.

[0035] The hardware profile information 12c is information about the physical buttons B formed in a partial area A on the front side (physical button forming area 11a) of the display unit 11, and includes information about the arrangement, operation, and attributes of the physical buttons B. The hardware profile information 12c may be part of the OS 12b.

[0036] The layout is, for example, information representing the layout of the physical button B on the display 11b (such as the area where the physical button B is formed). The action is, for example, information representing the action when the physical button B is touched. The attributes may include information representing the shape of the physical button B (such as a cross button, an ABXY button, a stick button, or a slider button), and information representing whether or not hardware support for pressure-sensitive detection of the physical button B is available.

[0037] The control unit 13 includes a processor (not shown). The processor is, for example, a CPU (Central Processing Unit). There may be one processor or multiple processors. The processor executes the OS 12b and program 12a1 loaded from the storage unit 12 into the memory 14 (for example, RAM), thereby functioning as a physical button control unit 13a, a contact operation detection unit 13b, an operation surface image display unit 13c, and a processing unit 13d, as shown in FIG. 1. Some or all of these may be realized by hardware.

[0038] The physical button control unit 13a forms a physical button B that the user touches and operates in a partial area A on the surface side (physical button forming unit 11a) of the display unit 11. For example, the physical button control unit 13a controls an actuator (not shown) to deform the partial area A on the surface side (physical button forming unit 11a) of the display unit 11 so that a physical button B (an example of an operation unit of the present disclosure) that the user touches and operates is formed in the partial area A, or so that the partial area A on which the physical button B is formed returns to its original state (for example, a flat state) before the physical button B was formed. The physical button B may be a convex button or a concave button.

[0039] The contact operation detection unit 13b detects a contact operation (such as a touch or a press via the touch panel 11c) on the physical button B by the user.

[0040] The operation surface image display unit 13c displays the operation surface image 12a2 (for example, the operation surface image 12a2 selected by the user) on the display 11b (display screen) in a state where the button image b designated by the button image designation information 12a3 is positioned to match the physical button B. Fig. 4 is a diagram illustrating a state where the operation surface image 12a2 is displayed on the display 11b (display screen) in a state where the button image b designated by the button image designation information 12a3 is positioned to match the physical button B.

[0041] The processing unit 13d executes processing corresponding to the contact operation detected by the contact operation detection unit 13b. Specific examples of the processing corresponding to the contact operation include, for example, a video or audio playback process when the button image b is a play button image included in the remote control image; a process appropriate for inputting up, down, left, or right directions corresponding to the button's shape in accordance with the game progress when the button image b is a cross-shaped button image displayed together with the game image; and a process for closing the window image when the button image b is a button image for closing the window image included in the window image.

[0042] Next, a first operational example (image display method) of the information processing device 1 having the above configuration will be described.

[0043] FIG. 5 is a flowchart of an operation example 1 (image display method) of the information processing device 1.

[0044] The following processing is mainly performed by the processor executing the OS 12b and the program 12a1 that are loaded from the storage unit 12 into the memory 14 (for example, RAM).

[0045] First, the application 12a is launched (step S10). The application 12a is launched by a predetermined user operation on the information processing device 1. At this time, the user may select an operation surface image 12a2 to be aligned with a physical button B formed in a partial area A on the front side (physical button forming portion 11a) of the display unit 11 from among a plurality of operation surface images 12a2. The selection of the plurality of operation surface images 12a2 is performed, for example, by operating a group of thumbnail-format images arranged on the screen or a tab window.

[0046] Next, a physical button B (see FIG. 2) is formed in a partial area A on the surface side (physical button forming area 11a) of the display unit 11 (step S11). This is achieved by the physical button control unit 13a. The physical button control unit 13a controls an actuator to deform the partial area A on the surface side (physical button forming area 11a) of the display unit 11 so that a physical button B that the user touches and operates is formed in the partial area A. As a result, the physical button B (see FIG. 2) is formed in the partial area A on the surface side (physical button forming area 11a) of the display unit 11.

[0047] Next, the operation surface image 12a2 (for example, the selected operation surface image 12a2) is displayed on the display 11b (display screen) with the button image b aligned to match the physical button B (step S12, see FIG. 4). This is achieved by the operation surface image display unit 13c. Specifically, the operation surface image display unit 13c displays the operation surface image 12a2 on the display 11b (display screen) with the button image b designated by the button image designation information 12a3 aligned to match the physical button B formed in the partial region A on the front side (physical button forming portion 11a) of the display unit 11 as described above (see FIG. 4).

[0048] Next, it is determined whether or not a user's touch operation on the physical button B has been detected (step S13). This is achieved by the touch operation detection unit 13b.

[0049] As a result of the determination in step S13, if it is determined that a user's touch operation on the physical button B has been detected (step S13: YES), a process corresponding to the touch operation is executed (step S14). This is realized by the processing unit 13d. The processing unit 13d executes a process corresponding to the touch operation detected by the touch operation detection unit 13b.

[0050] Thereafter, the processes of steps S13 and S14 are repeatedly executed.

[0051] Next, a second operation example (image display method) of the information processing device 1 having the above configuration will be described.

[0052] FIG. 6 is a flowchart of an operation example 2 (image display method) of the information processing device 1.

[0053] The following processing is mainly performed by the processor executing the OS 12b and the program 12a1 that are loaded from the storage unit 12 into the memory 14 (for example, RAM).

[0054] First, the application 12a is started (step S20). The application 12a is started by a predetermined user operation on the information processing device 1. At this time, the user may be allowed to select an operation surface image 12a2 to be aligned with a physical button B formed in a partial area A on the front side of the display unit 11 (physical button forming portion 11a) from among a plurality of operation surface images 12a2.

[0055] Next, the operation screen image 12a2 (for example, the selected operation screen image 12a2) is displayed at the initial position (step S21). For example, as shown in Fig. 7, the operation screen image 12a2 is displayed at the initial position P1. Fig. 7 is a diagram showing a state in which the operation screen image 12a2 is displayed on the display 11b (display screen) in a state in which the button image b designated by the button image designation information 12a3 is aligned to coincide with the physical button B.

[0056] Next, the user moves the operation surface image 12a2 displayed at the initial position P1 within the display 11b (display screen) toward a partial area A on the front side (physical button forming portion 11a) of the display unit 11 in accordance with the user operation (see arrow AR1 in FIG. 7). That is, the operation surface image 12a2 is an image that can be moved within the display 11b (display screen) in accordance with the user operation. Note that, although the initial position P1 is shown in the upper right corner in FIG. 7 for the sake of explanation, this is not limiting. Furthermore, the movement of the operation surface image 12a2 by the user operation is typically performed by a touch operation such as sliding, but may also be performed by an operation such as flicking to skip the operation surface image 12a2.

[0057] Next, it is determined whether or not the button image b included in the operation surface image 12a2, which moves in response to the user's operation as described above, has approached a partial area A on the front side (physical button forming portion 11a) of the display unit 11 (step S22).

[0058] As a result of the determination in step S22, if it is determined that the button image b included in the operation surface image 12a2 that moves in response to a user operation has approached the partial area A on the surface side of the display unit 11 (physical button forming area 11a) (step S22: YES), a physical button B (see FIG. 2) is formed in the partial area A on the surface side of the display unit 11 (physical button forming area 11a) (step S23). This is achieved by the physical button control unit 13a. The physical button control unit 13a controls the actuator to deform the partial area A on the surface side of the display unit 11 (physical button forming area 11a) so that the physical button B that the user touches and operates is formed in the partial area A. As a result, the physical button B (see FIG. 2) is formed in the partial area A on the surface side of the display unit 11 (physical button forming area 11a).

[0059] Next, the operation surface image 12a2 is displayed on the display 11b (display screen) with the button image b aligned to match the physical button B (step S24). This is achieved by the operation surface image display unit 13c. Specifically, the operation surface image display unit 13c displays the operation surface image 12a2 on the display 11b (display screen) with the button image b designated by the button image designation information 12a3 aligned to match the physical button B formed in the partial region A on the front side of the display unit 11 (physical button forming portion 11a) as described above (see FIG. 7).

[0060] Next, it is determined whether or not a user's touch operation on physical button B has been detected (step S25). This is achieved by the touch operation detection unit 13b.

[0061] As a result of the determination in step S25, if it is determined that a user's touch operation on physical button B has been detected (step S25: YES), a process corresponding to the touch operation is executed (step S26). This is realized by the processing unit 13d. The processing unit 13d executes a process corresponding to the touch operation detected by the touch operation detection unit 13b.

[0062] On the other hand, if it is determined in step S22 that the button image b included in the operation surface image 12a2 that moves in response to user operation is not close to a partial area A on the surface side of the display unit 11 (physical button forming area 11a) (step S22: NO), it is determined whether the button image b included in the operation surface image 12a2 that moves in response to user operation has moved away from the physical button B formed in the partial area A on the surface side of the display unit 11 (step S27).

[0063] As a result, if it is determined that the button image b included in the operation surface image 12a2, which moves in response to a user operation, has moved away from the physical button B formed in the partial region A on the front side of the display unit 11 (physical button forming portion 11a) (step S27: YES), the partial region A on which the physical button B is formed is returned to its original state (for example, a flat state) before the physical button B was formed (step S28). This is achieved by the physical button control unit 13a. The physical button control unit 13a controls the actuator to deform the partial region A on which the physical button B is formed so that the partial region A returns to its original state before the physical button B was formed.

[0064] Thereafter, the processes from step S22 onwards are repeatedly executed.

[0065] Next, a third operation example (image display method) of the information processing device 1 having the above configuration will be described.

[0066] FIG. 8 is a flowchart of an operation example 3 (image display method) of the information processing device 1.

[0067] The following processing is mainly performed by the processor executing the OS 12b and the program 12a1 that are loaded from the storage unit 12 into the memory 14 (for example, RAM).

[0068] The following description will be given assuming that the application 12a (program 12a1) stored in the storage unit 12 is a game program.

[0069] The physical button B used in Operation Example 3, which is formed in a partial area A on the front side of the display unit 11 (physical button forming area 11a), is a cross-shaped button that is generally used on game controllers and the like (see FIG. 9(a)). Also, the button image b used in Operation Example 3 is a cross-shaped button image (see FIG. 9(a)). Also, the game image is an example of a main image of the present disclosure, and the black band area is an example of an area other than the main image of the present disclosure.

[0070] First, the application 12a is started up by a predetermined user operation on the information processing device 1 (step S30).

[0071] Next, a game image (an example of a specific image of the present disclosure) is displayed on the display 11b (display screen) (step S31). The game image may be of various games (action games, shooting games, simulation games, racing games, adventure games, role-playing games, etc.) depending on the launched application 12a. FIG. 9(b) is an example of a game image HT1 displayed in a "partial area" of the display 11b (display screen). The portion marked with the symbol BK in FIG. 9(b) indicates the area other than the main game image HT1 as a so-called black area. FIG. 9(c) is an example of a game image HT2 displayed in the "entire area" of the display 11b (display screen).

[0072] Next, in order to determine whether the application 12a can use the physical button B formed in a partial area A on the front side of the display unit 11, it is determined whether the available physical button information 12a4 matches the hardware profile information 12c (step S32).

[0073] If it is determined in step S32 that the available physical button information 12a4 and the hardware profile information 12c match (step S32: YES), a physical button B (a cross-shaped button) is formed in a partial area A on the front side of the display unit 11 (physical button forming section 11a) (step S33). This is achieved by the physical button control section 13a. The physical button control section 13a controls an actuator to deform the partial area A on the front side of the display unit 11 (physical button forming section 11a) so that a physical button B that the user touches and operates is formed in the partial area A. As a result, the physical button B is formed in the partial area A on the front side of the display unit 11 (physical button forming section 11a).

[0074] Next, the button image b is aligned with the screen area of ​​the display 11b (display screen) where the physical button B overlaps, and the display mode of the button image b displayed on the display 11b (display screen) is adjusted according to the state of that image area (i.e., the screen area behind the physical button B) (step S34).

[0075] For example, if the screen area of ​​the display 11b (display screen) that the physical button B overlaps is a black band area BK (see FIG. 9(b)), the display mode of the button image b is adjusted to emphasize the button image b (blinking, enlarged, etc.). In this way, the appearance of the physical button B can be changed without affecting the progress of the game.

[0076] Furthermore, for example, if the screen area of ​​the display 11b (display screen) that overlaps with the physical button B is the game image HT2 (see FIG. 9(c)), the display mode of the button image b is adjusted so that the button image b is made transparent (not limited to being completely transparent, but may be semi-transparent, etc.). The display mode may also be adjusted so that the button image b is displayed with an adjusted degree of transparency. The button image b may be made transparent for the entire button image, or may be made transparent by leaving only a faint outline. In this way, the screen area behind the physical button B (game image HT2) is not covered by the button image b and can be seen, preventing any impact on the progress of the game.

[0077] Next, the button image b is displayed on the display 11b (display screen) with the button image b aligned to match the physical button B (step S35). Specifically, the button image b, whose display mode has been adjusted as described above, is displayed on the display 11b (display screen) with the button image b aligned to match the physical button B formed in the partial area A on the front side (physical button forming portion 11a) of the display unit 11 as described above.

[0078] Next, it is determined whether or not a user's touch operation on physical button B has been detected (step S36). This is achieved by the touch operation detection unit 13b.

[0079] As a result of the determination in step S36, if it is determined that a user's touch operation on physical button B has been detected (step S36: YES), a process corresponding to the touch operation is executed (step S37). This is realized by the processing unit 13d. The processing unit 13d executes a process corresponding to the touch operation detected by the touch operation detection unit 13b.

[0080] Thereafter, the processes of steps S36 and S37 are repeatedly executed.

[0081] On the other hand, if it is determined in step S32 that the available physical button information 12a4 and the hardware profile information 12c do not match (for example, if it is determined that they do not match when the button image in the available physical button information 12a4 does not contain information about a cross-shaped button) (step S32: NO), a button image is displayed in a predetermined position without forming a physical button B (cross-shaped button) in a partial area A on the front side of the display unit 11 (physical button forming unit 11a) (step S39).

[0082] Thereafter, the processes of steps S36 and S37 are repeatedly executed in the same manner as above.

[0083] The process of step S32 may be executed in the above-mentioned Operation Example 1. In the Operation Example 1, the process of step S32 may be executed, for example, between steps S10 and S11. Similarly, the process of step S32 may be executed in the above-mentioned Operation Example 2. In the Operation Example 2, the process of step S32 may be executed, for example, between steps S22 and S23.

[0084] As described above, according to embodiment 1, an image (for example, operation surface image 12a2 including operation image b in operation examples 1 and 2 or button image b in operation example 3) can be displayed in relation to a physical button B (an example of an operation unit of the present disclosure) formed in a partial area A on the surface side (physical button forming portion 11a) of the display unit 11.

[0085] Next, a modified example will be described.

[0086] The user may switch whether or not to form the physical button B.

[0087] For example, a dedicated physical key (not shown) may be provided in the information processing device 1, and the user may switch whether or not to form the physical button B in response to the user's operation of the dedicated physical key. In addition, if there are multiple physical buttons B, the combination pattern for forming the physical button B in response to the user's operation of the dedicated physical key may be switched cyclically.

[0088] Furthermore, whether or not the physical button B is formed may be switched depending on the orientation of the information processing device 1.

[0089] For example, when it is detected that the information processing device 1 is in landscape orientation (e.g., see FIG. 9(a)), the physical button B may be set to be formed at the left end as shown, whereas when it is detected that the information processing device 1 is in portrait orientation (e.g., see FIG. 7), it may be set not to form the physical button B. Note that when it is detected that the information processing device 1 is in portrait orientation (e.g., see FIG. 7), the button image b that has not been aligned may be displayed in its initial position.

[0090] Furthermore, a standard rule may be applied to physical button B. Since the position of physical button B can be determined by touch, it becomes easier for the user to perform blind operations.

[0091] For example, if a physical button B is formed in the upper right corner of the display 11b (physical button forming portion 11a), the physical button B may have the function of a button for closing a window image.

[0092] Furthermore, for example, if a long physical button B is formed on the right edge of the display 11b (physical button forming portion 11a), the physical button B may have the function of volume UP / DOWN.

[0093] Furthermore, the object to be moved is not limited to button image b, but may be, for example, a QR code on a browser screen moved to a specific position (for example, the outer frame of the formed physical button B). Positioning the QR code (registered trademark) in a predetermined frame makes it easier for visually impaired people to perform operations such as holding a smartphone over a QR code reading terminal device.

[0094] Furthermore, the location of the pull-in may be changed depending on the attribute and whether or not pressure is detected for physical button B. For example, it is possible to use a cross button and a stick button in a different way, or to use a touch operation and a pressure-sensitive pressing operation in a different way.

[0095] All the numerical values ​​shown in the above embodiment are merely examples, and it goes without saying that other appropriate numerical values ​​can be used.

[0096] The above-described embodiments are merely examples in all respects. The present disclosure should not be construed as being limited by the description of the above-described embodiments. The present disclosure can be implemented in various other forms without departing from the spirit or main features thereof. [Explanation of symbols]

[0097] 1. Information processing equipment 11 Display section 11a Physical button forming part 11b Display 11c Touch Panel 12 Storage section 12a Apps (Applications) 12a1 Program 12a2 Operation screen image 12a3 Button image specification information 12a4 Available physical button information (available operation unit information) 12b OS 12c Hardware Profile Information 13 Control Unit 13a Physical button control section 13b Touch operation detection unit 13c Operation surface image display section 13d Processing section 14 Memory A Some areas B Physical Button BK Black area (area other than the main image) HT1, HT2 game images (specific images) P1 initial position S cavity b Button image (operation image)

Claims

1. An information processing device comprising: a display unit including a display screen on which an image including at least one operation image is displayed and a touch panel that detects a user's touch operation on the operation image; and a control unit that forms an operation unit that is touched by the user in a partial area on the front surface of the display unit, a determining step of determining whether the operation unit is available based on available operation unit information regarding the available operation unit and hardware profile information regarding the operation unit; an operation unit control step of deforming the partial region so that the operation unit is formed in the partial region or so that the partial region on which the operation unit is formed returns to its original state before the operation unit is formed; an image display step of displaying the operation image on the display screen in a state where the operation image is aligned with the operation unit; a touch operation detection step of detecting a touch operation by the user on the operation unit, The operation unit control step includes, when it is determined that the operation unit is available, transforming the partial area so that the operation unit is formed in the partial area.

2. The image display method according to claim 1 , wherein the available operation unit information includes information about the layout, operation, and attributes of the available operation units.

3. The image display method according to claim 2 , wherein the attributes include information indicating the type of available operation unit and information indicating whether or not the available operation unit detects pressure.

Images