Information processing device, computer program and information processing method

Abstract

To improve operability in a multi-window user interface.SOLUTION: A tablet terminal 100 arranges a code window for displaying a programming code on the front, and arranges an execution window for displaying a program execution screen on the back of the code window. When performing data input after a user instructs a focus change in a function determination area, the data input to the execution window is allowed while keeping the code window on the front of the execution window.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a multi-window user interface technology. [Background technology]

[0002] With the advent of mobile computers such as laptop PCs, tablet PCs, and smartphones, the number of opportunities to use computers has increased dramatically. Mobile computers are required to be lightweight and compact. For this reason, when designing the user interface of mobile computers, the key point is how to efficiently input and output as much information as possible onto the limited size of the monitor screen.

[0003] Generally, a multi-window user interface is adopted, which simultaneously displays multiple pieces of information in multiple windows (areas for displaying images). A "layer (display layer)" is set for each window. Multiple windows are displayed as if the back (lower layer) window is entirely or partially hidden behind (below) the front (upper layer) window. The user inputs data into the front window. Hereinafter, the window into which data is input is referred to as the "focus window". [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-168717 A Summary of the Invention [Problem to be solved by the invention]

[0005] When the window into which you want to enter information is in the background, you need to change the display order so that this window becomes the front layer (hereafter referred to as "bringing to the foreground"). Specifically, you bring the background window to the foreground by minimizing (deactivating) the foreground window. Alternatively, you may perform an operation to display miniature versions of multiple windows in parallel, and then select the focus window (the window into which you want to enter information) from among them. The operation to bring a window to the foreground, or in other words, the effort required to bring it to the foreground, can easily lead to a decrease in workability when you want to frequently change the focus window while entering information.

[0006] An object of the present invention is to provide a technique for improving the operability of a multi-window type user interface. [Means for solving the problem]

[0007] An information processing device in one aspect of the present invention includes a display control unit that places a first area for displaying a first image in the foreground and a second area for displaying a second image behind the first area, and an operation detection unit that detects a first operation for designating the second area and a second operation for inputting data from a user. When a second operation is detected after the first operation is detected, the display control unit changes the second image based on the second operation while maintaining the first area in front of the second area. Effect of the Invention

[0008] According to the present invention, it becomes easier to improve the operability in a multi-window type user interface. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is an external view of a tablet terminal. [Diagram 2] FIG. 2 is a schematic diagram for explaining a layer structure of a plurality of windows. [Diagram 3] FIG. 2 is a diagram illustrating a hardware configuration of a tablet terminal. [Figure 4] FIG. 2 is a functional block diagram of the tablet terminal. [Diagram 5] 11 is an external view for explaining the configuration of a function judgment region. FIG. [Figure 6] FIG. 13 is a schematic diagram for explaining a normal input. [Figure 7] FIG. 13 is a schematic diagram for explaining a penetration input. [Figure 8] 11 is a flow chart illustrating the process that is performed when a touch on a target selection area is detected. [Figure 9] 13 is a flowchart showing a process when a touch outside the selection area is detected. [Figure 10] 11 is a flowchart showing a process when detachment is detected. [Figure 11] FIG. 13 is an external view for explaining switching of a data input target. [Figure 12] FIG. 13 is a first screen diagram showing a state when a penetration input is made to an execution window. [Figure 13] FIG. 2 is a second screen diagram showing the state when normal input is made to the code window. [Figure 14] FIG. 13 is a third screen diagram showing the state when normal input is made to the code window. [Figure 15] FIG. 4 is a fourth screen diagram showing the state when a circle is drawn in the execution window. [Figure 16] FIG. 13 is a diagram showing a screen when an explanation of function input is displayed. [Figure 17] FIG. 13 is a diagram showing a screen when a gesture is input. [Figure 18] FIG. 13 is a conceptual diagram for explaining focus control of three or more windows. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] In this embodiment, multiple windows are arranged and displayed in multiple layers. A window is an image area that resembles a single sheet, and a layer is a concept that indicates the overlapping position of multiple sheets when they are stacked.

[0011] First, imagine physical paper sheets 1, 2, and 3. Sheets 1, 2, and 3 each contain information such as pictures and text. Let's stack these three sheets in order. Suppose sheet 1 is on top, sheet 2 is in the middle, and sheet 3 is at the bottom. In this case, the user can see all of sheet 1, the top sheet. Of sheet 2, the second from the top, only the part that is not overlapping with sheet 1 is visible. Of sheet 3, the third from the top, only the part that is not overlapping with either sheets 1 or 2 is visible. A multi-window system is a computer program that achieves the same visibility as with such stacked sheets.

[0012] Similarly, imagine electronic windows 1, 2, and 3 in a multi-window system. Windows 1, 2, and 3 display information in the same way as sheets. In this case, the "top" layer corresponding to sheet 1 is called the "front layer." "Front" means that it is in front (or appears to be in front) of the user looking at the monitor. Let's try placing three windows on each of the three layers. Suppose window 1 is placed on the top layer, window 2 on the middle layer, and window 3 on the bottom layer. In this case, window 1 is said to be placed on the layer "in front" of windows 2 and 3. Similarly, window 2 is said to be placed on the layer "behind" window 1, and on the layer "in front" of window 3. Window 3 is said to be placed on the "back" layer.

[0013] The method of arranging multiple windows on multiple layers, such as window 1, 2, 3, is called the "display order." In the above example, it would be expressed as "The display order of multiple windows, from the front, is window 1, 2, 3." Also, changing the layer of a window along with a change in the display order is called "rearrangement" of the window.

[0014] In the following, "bringing to the foreground" means changing the arrangement of the layer of a specified window to the foreground in a multi-window user interface. "Targeting" means setting a specified window as a target for data input, in other words, setting a specified window as the focus window. "Data input" here means adding or modifying the content displayed by the window, specifically, input for changing text or images. "Activating" means turning on a special function layer. In a general multi-window system, "bringing to the foreground" and "targeting" are essentially the same concept, but in this embodiment, "targeting without bringing to the foreground" is realized.

[0015] FIG. 1 is an external view of a tablet terminal 100. As shown in FIG. The tablet terminal 100 (information processing device) is a thin mobile computer equipped with a monitor screen 102. A touch panel is installed on the monitor screen 102. In this embodiment, a description will be given assuming a class in which programming is taught to users. Students are users of the tablet terminal 100.

[0016] Application software for programming education (hereinafter referred to as "learning software") is pre-installed on the tablet terminal 100. The purpose of the lesson is to have the user write simple program code (hereinafter simply referred to as "code") and execute the created program, thereby deepening the user's understanding of how software works. As will be described in detail later, a window for creating code and a window that is the program execution screen are displayed separately as multiple windows.

[0017] In the following, a description will be given assuming a touch panel, but in addition to the touch panel, input devices such as a keyboard and a mouse can also be connected to the tablet terminal 100. The tablet terminal 100 may also be capable of accepting voice input.

[0018] FIG. 2 is a schematic diagram for explaining the layer structure of a plurality of windows. The learning software has a code window X1 (first area) and an execution window X2 (second area) as "normal layers" for inputting and outputting data. It also includes a function judgment area M1 and a gesture area G1 as "function layers" for supporting input to the code window X1 and the execution window X2. Both function layers are invisible. From the front (top), there are four layers: the function judgment area M1, the gesture area G1, the code window X1, and the execution window X2. The code window X1 and the execution window X2 are the windows that serve as the working area of ​​the learning software. Normally, the code window X1 is the focus window. The function judgment area M1 is always enabled, and the gesture area G1 is normally disabled.

[0019] The function judgment area M1 is located in the foreground. The user can change the execution window X2 to the focus window by accessing a selection area formed in part of the function judgment area M1 (details will be described later). The user can also activate the gesture area G1 by accessing another selection area formed in part of the function judgment area M1. In the following, the state in which the code window X1 becomes the focus window is called the "normal mode," and the state in which the execution window X2, whose display area is blocked because it is located in a layer below the code window X1, becomes the focus window is called the "penetration mode (special mode)." The state in which the gesture area G1 is activated is called the "gesture mode." The learning software has these three modes, which are mutually exclusive.

[0020] The gesture area G1 is located behind the function judgment area M1. The gesture area G1 is invisible and is not normally enabled. When the gesture area G1 is enabled by a method described below, the user can input various gestures using the entire monitor screen 102. The details of gesture input will be described later in conjunction with FIG. 17.

[0021] The code window X1 is displayed semi-transparently, and provides a screen (first image) for writing code. The term "semi-transparent display" here refers to displaying the code window X1 with its transparency adjusted to such an extent that the image in the code window X1 is visible, and the image behind it can also be seen through the code window X1. The code window X1 is located behind the gesture area G1, but is in the foreground in the normal layer. In normal mode or gesture mode (when both the code window X1 and the execution window X2 are displayed), the code window X1 becomes the focus window.

[0022] The execution window X2 is displayed opaquely (normally displayed) like a general window. The execution window X2 provides an execution screen (second image) of the program created in the code window X1. The execution window X2 is located behind the code window X1. Because the code window X1 is semi-transparent, the user can simultaneously see not only the code window X1 but also the execution window X2 behind it. When the mode is changed to the through mode, or when window X1, which is a visible layer in front of the execution window X2, is not displayed, the execution window X2 is targeted, i.e., the focus window.

[0023] To summarize, it is as follows: 1. Function layer: invisible 1-1. Function judgment area M1: Always enabled. Accepts input for changing modes. 1-2. Gesture area G1: Disabled in normal mode and piercing mode, and enabled by function judgment area M1 (gesture mode). Accepts gesture input. 2. Normal layer: visible 2-1. Code window X1: Displayed semi-transparently and objectified in normal mode. It may also be temporarily made fully transparent (invisible) as described below. 2-2. Execution window X2: Displayed opaque and objectified in piercing mode.

[0024] The display order of the code window X1 and the execution window X2, in other words, the positional relationship between the layers, may be interchangeable; however, unless otherwise specified below, it will be explained as if the display order (front-to-back relationship) of the function judgment area M1, gesture area G1, code window X1, and execution window X2 is not interchangeable. Although it is possible to close the code window X1 and the execution window X2, in the following description, unless otherwise specified, it is assumed that the code window X1 and the execution window X2 cannot be closed and that both are visible in the normal mode.

[0025] FIG. 3 is a diagram showing the hardware configuration of the tablet terminal 100. As shown in FIG. The tablet terminal 100 includes a storage 312 as a non-volatile memory for storing computer programs, a volatile memory 304 for expanding programs and data, and a processor 300 (CPU: Central Processing Unit) that incorporates registers, an arithmetic unit, an instruction decoder, etc. and reads and executes programs from the memory 304. The processor 300 is connected to a relatively high-speed first bus 302. In addition to the memory 304, a NIC (Network Interface Card) is connected to the first bus 302. In addition to the memory 304, other devices such as a GPU (Graphics Processing Unit) may also be connected to the first bus 302.

[0026] The first bus 302 is connected to a relatively slow second bus 310 via a bridge 308. In addition to a storage 312, an output device 316 such as a monitor screen 102 or a speaker is connected to the second bus 310. In addition, an input device 314 such as a mouse or keyboard, and a peripheral device 318 such as a printer may also be connected to the second bus 310.

[0027] FIG. 4 is a functional block diagram of the tablet terminal 100. As described above, each component of the tablet terminal 100 is realized by hardware including computing units such as a CPU and various co-processors, storage devices such as memory and storage, and wired or wireless communication lines connecting them, and software stored in the storage devices and supplying processing instructions to the computing units. The computer programs may be configured by device drivers, an operating system, various application programs located at higher layers than these, and libraries that provide common functions to these programs. Each block described below represents a functional block rather than a hardware-based configuration.

[0028] The tablet terminal 100 includes a user interface processing unit 104, a data processing unit 106, a communication unit 126, and a data storage unit 108. The user interface processing unit 104 accepts operations from the user via the touch panel, and is responsible for processing related to the user interface, such as image display and audio output. The communication unit 126 is responsible for communication processing with other devices via the Internet. The data storage unit 108 stores various data. The data processing unit 106 executes various processes based on data acquired by the user interface processing unit 104 and the communication unit 126 and data stored in the data storage unit 108. The data processing unit 106 also functions as an interface between the user interface processing unit 104, the communication unit 126, and the data storage unit 108.

[0029] The user interface processing unit 104 includes an operation detection unit 110 that receives input from a user, and an output unit 112 that outputs various information such as images and sounds to the user. The operation detection unit 110 includes a function input unit 114, a gesture input unit 116, and a data input unit 118. The function input unit 114 detects a function selection input (first operation) that is a predetermined input to the function judgment area M1. The function selection input is an input for controlling the function or state of the window itself, such as activation or objectification, rather than the contents of the window itself, and will be described in detail later with reference to FIG. 5 and the like. The gesture input unit 116 detects a gesture input that is a predetermined input to the gesture area G1. The data input unit 118 detects a data input (second operation) to the code window X1 or the execution window X2.

[0030] The data storage unit 108 stores learning software as well as programs created by the user.

[0031] The data processing unit 106 includes a display control unit 120 that controls the display of the code window X1 and the execution window X2. The display control unit 120 includes a display update unit 122 and a layer management unit 124. The display update unit 122 manages image generation and updating in the code window X1 and the execution window X2. The layer management unit 124 manages the layer, position, and size of each window. The layer management unit 124 also manages modes related to normal mode, special mode, and gesture mode.

[0032] FIG. 5 is an external view for explaining the configuration of the function judgment area M1. In invisible function judgment area M1, selection areas 128L1, 128R1, 128L2, and 128R2 (hereinafter collectively referred to as “selection areas 166”) are set at the four corners of monitor screen 102.

[0033] When the user touches the selection area 128L1 or the selection area 128R1, in other words, when the user touches the lower left corner or the lower right corner of the monitor screen 102, the layer management unit 124 transitions from the normal mode to the through mode (special mode). At this time, the execution window X2 on the rear side is objectified. Hereinafter, the two selection areas 128L1 and 128R1 for objectifying the execution window X2 among the selection areas 166 will be collectively referred to as the "objectified selection area 162."

[0034] When the user stops touching the object selection area 162 (detaches), the layer management unit 124 returns the mode from the through mode to the normal mode. At this time, the code window X1 on the foreground side is objectified again.

[0035] When the user touches selection area 128L2 or selection area 128R2, in other words, when the user touches the upper left corner or upper right corner of monitor screen 102, layer management unit 124 transitions from normal mode to gesture mode. Layer management unit 124 activates gesture area G1, making gesture input possible. Hereinafter, selection area 128L2 and selection area 128R2 for activating gesture area G1 of selection area 166 will be collectively referred to as "gesture activation area 164."

[0036] When the user stops touching the gesture validation area 164 (detaches), the layer management unit 124 transitions from the gesture mode to the normal mode. At this time, the gesture area G1 is disabled. Touching any of the four selection areas 166 is called a "function selection input."

[0037] In this embodiment, there are two types of data input: "normal input" in normal mode and "through input (special input)" in through mode. Data input (mainly input of characters or images) can be made by touch input with a finger or stylus into the window area, but other input devices such as a mouse can also be used. In normal mode, the code window X1, which is in front of the execution window X2, is the focus window, and the user inputs data into the code window X1. This is called "normal input."

[0038] When the user touches the target selection area 162, the mode is set to the through mode, the execution window X2 on the back side becomes the focus window, and the user inputs data into the execution window X2. This is called "through input."

[0039] During through input, the layer management unit 124 does not change the anteroposterior relationship between the code window X1 and the execution window X2. In other words, through input provides a unique operational feeling as if you were touching the execution window X2, which is in the background, through the code window X1, even though the code window X1 is in the foreground. This operational feeling will be further explained with reference to the following Figures 6 and 7.

[0040] FIG. 6 is a schematic diagram for explaining normal input. As mentioned above, the layers visible to the user are the two normal layers: the code window X1 and the execution window X2. The code window X1 is in the foreground, and the execution window X2 is in the background. Because the code window X1 is semi-transparent, the user can simultaneously see not only the code window X1, but also the execution window X2 in the background. Therefore, the user can check the execution status of the program while writing and checking the program code.

[0041] In normal mode, the code window X1 is the focus window. When the user touches the monitor screen 102, the data input unit 118 determines this as data input (normal input) to the code window X1 (focus window). The user inputs data into the code window X1 by touching. The display update unit 122 displays the input data in the code window X1 according to the input result, or moves the cursor. It is also possible to move the code window or scroll the character area.

[0042] FIG. 7 is a schematic diagram for explaining the penetration input. When the user touches the target selection area 162 (selection area 128L1 or selection area 128R1), the mode transitions from normal mode to through mode. The layer management unit 124 sets the execution window X2, not the code window X1, as the focus window. In this state, when the user touches the monitor screen 102, the data input unit 118 determines this as data input (through input) to the execution window X2. The user inputs data by touching the execution window X2. The display update unit 122 draws an image in the execution window X2 according to the input result. The display update unit 122 also updates the code window X1 according to the result of the image input to the execution window X2.

[0043] In the through mode, the user feels as if his / her finger is penetrating through the code window X1 in the foreground and operating the execution window X2 in the background. In the through mode, the display update unit 122 may temporarily hide the code window X1 (make it completely transparent). In this case, the user can input data while looking only at the execution window X2. In the following description, it is assumed that in the through mode, the foreground code window X1 is not displayed.

[0044] FIG. 8 is a flow chart showing the process that occurs when a touch on the target selection area 162 is detected. The process shown in FIG. 8 is executed when a touch on the object selection area 162 of the monitor screen 102 is detected. First, when a touch on the monitor screen 102 is detected, the function input unit 114 judges whether or not the touch is on the object selection area 162 (selection area 128L1, selection area 128R1) (the lower left corner or the lower right corner of the monitor screen 102). If the touch is on the object selection area 162, the layer management unit 124 sets the through mode (S10). At this time, the execution window X2 becomes the focus window. The display update unit 122 sets the code window X1 to hidden (S12). That is, when the object selection area 162 is touched and the mode shifts to the through mode, the execution window X2 becomes the focus window, and the code window X1 in front of the execution window X2 becomes completely transparent (hidden). The code window X1 becomes invisible, but is not closed. Also, the code window X1 is not changed to a layer behind the execution window X2. The process performed when a touch on the gesture validation area 164 is detected will be described later with reference to FIG.

[0045] FIG. 9 is a flow chart showing the process when a touch outside the selection area 166 is detected. When a touch outside the selection area 166 is detected, if the mode is normal (S20: normal mode), the data input unit 118 determines whether the touch point is in the area where the code window X1 and the execution window X2 overlap (hereinafter referred to as the "overlap area") (S22). If the touch is on the overlap area (Y in S22), the data input unit 118 accepts data input with the code window X1 as the target (S24). That is, when the overlap area of ​​the code window X1 and the execution window X2 is touched and the code window X1 (foreground) is the focus window, normal input is performed.

[0046] If the touched point is not in an overlapping area (N in S22), data input unit 118 accepts data input targeting code window X1 or execution window X2 (S26). At this time, layer management unit 124 temporarily treats the window accepting the touch operation as the focus window. For example, when only code window X1 is at the touch point, it becomes normal input to code window X1, and when only execution window X2 is there, it is treated as normal input to execution window X2. In either case, the hierarchical relationship of layers, with code window X1 in the foreground and execution window X2 in the background, is maintained.

[0047] When a touch is detected, if it is the through mode (S20: through mode), the data input unit 118 judges whether the touch point is in the overlapping area (S28). If it is a touch on the overlapping area (Y in S28), the data input unit 118 accepts data input for the execution window X2 (S30). That is, when the overlapping area of ​​the code window X1 and the execution window X2 is touched and the execution window X2 (back side) is the focus window, a through input is made.

[0048] If the touched point is not in the overlapping area (N in S28), the data input unit 118 accepts data input for the touched window out of the code window X1 or the execution window X2 (S32). In this case, the layer management unit 124 also temporarily treats the window accepting the touch operation as the focus window.

[0049] FIG. 10 is a flow chart showing the process that occurs when a detachment is detected. The process shown in Fig. 10 is executed when detachment (touch off) from the monitor screen 102 is detected. First, in the case of detachment during the penetration mode (Y in S40), the display update unit 122 redisplays the code window X1 that was hidden in S12 (Fig. 8) (S42). At this time, the code window X1 is displayed in a normal semi-transparent manner. The layer management unit 124 sets the normal mode (S44). The code window X1 is objectified.

[0050] The display update unit 122 updates the data of the non-focused windows in accordance with the data input. For example, when data is input to the code window X1, the display update unit 122 changes the image of the execution window X2 in accordance with the input content to the code window X1. When data is input to the execution window X2, the display update unit 122 changes the image of the code window X1 in accordance with the input content to the execution window X2. Data updating will be described in detail with reference to FIG. 12 and subsequent figures.

[0051] In the case of detachment during normal mode (N in S40), the process proceeds to S46.

[0052] FIG. 11 is an external view for explaining switching of a data input target. The user can freely change the position and size of the code window X1, etc., just as with a typical multi-window system. In Fig. 11, the code window X1 on the foreground side has been moved to the lower right. Meanwhile, the execution window X2 on the background side is displayed across the entire monitor screen 102.

[0053] The overlapping area 132 indicates an area where the execution window X2 (background) is hidden by the code window X1 (foreground). The exposed area 134 indicates an area where the background execution window X2 is exposed as a result of the foreground code window X1 moving to the lower right. When the user touches point P1 of the exposed area 134, the exposed background execution window X2 becomes the focus window (the window into which data is input) regardless of whether the through mode is in effect. When the exposed area 134 is touched, the layer management unit 124 temporarily treats the execution window X2 as the focus window.

[0054] When the user touches point P2 of overlapping area 132, as described above, either the code window X1 or the execution window X2 becomes the focus window (the window into which data is input) depending on whether the mode is the through mode or the normal mode. Note that when the execution window X2 is moved and a touch is made in an area where only the code window X1 exists, the code window X1 becomes the target for data input.

[0055] Next, as a specific example of the user interface, a process of creating code for drawing a circle while selectively using penetrating input and normal input will be described with reference to Figs. 12 to 15.

[0056] FIG. 12 is a first screen diagram showing a state when a penetration input is made to the execution window X2. The user touches the monitor screen 102 with the middle finger of his right hand while touching the selection area 128L1 (object selection area 162) with the thumb of his right hand. Touching the selection area 128L1 switches to the penetration mode. When the selection area 128L1 is touched, the display update unit 122 makes the code window X1 invisible (completely transparent). The data input unit 118 detects the touch at the coordinates (849,581) as a penetration input to the code window X1. The data input unit 118 detects the touch at the coordinates P3 as a dot touching operation to the execution window X2.

[0057] FIG. 13 is a second screen view showing the state during normal input into the code window X1. When the user removes (detaches) the thumb of his / her right hand from the selection area 128L1, the display update unit 122 redisplays the code window X1. Because the code window X1 is displayed semi-transparently, the user can see the typing point P3 in the execution window X2 behind it through the code window X1. The display update unit 122 displays the chord menu 136 near the coordinate P3 (849,581) in response to the immediately preceding typing operation. The chord menu 136 is part of the code window X1, and includes a number of buttons for creating chords.

[0058] Various codes can be created using the buttons in the code menu 136. Let's say the user wants to execute code to draw a circle. The user first touches the start point button 135 to register coordinates P3. Similarly, the user touches the second point coordinates P4 in the execution window X2 by penetrating input. After completing the above inputs, the user touches the circle drawing button 138. At this time, the coordinates P3 become the center coordinates of the circle, and the distance from the coordinates P3 to the coordinates P4 becomes the radius of the circle.

[0059] FIG. 14 is a third screen diagram showing the state during normal input into the code window X1. The drawing color of the program code input by the button of the circle drawing 138 may be selected in advance by the user from a color selection palette (not shown), or may be a color that is designated in advance as an initial setting. Here, it is assumed that "radius: 100" and "color: red" are designated. The display update unit 122 displays the code "drawcircle (849, 581, 100, color.red)" in the code window X1. This "drawcircle" is a code for instructing the drawing of a circle, and is associated in advance with the circle drawing button 138. The first argument indicates the X coordinate of the center point, and the second argument indicates the Y coordinate of the center point. In this embodiment, the coordinates (849, 581) of the point P3 in FIG. 12 are directly substituted as the coordinates of the center point. The third argument "100" indicates the radius. The fourth argument "color.red" indicates that the circle is drawn in "red". In this way, the user can create code by selecting a button from the code menu 136. Alternatively, the user can directly enter the code "drawcircle" into the code window X1.

[0060] FIG. 15 is a fourth screen diagram showing a state when a circle is drawn in the execution window X2. When the code "drawcircle" is input, the display control unit 120 draws the specified circle 140 in the execution window X2. The circle 140 is a red circle with a radius of 100 centered on the point P3 (849, 581) as instructed by the code "drawcircle (849, 581, 100, color.red)". In FIG. 15, the code window X1 is hidden, but the code window X1 may remain semi-transparent. In this way, the user can set the hitting point P3 (penetrating input to the execution window X2) and input the code based on the hitting point P3 (normal input to the code window X1) while visually checking the code window X1 and the execution window X2, and then check the execution result of the code in the execution window X2. According to such a control method, a user interface that makes it easy to check the connection between the code and its execution result can be provided.

[0061] Suppose the user wants to create a program that draws a circle. In this case, the center coordinates of the circle may be directly input as an argument for the code "drawcircle". However, for a beginner programmer, the task of finding out what arguments to input for each code is a heavy burden. Therefore, in this embodiment, the user is made to directly specify "the point (point P) to be the center of the circle" by touching the execution window X2 that shows the execution result of the program. After pointing points P3 and P4 in the execution window X2, support tools such as the circle drawing button 138 are provided, and the user is made to select the circle drawing button 138. As a result, the code "drawcirc le(849,581,100,color.red)" is displayed in the code window X1.

[0062] The code "drawcircle(849,581,100,color.red)" is generated semi-automatically, so that the user can intuitively understand the common programming principle that to draw a circle, all one needs to do is input the code (command) "drawcircle" and specify the coordinates of the center point, radius, and color as arguments. After inputting the code "drawcircle", a circle is actually drawn in the execution window X2 according to this code, so the user can confirm that the circle he or she imagined is drawn by the code. By maintaining the visibility and ease of input in both the code window X1 and the execution window X2, it becomes easier for the user to be aware of the interrelationship between the two windows. This control method allows the user to recognize the relationship between "code" and "execution result", in other words, "how to write code results in what kind of output."

[0063] FIG. 16 is a diagram showing the screen when the explanation of the function input is displayed. The user sets the pierce mode by touching the objectification selection area 162 (selection areas 128L1, 128R1), and sets the gesture mode by touching the gesture activation area 164 (selection areas 128L2, 128R2). When a predetermined time, for example, 5 seconds, has elapsed without the user inputting data while continuing to touch the selection area 128L1, the display update unit 122 displays the explanation area 142 for explaining the pierce input function that is activated by the selection area 128L1. The same applies to the selection area 128R1. When the touch continues on the gesture activation area 164 (selection areas 128L2, 128R2), the display update unit 122 displays the explanation area 142 regarding the gesture input.

[0064] When there are many selection areas 128 or many types of selection areas 128, the user may forget or find it difficult to understand what function each selection area 128 provides, even though the user recognizes that invisible selection areas 128 exist in the four corners of the monitor screen 102. By continuing to touch an invisible selection area 128 at the edge of the monitor screen 102 for a while, an explanation area 142 is displayed, making it easier for the user to easily check the functions of the selection areas 128, etc.

[0065] When the selection area 166 is touched, the display update unit 122 may immediately display the explanation area 142. However, in this case, the explanation area 142 is displayed every time the user touches the selection area 166 on the monitor screen 102, so it is considered more preferable to display the explanation area 142 on the condition that the selection area 166 is continuously touched for a predetermined time. Also, the explanation area 142 may be displayed on the condition that the user makes another input while touching the selection area 166. For example, the explanation area 142 may be displayed when the user speaks a predetermined word, for example, "explanation," after touching the selection area 128L1.

[0066] FIG. 17 is a diagram showing a screen when a gesture is input. The user can activate the gesture area G1 by touching the gesture activation area 164 (selection areas 128L2, 128R2). In FIG. 17, the user touches the selection area 128L2 with the thumb of the left hand. At this time, the layer management unit 124 activates the gesture area G1 and transitions from the normal mode to the gesture mode. The gesture area G1 remains invisible even after activation. The display update unit 122 may change the display mode, such as by changing the color of the entire screen, so that the user can easily understand that the mode is the gesture mode. Alternatively, an icon indicating that the mode is the gesture mode may be displayed in a predetermined area of ​​the monitor screen 102. The gesture input unit 116 detects all inputs during the gesture mode as gesture inputs.

[0067] The gesture input unit 116 determines whether or not the movement of the user's fingers on the monitor screen 102 corresponds to any of the gesture patterns registered in advance. If the movement corresponds to a gesture pattern, a process according to the gesture pattern is executed on the code window X1 (focus window). For example, as shown in FIG. 17, when the user slides two fingers up and down at the same time, the display update unit 122 scrolls the code window X1 up and down. The display update unit 122 may be configured to be able to scroll the code window X1 up and down faster than usual by gesture input.

[0068] In addition, the following gesture patterns are registered. When the user taps the monitor screen 102 with one finger, a context menu can be displayed at the tapped location in the code window X1. A context menu is generally displayed when the user right-clicks the mouse, and displays a list of various operation options. The user can select a menu to be executed from the context menu. When the user double-tap the monitor screen 102 with two fingers, the display update unit 122 enlarges or reduces the display of the code window X1. When the user pinches the monitor screen 102 with two fingers, the display update unit 122 also enlarges or reduces the display of the code window X1. When the user slides two fingers left and right, the display update unit 122 changes the page of the code window X1. When the user double-tap the monitor screen 102 with three fingers, the display update unit 122 enlarges the code window X1. The above are merely examples, and various other gesture patterns may be registered. For example, various gesture patterns can be defined according to the number or combination of fingers used, or according to circular movement, linear movement, polygonal movement (e.g., N-shaped movement, W-shaped movement), etc.

[0069] If the gesture mode is set, various gestures can be input using the entire monitor screen 102. The user can perform various operations on the code window X1 by touching the selection area 128L2 with the left hand that is holding the tablet terminal 100, and drawing a large gesture with the finger of the right hand on the entire monitor screen 102.

[0070] The above describes the tablet terminal 100 based on the embodiment. According to the tablet terminal 100 of the present embodiment, when the code window X1 and the execution window X2 are displayed in an overlapping manner, data can be input into the execution window X2 on the rear side without performing a bring-to-front operation for switching the anteroposterior relationship between the code window X1 and the execution window X2.

[0071] In the past, when the code window X1 was displayed in the foreground and the execution window X2 in the background, the execution window X2 had to be brought to the foreground in order to input data into the execution window X2. The foreground operation brings the execution window X2 to the foreground and the code window X1 to the background. To bring the window to the foreground, a touch panel, mouse, or other device had to be pressed and released. If using a mouse, the operation would involve clicking the "close button" or "shrink button" of the code window X1. In this case, a series of operations would be required: move the mouse pointer to the position of the "close button," press the left mouse button, and release the mouse. When performing the foreground operation, the user would be aware of the layer swapping operation of "moving the obtrusive code window X1 in the foreground to bring the execution window X2 in the background to the foreground." The foreground operation also works the same way when using shortcut keys to swap the layers of windows.

[0072] On the other hand, in the tablet terminal 100 of this embodiment, the user can objectify the execution window X2 simply by touching the objectification selection area 162. For example, the user can touch the objectification selection area 162 with the left hand holding the tablet terminal 100 and input data into the execution window X2 on the back with the right hand. When the user wants to input data into the code window X1, he or she just needs to remove their finger from the objectification selection area 162. Since the focus window is selected with the left hand and data is input with the right hand, the user can easily continue working without feeling interrupted even when the focus window is changed. In particular, when working while frequently going back and forth between the code window X1 and the execution window X2, the user only needs to repeatedly touch and detach the objectification selection area 162 with the free hand, which significantly improves operability when changing the focus window.

[0073] Furthermore, when the focus window is changed, the context between the code window X1 and the execution window X2 is maintained, so the user does not need to be aware of changes in the layer structure. In other words, when there are many windows, even if the focus window is switched one after another, there is no need to worry about "which window has moved to which layer." In this embodiment, the visibility of the execution window X2 (focus window) is further improved by hiding the code window X1 when the object selection area 162 is touched.

[0074] In normal mode, the code window X1 is displayed semi-transparently, so that the code window X1 and the execution window X2 can be viewed simultaneously. There is no need to place the code window X1 and the execution window X2 side by side, or to move or close the code window X1 in order to view the execution window X2.

[0075] When the user continues to touch a selection area 166 or the like, an explanation area 142 is displayed. Even if the function judgment area M1 is not visible, the user understands that a function can be selected and input by touching near the edge of the monitor screen 102. Since the explanation area 142 is displayed by continuing to touch the edge of the monitor screen 102, the user can easily understand what function can be selected and input by touching where.

[0076] As another user assistance, the display update unit 122 may display information indicating the available selection areas or the role of each selection area. This information display may be displayed by lighting each selection area together with its function name. For example, when a certain time has passed since the user inputs a certain function selection input without inputting the next function, in other words, when a certain time has passed since the appearance of the explanation area 142, the display update unit 122 may activate such an assistance function.

[0077] A user can activate gesture area G1 by touching gesture activation area 164 (selection areas 128L2, 128R2). When gesture area G1 is activated, the user can input various gestures by moving his / her finger over the entire monitor screen 102. Since gesture input can be performed using a large area of ​​the monitor screen 102, high operability can be achieved even when the monitor screen 102 is small. Furthermore, since gesture input and data input can be switched by touching and detaching the gesture activation area 164, data input will not be performed erroneously while performing gesture input. Similarly, an operation for data input will not be erroneously recognized as gesture input.

[0078] As described above, according to this embodiment, for example, a right-handed user can perform data input work with his / her right hand and use his / her free left hand to change the mode as appropriate.

[0079] The present invention is not limited to the above-mentioned embodiment and modified examples, and the components can be modified without departing from the gist of the present invention. Various inventions can be formed by appropriately combining multiple components disclosed in the above-mentioned embodiment and modified examples. In addition, some components can be deleted from all the components shown in the above-mentioned embodiment and modified examples.

[0080] Although the learning software has been described as being executed by one tablet terminal 100, some of the functions of the learning software or the tablet terminal 100 may be realized by another device such as a server.

[0081] [Variations] FIG. 18 is a conceptual diagram for explaining focus control of three or more windows. 18, six layers are formed by four windows W1 to W4 in addition to a function judgment area M1 and a gesture area G1. A gesture validation area 164 is set in the upper left portion of the function judgment area M1, and four object selection areas 162 are set at the bottom, namely, a W1 selection area 150, a W2 selection area 152, a W3 selection area 154, and a W4 selection area 156. When the gesture validation area 164 is touched, the gesture area G1 is validated.

[0082] Window W1 corresponds to W1 selection area 150, window W2 corresponds to W2 selection area 152, window W3 corresponds to W3 selection area 154, and window W4 corresponds to W4 selection area 156. When the user touches W1 selection area 150, he or she can set window W1 as the focus window. Similarly, when the user touches W2 selection area 152, he or she can set window W2 as the focus window. The same applies to W3 selection area 154 and W4 selection area 156.

[0083] Even if the target window is changed, the layer management unit 124 does not change the layers. Therefore, the anteroposterior relationship of the windows W1 to W4 is maintained. For example, suppose that window W1 is a web screen, window W2 is an editing screen of word processing software, window W3 is an editing screen of graphic editing software, and window W4 is an editing screen of spreadsheet software. When the user works while switching between the four software programs at any time, the user can simply switch the focus window at any time using the objectification selection area 162. Since the anteroposterior relationship of the layers does not change, there is no need to worry about which layer the desired window is on, even when there are multiple windows.

[0084] It should be noted that the selected window may be set as the focus window under the condition that the selected window is partially or entirely covered by any of the windows in front of it among the windows W1 to W4. In other words, when the front of the window is not covered, the focus change by the function selection input may be treated as invalid. For example, assume that the window W3 is selected by the W3 selection area 154. When the window W3 is partially covered by the window W1 or window W2 in front of it, the layer management unit 124 sets the window W3 as the focus window. On the other hand, when the front of the window W3 is not covered because both the window W1 and the window W2 are closed, the window W3 may not be set as the focus window even if the W3 selection area 154 is touched.

[0085] In the present embodiment, the code window X1 is displayed semi-transparently so that the code window X1 and the execution window X2 can be simultaneously viewed. As a modified example, the code window X1 may be displayed normally (opaquely).

[0086] In this embodiment, the selection areas 166 are arranged at the four corners of the function judgment area M1 (monitor screen 102). As a modified example, a plurality of selection areas 166 may be arranged on the left side or bottom side of the function judgment area M1. Alternatively, the selection areas 166 may be arranged on any side, such as the top side or right side.

[0087] In this embodiment, the learning software is described as being executed on the tablet terminal 100, but the learning software may be executed on other mobile computers such as a smartphone or a laptop PC. Similarly, the learning software may be executed on a stationary computer such as a desktop PC.

[0088] In the present embodiment, the code window X1 is set as the focus window in the normal mode. As a modification, the execution window X2 on the back side may be set as the focus window in the normal mode, and the code window X1 on the front side may be changed to the focus window in the special mode.

[0089] The change to the penetration mode can be made by a method other than touching the object selection area 162. For example, the mode may be changed to the penetration mode by double-clicking the mouse or pressing a specific key on the keyboard. In addition, the mode may be changed to the penetration mode by voice input, or when the user focuses his / her gaze on a specific area of ​​the monitor screen 102. The mode may be changed to the penetration mode by using a foot device such as a pedal. Similarly, data may be input by touch, mouse, keyboard, voice, or the like.

[0090] In the present embodiment, the through mode is set only while the touch on the object selection area 162 continues. As a modified example, the layer management unit 124 may continue the through mode for a predetermined effective period, for example, three seconds, after the object selection area 162 is touched. If a through input is made during the effective period, the effective period may be extended for another three seconds. When no input continues for three seconds, the layer management unit 124 may return from the through mode to the normal mode.

[0091] In the present embodiment, the code window X1 in the foreground is described as being displayed semi-transparently. As a modified example, the display update unit 122 may display the code window X1 normally (opaquely). The display update unit 122 may change the display of the code window X1 to semi-transparent or transparent on the condition that the code window X1 and the execution window X2 overlap. Alternatively, when the code window X1 and the execution window X2 overlap, the display update unit 122 may set only the overlapping area of ​​the code window X1 to semi-transparent or transparent display.

[0092] In the present embodiment, the layer management unit 124 is described as not switching the front-back relationship between the code window X1 and the execution window X2. As a modified example, the layer management unit 124 may conditionally switch the front-back relationship between the code window X1 and the execution window X2. When a data input operation is performed on the exposed area of ​​the execution window X2, in other words, on the area of ​​the execution window X2 that is not covered by the code window X1, the layer management unit 124 may switch the front-back relationship between the execution window X2 and the code window X1. For example, when point P1 (exposed area 134) in FIG. 11 is touched, the layer management unit 124 may display the execution window X2 in front of the code window X1 and set the execution window X2 as the focus window.

[0093] In the through mode, when the user touches point P1 of the exposed area 134, the layer management unit 124 may maintain the anteroposterior relationship between the code window X1 and the execution window X2 as is. When the user touches the exposed area 134 of the execution window X2 (back), the anteroposterior relationship may be maintained in the through mode, and the execution window X2 may be moved to the front in the normal mode.

[0094] In the present embodiment, it has been described that the semi-transparent code window X1 is made invisible (completely transparent) when the through mode is set. As a modified example, both the code window X1 and the execution window X2 may be displayed normally. When the through mode is set, the display update unit 122 may increase the transparency of the code window X1 to improve the visibility of the execution window X2 (focus window).

[0095] Also, when the through mode is set, the transparency of the execution window X2 may be reduced. For example, assume that both the code window X1 and the execution window X2 are displayed semi-transparently. When the through mode is selected, the transparency of the execution window X2, which is the focus window, may be reduced to increase the visibility of the execution window X2 (focus window).

[0096] The same applies to the case where three or more windows are provided as shown in Fig. 18. As an example, all of windows W1 to W4 are displayed semi-transparently. When the user focuses on window W2, the transparency of window W2 can be reduced so that the user can easily recognize that window W2 is the focused window. At this time, the transparency of window W1 in the foreground may be increased.

[0097] The window on the rear side may be a window equipped with an input interface such as a button, for example, a dialog box or a software keyboard. When the user wants to operate the dialog box, he or she only has to touch the target selection area 162 (selection areas 128L1, 128R1). At this time, the user may operate the window on the rear side as an operation console by penetrating input, and then return to the main window on the front side and operate it by normal input.

[0098] In this embodiment, the mode is switched to the through mode by touching the object selection area 162, and data input operation is performed in this state. As a modified example, it is possible to dynamically select whether the code window X1 or the execution window X2 is to be the focus window depending on the setting when the data input operation is performed.

[0099] As an example, the touch panel may have a function for detecting touch pressure. In this case, the layer management unit 124 may set the code window X1 as the focus window when the touch is light, and may set the execution window X2 as the focus window when the touch is strong. The operation detection unit 110 may also detect which finger is touching based on the area of ​​the touching finger. For example, the code window X1 may become the focus window when the index finger is touched, and the execution window X2 may be set as the focus window when the little finger is touched.

[0100] The operation detection unit 110 may distinguish between a touch by a finger and a touch by a stylus pen based on the difference in capacitance. In this case, the code window X1 may be set as the focus window when touched by a finger, and the execution window X2 may be set as the focus window when touched by a stylus pen.

[0101] A plurality of types of stylus pens may be prepared, and the operation detection unit 110 may identify each of them. A window may be associated with each of the plurality of stylus pens. In this case, data can be input to a plurality of windows by using a plurality of stylus pens appropriately. For example, software may be used to create a picture like an oil painting by overlapping a window for underpainting, a window for sketching, a window for rough sketching, and a window for finishing. In this case, the user may operate four semi-transparent windows simultaneously while changing the stylus pen (type of brush). Then, the picture may be completed by overlapping the four semi-transparent windows.

[0102] Gesture input may be enabled during the penetration mode. For example, when the user touches only the target selection area 162, the code window X1 is the target of gesture input. On the other hand, when the user touches the target selection area 162 and the gesture activation area 164 at the same time, the execution window X2 may be operated by the user's gesture input.

[0103] The penetrating input shown in this embodiment can be used in applications other than learning software. For example, two types of windows are prepared in a computer game. One is a game field window, and a menu window for selecting equipment is prepared behind it. In this case, the user may operate the menu window by penetrating input as necessary while targeting the window on the front side as the operation target. Also, multiple windows may be prepared corresponding to multiple game fields. The game may be played simultaneously in the front game field and the back game field, and the user may play the game by switching the focus between the two windows.

[0104] In SNS (Social Networking Service) software, a pop-up screen may appear while inputting data on the chat screen, and data input on the chat screen may be recognized as erroneous input on the pop-up screen. As a countermeasure against this, the chat screen may be set as the focus window in normal mode. Even if a pop-up screen is displayed in front of the chat screen, the chat screen on the back side remains the focus window, so that input on the pop-up screen during chatting can be prevented. When the user wants to change to input on the pop-up screen, he or she can touch the selection area 166 to set the pop-up screen as the focus window. Even in this case, the anteroposterior relationship between the pop-up screen and the chat screen is maintained. This control method can prevent the user from interpreting input on the chat screen as input on a pop-up screen that suddenly appeared. [Explanation of symbols]

[0105] 100 tablet terminal, 102 monitor screen, 104 user interface processing unit, 106 data processing unit, 108 data storage unit, 110 operation detection unit, 112 output unit, 114 function input unit, 116 gesture input unit, 118 data input unit, 120 display control unit, 122 display update unit, 124 layer management unit, 126 communication unit, 128L1 selection area, 128L2 selection area, 128R1 selection area, 128R2 selection area, 132 superimposition area, 134 exposure area, 135 start point button, 136 code menu, 138 circle drawing button, 140 circle, 142 description area, 150 W1 selection area, 152 W2 selection area, 154 W3 selection area, 156 W4 selection area, 162 objectification selection area, 164 gesture activation area, 166 Selection area, 300 processor, 302 first bus, 304 memory, 308 bridge, 310 second bus, 312 storage, 314 input device, 316 output device, 318 peripheral device, G1 gesture area, M1 function determination area, X1 code window, X2 execution window

Claims

1. a display control unit that places a first area in which a first image is to be displayed in front and a second area in which a second image is to be displayed in back of the first area; an operation detection unit that detects a first operation for designating the second area and a second operation for inputting data from a user; The information processing device is characterized in that, when the second operation is detected after the first operation is detected, the display control unit changes the second image based on the second operation while maintaining the first area in front of the second area.

2. The information processing device according to claim 1, characterized in that after detecting the first operation, when the second operation is detected within an overlapping area of ​​the first area and the second area, the display control unit determines that the second operation is data input into the second area.

3. The information processing device described in claim 1 or 2, characterized in that when the second operation is detected after the first operation is detected, the display control unit changes the second image based on the second operation, and further changes the first image in accordance with the change in the second image, thereby displaying the changes in both the first image and the second image in a visually noticeable manner.

4. The information processing device according to any one of claims 1 to 3, characterized in that when the second operation is detected within an exposed area of ​​the second area where the first area is not overlapped, the display control unit changes the display of the second area to be in front of the first area.

5. The information processing device according to claim 4 , characterized in that when the second operation is detected in the exposed area after the first operation is detected, the display control unit maintains a posterior relationship between the first area and the second area.

6. The information processing device according to any one of claims 1 to 5, characterized in that the display control unit sets a plurality of areas on each of a plurality of display layers, and sets any one of the areas to be selectable as a target for data input in accordance with a selection instruction from a user as the first operation.

7. the operation detection unit detects, when a touch by a user is detected in any of a plurality of selection areas formed on a screen, the touch as the first operation; The information processing apparatus according to claim 6 , wherein the display control unit sets an area corresponding to the selection area that is the target of the first operation as an area that is the target of data input.

8. The information processing device according to claim 6 or 7, characterized in that the display control unit sets the selected area as an area to be targeted for data input, on condition that all or part of the selected area is obscured by an area in front.

9. The information processing apparatus according to claim 1 , wherein the display control unit further increases a transparency of the first area when the first operation is detected.

10. The information processing apparatus according to claim 1 , wherein the display control unit further reduces a transparency of the second area when the first operation is detected.

11. The information processing device according to claim 1, characterized in that when the second operation is detected while the first operation is continuing, the display control unit changes the second image in accordance with the second operation, and when the first operation is ended, changes the first image in accordance with the second operation.

12. a display control unit that arranges a first region and arranges a second region behind the first region; an operation detection unit that detects a second operation for inputting data from a user, When the second operation is detected in an overlapping area of ​​the first area and the second area, the display control unit performs the following while maintaining the display states of the first area and the second area: an information processing apparatus that selects either the first area or the second area as an image area to be a target for data input by the second operation, depending on a setting at the time of detection;

13. A function of arranging a first area in which a first image is to be displayed; A function of placing a second area in which a second image is to be displayed behind the first area; A function of detecting a first operation for designating the second area; detecting a second operation for inputting data from a user; A computer program characterized by causing a computer to perform a function of changing the second image based on the second operation while maintaining the first area in front of the second area when the second operation is detected after the first operation is detected.

14. locating a first area in which a first image is to be displayed; placing a second area, in which a second image is to be displayed, behind the first area; detecting a first operation that designates the second area; detecting a second operation for inputting data from a user; An information processing method comprising a step of, when the second operation is detected after the first operation is detected, changing the second image based on the second operation while maintaining the first area in front of the second area.

Images