Information processing device, computer program, and information processing method

Abstract

To improve operability in a multi-window user interface.SOLUTION: A tablet terminal 100 arranges a code window for displaying a programming code on the front, and arranges an execution window for displaying a program execution screen on the back of the code window. When performing data input after a user instructs a focus change in a function determination area, the data input to the execution window is allowed while keeping the code window on the front of the execution window.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to multi-window user interface technology.

Background Art

[0002] With the emergence of mobile computers such as laptop PCs, tablet PCs, and smartphones, the opportunities to use computers have increased dramatically. Mobile computers are required to be lightweight and compact. Therefore, in the user interface design of mobile computers, it is an important point how much information can be efficiently input and output on a monitor screen of limited size.

[0003] Generally, a multi-window user interface that simultaneously displays multiple pieces of information in a plurality of windows (areas for image display) is often adopted. A "layer (display layer)" is set for each window. A plurality of windows are displayed such that all or part of the window on the back side (lower layer) is hidden behind the window on the front side (upper layer). The user inputs data with respect to the front window. Hereinafter, the window that is the target of data input is referred to as a "focus window".

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When the window into which you want to input information is in the background, you need to change the display order so that this window becomes the front layer (hereinafter referred to as "bringing to the foreground"). Specifically, you bring the window in the background to the foreground by minimizing (deactivating) the window that is in the foreground. Alternatively, you may perform an operation to display minimized versions of multiple windows side by side, and then select the focus window (the window into which you want to input information) from among them. The operation to bring a window to the foreground, or in other words, the extra steps required to bring a window to the foreground, can easily lead to a decrease in work efficiency when you want to input information while frequently changing the focus window.

[0006] The object of the present invention is to provide a technology for improving operability in a multi-window type user interface. [Means for solving the problem]

[0007] An information processing device in one aspect of the present invention includes a display control unit which has a first area for displaying a first image on its front and a second area for displaying a second image on its back, and an operation detection unit which detects a second operation from a user for data input. When the display control unit detects a second operation after detecting a first operation, it changes the second image based on the second operation while keeping the first area in front of the second area. [Effects of the Invention]

[0008] According to the present invention, it becomes easier to improve the operability of a multi-window type user interface. [Brief explanation of the drawing]

[0009] [Figure 1] This is an external view of a tablet device. [Figure 2] This is a schematic diagram illustrating the layer structure of multiple windows. [Figure 3] This is a hardware configuration diagram of a tablet device. [Figure 4] This is a functional block diagram of a tablet device. [Figure 5] This is an external view illustrating the configuration of the functional determination area. [Figure 6] This is a schematic diagram to explain normal input. [Figure 7] This is a schematic diagram to explain through-input. [Figure 8] This flowchart shows the processing steps when a touch is detected in the targeting selection area. [Figure 9] This flowchart shows the processing steps when a touch outside the selected area is detected. [Figure 10] This flowchart shows the processing steps when a detachment is detected. [Figure 11] This is an external view diagram illustrating the switching of data input targets. [Figure 12] This is the first screen diagram showing the state when input is passed through to the execution window. [Figure 13] This is the second screen diagram, showing the state when normal input is being made into the code window. [Figure 14] This is the third screen diagram, showing the state when normal input is being made into the code window. [Figure 15] This is the fourth screen diagram showing the state when a circle is drawn in the execution window. [Figure 16] This is a screenshot showing the explanation of the function input. [Figure 17] This is a screenshot of the screen when using gesture input. [Figure 18] This is a conceptual diagram illustrating focus control for three or more windows. [Modes for carrying out the invention]

[0010] In this embodiment, multiple windows are displayed arranged on multiple layers. A window is an image area that mimics a single sheet, and a layer is a concept that indicates the overlapping position of sheets when multiple sheets are stacked on top of each other.

[0011] First, let's assume physical paper sheets 1, 2, and 3. Information such as pictures and characters is described on each of the sheets 1, 2, and 3. Let's stack these three sheets in order. Suppose sheet 1 is placed on the top, sheet 2 in the middle, and sheet 3 at the bottom. At this time, the user can see all of sheet 1 on the top. Only the part of the second sheet 2 from the top that does not overlap with sheet 1 is visible. Only the part of the third sheet 3 from the top that does not overlap with either sheet 1 or 2 is visible. What realizes visibility similar to such stacked sheets by a computer program is a multi-window system.

[0012] In a multi-window system, similarly assume electronic windows 1, 2, and 3. Windows 1, 2, and 3 display information in the same way as sheets. At this time, the "top" layer corresponding to sheet 1 is called the "front layer". "Front" is an expression meaning (appearing as if it is) on the front side as seen from the user looking at the monitor. Try placing the three windows on each of the three layers. Suppose window 1 is placed on the top layer, window 2 on the middle layer, and window 3 on the bottom layer. In this case, it is expressed that window 1 is placed on a layer "in front" of windows 2 and 3. Similarly, window 2 is placed on the "back" layer of window 1 and on the "front" layer of window 3. Window 3 is expressed as being placed on the "back" layer.

[0013] The method of placing each of a plurality of windows such as windows 1, 2, and 3 on each of a plurality of layers is called the "display order". According to the above example, it is expressed as "the display order of the plurality of windows is windows 1, 2, and 3 in order from the front". Also, changing the layer of a window along with changing the display order is called "arrangement change" of the window.

[0014] In the following, "bringing to the foreground" means moving the layer of a specified window to the foreground in a multi-window user interface. "Targeting" means setting a specified window as the target of data input, in other words, setting a specified window as the focus window. "Data input" here means adding or modifying the content displayed by the window, specifically inputting text or images to change them. "Activating" means turning on a special functional layer. In a typical multi-window system, "bringing to the foreground" and "targeting" are essentially the same concept, but in this embodiment, "targeting without bringing to the foreground" is achieved.

[0015] Figure 1 is an external view of the tablet device 100. The tablet terminal 100 (information processing device) is a thin mobile computer equipped with a monitor screen 102. A touch panel is installed on the monitor screen 102. In this embodiment, the explanation will be based on the assumption of a class in which programming is taught to users. The students become the users of the tablet terminal 100.

[0016] The tablet device 100 comes pre-installed with application software for programming education (hereinafter referred to as "learning software"). The purpose of the lesson is to deepen the user's understanding of how software works by having the user write simple program code (hereinafter simply referred to as "code") and run the program they have created. As will be explained in more detail later, the window for creating the code and the window for running the program are displayed separately as a multi-window.

[0017] The following explanation assumes a touch panel, but the tablet device 100 can also be connected to input devices such as a keyboard or mouse. The tablet device 100 may also be capable of voice input.

[0018] Figure 2 is a schematic diagram illustrating the layer structure of multiple windows. The learning software includes a "normal layer" for data input and output, consisting of a code window X1 (first area) and an execution window X2 (second area). It also includes a "functional layer" for input assistance to the code window X1 and execution window X2, consisting of a function determination area M1 and a gesture area G1. Both functional layers are invisible. From front to back, the four layers are arranged as follows: function determination area M1, gesture area G1, code window X1, and execution window X2. The code window X1 and execution window X2 are the windows that constitute the learning software's workspace. Normally, the code window X1 is the focused window. The function determination area M1 is always enabled, while the gesture area G1 is normally disabled.

[0019] The function determination area M1 is located in the foreground. The user can change the execution window X2 to the focus window by accessing a selection area formed within a part of the function determination area M1 (details below). The user can also activate the gesture area G1 by accessing another selection area formed within a part of the function determination area M1. In the following, the state in which the code window X1 is the focus window will be called "normal mode," and the state in which the execution window X2, whose display area is obstructed because it is located in a layer below the code window X1, becomes the focus window will be called "penetration mode (special mode)." The state in which the gesture area G1 is activated will be called "gesture mode." There are three modes in the learning software, and they are mutually exclusive.

[0020] The gesture area G1 is located behind the function determination area M1. The gesture area G1 is invisible and is normally not enabled. When the gesture area G1 is enabled using the method described later, the user can perform various gesture inputs using the entire monitor screen 102. Details of gesture input will be described later in relation to Figure 17.

[0021] The code window X1 is displayed semi-transparently and provides a screen (first image) for writing code. Semi-transparent display here means that the image of the code window X1 is visible, and the transparency of the code window X1 is adjusted so that the image behind it is also visible. The code window X1 is located behind the gesture area G1, but is in the foreground in normal layers. In normal mode or gesture mode (when both the code window X1 and the execution window X2 are displayed), the code window X1 becomes the focus window.

[0022] The execution window X2 is displayed opaquely (normally displayed), just like any other window. The execution window X2 provides the execution screen (second image) of the program created in the code window X1. The execution window X2 is located behind the code window X1. Because the code window X1 is semi-transparent, the user can see not only the code window X1 but also the execution window X2 behind it simultaneously. When the mode is changed to pass-through mode, or when the visible layer in front of the execution window X2, such as window X1, is not visible, the execution window X2 becomes the focus window.

[0023] In summary, it is as follows: 1. Functional layer: Invisible 1-1. Functional determination area M1: Always active. Accepts input for mode changes. 1-2. Gesture area G1: Inactive in normal mode and penetration mode, and enabled by functional determination area M1 (gesture mode). Accepts gesture input. 2. Normal layer: Visible 2-1. Code window X1: Semi-transparent and targeted in normal mode. May be temporarily made completely transparent (invisible) as described later. 2-2. Execution window X2: Opaque and targeted in penetration mode.

[0024] The display order of the code window X1 and the execution window X2, in other words, the relative positions of the layers, may be interchangeable. However, unless otherwise specified, the display order (front-to-back relationship) of the function determination area M1, gesture area G1, code window X1, and execution window X2 will remain unchanged in the following explanation. Furthermore, while it is possible to close the code window X1 and the execution window X2, unless otherwise specified, the following explanation will assume that the code window X1 and the execution window X2 are not closed and are both visible in normal mode.

[0025] Figure 3 is a hardware configuration diagram of the tablet device 100. The tablet terminal 100 includes a storage device 312 as non-volatile memory for storing computer programs, a volatile memory 304 for expanding programs and data, registers, an arithmetic unit, an instruction decoder, etc., and a processor 300 (CPU: Central Processing Unit) that reads and executes programs from the memory 304. The processor 300 is connected to a relatively high-speed first bus 302. In addition to the memory 304, a NIC (Network Interface Card) is connected to the first bus 302. Other devices such as a GPU (Graphics Processing Unit) may also be connected to the first bus 302.

[0026] The first bus 302 is connected to the relatively slower second bus 310 via a bridge 308. The second bus 310 is connected to storage 312, as well as output devices 316 such as a monitor screen 102 or speakers. Input devices 314 such as a mouse or keyboard, and peripheral devices 318 such as a printer may also be connected to the second bus 310.

[0027] Figure 4 is a functional block diagram of the tablet device 100. As described above, each component of the tablet terminal 100 is realized by hardware including arithmetic units such as a CPU and various coprocessors, memory and storage devices, and wired or wireless communication lines connecting them, and software stored in the storage devices that supplies processing instructions to the arithmetic units. The computer program may consist of device drivers, an operating system, various application programs located at a higher layer, and libraries that provide common functions to these programs. The blocks described below represent functional units, not hardware units.

[0028] The tablet terminal 100 includes a user interface processing unit 104, a data processing unit 106, a communication unit 126, and a data storage unit 108. The user interface processing unit 104 accepts user input via the touch panel and is responsible for processing related to the user interface, such as displaying images and outputting sound. The communication unit 126 is responsible for communication processing with other devices via the internet. The data storage unit 108 stores various types of data. The data processing unit 106 executes various processes based on the data acquired by the user interface processing unit 104 and the communication unit 126, and the data stored in the data storage unit 108. The data processing unit 106 also functions as an interface for the user interface processing unit 104, the communication unit 126, and the data storage unit 108.

[0029] The user interface processing unit 104 includes an operation detection unit 110 that receives input from the user and an output unit 112 that outputs various information such as images and sounds to the user. The operation detection unit 110 includes a function input unit 114, a gesture input unit 116, and a data input unit 118. The function input unit 114 detects a function selection input (first operation), which is a predetermined input to the function determination area M1. A function selection input is an input to control the function or state of the window itself, rather than the content of the window, such as activation or targeting, but the details will be described later in relation to Figure 5, etc. The gesture input unit 116 detects a gesture input, which is a predetermined input to the gesture area G1. The data input unit 118 detects a data input (second operation) to the code window X1 or the execution window X2.

[0030] The data storage unit 108 stores not only the learning software but also programs created by the user.

[0031] The data processing unit 106 includes a display control unit 120 that controls the display of the code window X1 and the execution window X2. The display control unit 120 includes a display update unit 122 and a layer management unit 124. The display update unit 122 manages image generation and updating in the code window X1 and the execution window X2. The layer management unit 124 manages the layers, position, and size of each window. The layer management unit 124 also manages modes related to normal mode, special mode, and gesture mode.

[0032] Figure 5 is an external view illustrating the configuration of the functional determination area M1. In the invisible function determination area M1, selection areas 128L1, 128R1, 128L2, and 128R2 (hereinafter collectively referred to as "selection area 166") are set at the four corners of the monitor screen 102.

[0033] When the user touches selection area 128L1 or selection area 128R1, in other words, when the user touches the lower left corner or lower right corner of the monitor screen 102, the layer management unit 124 switches from normal mode to penetration mode (special mode). At this time, the execution window X2 on the back side is targeted. Hereinafter, the two selection areas 128L1 and 128R1 of the selection area 166 used to target the execution window X2 will be collectively referred to as the "targeted selection area 162".

[0034] When the user stops touching (detaches) the target selection area 162, the layer management unit 124 returns from through mode to normal mode. At this time, the front code window X1 is targeted again.

[0035] When the user touches selection area 128L2 or selection area 128R2, in other words, when the user touches the upper left corner or upper right corner of the monitor screen 102, the layer management unit 124 switches from normal mode to gesture mode. The layer management unit 124 activates gesture area G1, enabling gesture input. Hereinafter, when referring to the selection areas 128L2 and 128R2 of the selection area 166 that activate gesture area G1, they will be collectively referred to as the "gesture activation area 164".

[0036] When the user stops touching the gesture activation area 164 (detaches), the layer management unit 124 switches from gesture mode to normal mode. At this time, the gesture area G1 is disabled. Touching any of the four selection areas 166 is called "function selection input".

[0037] In this embodiment, there are two types of data input: "normal input" in normal mode and "penetration input (special input)" in penetration mode. Data input (primarily text or image input) can be performed by touch input using a finger or stylus within the window area, but other input devices such as a mouse can also be used. In normal mode, the code window X1, which is in front of the execution window X2, is the focus window, and the user performs data input targeting the code window X1. This is called "normal input".

[0038] When the user touches the target selection area 162, the system enters pass-through mode, and the execution window X2 on the back side becomes the focus window. The user then inputs data targeting the execution window X2. This is called "pass-through input".

[0039] During through-in input, the layer management unit 124 does not change the relative positions of the code window X1 and the execution window X2. In other words, during through-in input, even though the code window X1 is in the foreground, a unique user experience is provided, as if you were directly touching the execution window X2 behind the code window X1. This user experience will be further explained in relation to Figures 6 and 7.

[0040] Figure 6 is a schematic diagram illustrating a typical input. As mentioned above, the layers visible to the user are typically two layers: the code window X1 and the execution window X2. The code window X1 is in the foreground, and the execution window X2 is in the background. Because the code window X1 is semi-transparent, the user can see not only the code window X1 but also the execution window X2 in the background simultaneously. Therefore, the user can write and verify program code while simultaneously checking the execution status of the program.

[0041] In normal mode, the code window X1 is the focus window. When the user touches the monitor screen 102, the data input unit 118 determines this as data input (normal input) to the code window X1 (focus window). The user inputs data to the code window X1 using touch operation. The display update unit 122 displays the entered data in the code window X1 or moves the cursor according to the input result. It can also move the code window or scroll the text area.

[0042] Figure 7 is a schematic diagram illustrating the through-input. When the user touches the target selection area 162 (selection area 128L1 or selection area 128R1), the system switches from normal mode to through mode. The layer management unit 124 sets the execution window X2 as the focus window instead of the code window X1. In this state, when the user touches the monitor screen 102, the data input unit 118 determines this to be data input (through input) to the execution window X2. The user inputs data to the execution window X2 by touching it. The display update unit 122 draws an image on the execution window X2 according to the input result. The display update unit 122 also updates the code window X1 in response to the image input result to the execution window X2.

[0043] In through-mode, the user has the sensation that their finger is piercing through the code window X1 in the foreground and manipulating the execution window X2 in the background. In through-mode, the display update unit 122 may temporarily hide (make completely transparent) the code window X1. In this case, the user can input data while only viewing the execution window X2. In the following explanation, we will assume that the front code window X1 is hidden in pass-through mode.

[0044] Figure 8 is a flowchart showing the processing steps when a touch is detected on the targeting selection region 162. The process shown in Figure 8 is executed when a touch is detected on the targeting selection area 162 on the monitor screen 102. First, when a touch is detected on the monitor screen 102, the function input unit 114 determines whether the touch is on the targeting selection area 162 (selection area 128L1, selection area 128R1) (the lower left corner or lower right corner of the monitor screen 102). If the touch is on the targeting selection area 162, the layer management unit 124 sets it to pass-through mode (S10). At this time, the execution window X2 becomes the focus window. The display update unit 122 sets the code window X1 to hidden (S12). In other words, when the targeting selection area 162 is touched and the system switches to pass-through mode, the execution window X2 becomes the focus window, and the code window X1, which is in front of the execution window X2, becomes completely transparent (hidden). The code window X1 becomes invisible, but it is not closed. Also, the code window X1 is not layered behind the execution window X2. The processing that occurs when a touch is detected in the gesture activation area 164 will be described later in relation to Figure 17.

[0045] Figure 9 is a flowchart showing the processing steps when a touch is detected outside of the selected area 166. When a touch is detected outside of the selection area 166, in normal mode (S20: normal mode), the data input unit 118 determines whether the touch point is in the area where the code window X1 and the execution window X2 overlap (hereinafter referred to as the "overlapping area") (S22). If the touch is in the overlapping area (Y in S22), the data input unit 118 accepts data input targeting the code window X1 (S24). In other words, when the overlapping area of ​​the code window X1 and the execution window X2 is touched, and the code window X1 (front side) is the focus window, normal input is performed.

[0046] When the touched point is not in the overlapping area (N in S22), the data input unit 118 accepts data input targeting either the code window X1 or the execution window X2 (S26). At this time, the layer management unit 124 temporarily treats the window accepting the touch operation as the focus window. For example, if only the code window X1 is present at the touch point, it is treated as normal input to the code window X1, and if only the execution window X2 is present, it is treated as normal input to the execution window X2. In either case, the layer order, with the code window X1 in the foreground and the execution window X2 in the background, is maintained.

[0047] When a touch is detected, if it is in pass-through mode (S20: pass-through mode), the data input unit 118 determines whether the touch point is in the overlapping area (S28). If the touch is in the overlapping area (Y in S28), the data input unit 118 accepts data input for the execution window X2 (S30). That is, pass-through input is performed when the overlapping area of ​​the code window X1 and the execution window X2 is touched, and the execution window X2 (back side) is the focus window.

[0048] When the touched point is not in the overlapping area (N in S28), the data input unit 118 accepts data input targeting the touched window among the code window X1 and the execution window X2 (S32). In this case as well, the layer management unit 124 temporarily treats the window accepting the touch operation as the focus window.

[0049] Figure 10 is a flowchart showing the processing steps when a detachment is detected. The process shown in Figure 10 is executed when a detachment (touch-off) from the monitor screen 102 is detected. First, in the case of a detachment during penetration mode (Y in S40), the display update unit 122 redisplays the code window X1 that was hidden in S12 (Figure 8) (S42). At this time, the code window X1 is displayed in its normal semi-transparent state. The layer management unit 124 is set to normal mode (S44). The code window X1 is targeted.

[0050] The display update unit 122 updates the data of the non-focused window according to the data input. For example, when data is input to the code window X1, the display update unit 122 changes the image of the execution window X2 according to the input content to the code window X1. Similarly, when data is input to the execution window X2, the display update unit 122 changes the image drawn in the code window X1 according to the input content to the execution window X2. The data update will be described in detail in relation to Figure 12 and later.

[0051] In the case of detachment during normal mode (N in S40), processing proceeds to S46.

[0052] Figure 11 is an external view illustrating the switching of data input targets. As with typical multi-window setups, users can freely change the position and size of the code window X1, etc. In Figure 11, the front code window X1 has been moved to the lower right. Meanwhile, the back execution window X2 is displayed across the entire monitor screen 102.

[0053] The superimposed area 132 indicates the area where the execution window X2 (back) is hidden by the code window X1 (front). The exposed area 134 indicates the area where the execution window X2 on the back is exposed when the code window X1 on the front moves to the lower right. When the user touches point P1 in the exposed area 134, the execution window X2 on the back, which is exposed, becomes the focus window (the window that is the target of data input), regardless of whether it is in pass-through mode or not. The layer management unit 124 temporarily treats the execution window X2 as the focus window when the exposed area 134 is touched.

[0054] When the user touches point P2 in the superimposed area 132, as described above, either the code window X1 or the execution window X2 becomes the focus window (the window targeted for data input), depending on whether it is in through mode or normal mode. If the execution window X2 is moved and the touch occurs in an area where only the code window X1 exists, the code window X1 becomes the target for data input.

[0055] Next, as a concrete example of a user interface, we will explain the process of creating code to draw a circle while using both through-input and normal input, with reference to Figures 12 to 15.

[0056] Figure 12 is the first screen view showing the state when input is passed through to the execution window X2. The user touches selection area 128L1 (targeted selection area 162) with their right thumb while touching the monitor screen 102 with their right middle finger. Touching selection area 128L1 activates penetration mode. When selection area 128L1 is touched, the display update unit 122 hides (makes completely transparent) the code window X1. The data input unit 118 detects a touch at coordinates (849, 581) as a penetration input to the code window X1. The data input unit 118 detects this touch at coordinates P3 as a point input operation to the execution window X2.

[0057] Figure 13 is a second screen diagram showing the state when normal input is made to the code window X1. When the user removes their right thumb from the selection area 128L1 (detaches), the display update unit 122 redisplays the code window X1. Since the code window X1 is displayed semi-transparently, the user can see the input point P3 of the execution window X2 behind it through the code window X1. The display update unit 122 displays the code menu 136 near coordinates P3 (849, 581) in response to the previous input operation. The code menu 136 is part of the code window X1 and includes several buttons for creating code.

[0058] Various codes can be created using the buttons in Code Menu 136. Let's say the user wants to execute a code to draw a circle. First, the user registers coordinates P3 by touching the starting point button 135. Similarly, in the execution window X2, the user touches the second point, coordinates P4, using through input. After making these inputs, the user touches the circle draw button 138. At this point, coordinates P3 become the center coordinates of the circle, and the distance from coordinates P3 to P4 becomes the radius of the circle.

[0059] Figure 14 is a third screen diagram showing the state when normal input is being made to the code window X1. The drawing color of the program code entered by the circle drawing button 138 can be selected by the user from a color selection palette (not shown) or from a color pre-specified as the default setting. Here, let's assume that "radius: 100" and "color: red" are specified. The display update unit 122 displays the code "drawcircle(849,581,100,color.red)" in the code window X1. This "drawcircle" is a code to instruct the drawing of a circle and is pre-associated with the circle drawing button 138. The first argument indicates the X coordinate of the center point, and the second argument indicates the Y coordinate of the center point. In this embodiment, the coordinates of point P3 in Figure 12 (849,581) are directly substituted as the coordinates of the center point. The third argument "100" indicates the radius. The fourth argument "color.red" indicates that the circle should be drawn in "red". In this way, the user can create code by selecting a button from the code menu 136. Alternatively, users can directly enter the code "drawcircle" into the code window X1.

[0060] Figure 15 is a fourth screen diagram showing the state when a circle is drawn in the execution window X2. When the code "drawcircle" is entered, the display control unit 120 draws the specified circle 140 in the execution window X2. Circle 140 is a red circle with a radius of 100 centered at point P3 (849,581), as instructed by the code "drawcircle(849,581,100,color.red)". In Figure 15, the code window X1 is hidden, but the code window X1 may remain semi-transparent. In this way, the user can view the code window X1 and the execution window X2, set the point P3 (through input to the execution window X2), input the code based on point P3 (normal input to the code window X1), and then check the execution result of the code in the execution window X2. This control method provides a user interface that makes it easy to check the connection between the code and its execution result.

[0061] Suppose a user wants to create a program that draws a circle. In this case, the user could directly input the coordinates of the circle's center as an argument to the code "drawcircle". However, for a programming novice, figuring out what arguments to input for each code is a significant burden. Therefore, in this embodiment, the user is allowed to directly specify the desired center point (point P) on the execution window X2, which displays the program's execution results, by touching it. After points P3 and P4 are placed on the execution window X2, the user is given support tools such as the circle drawing button 138 and prompted to select the circle drawing button 138. As a result, the code "drawcircle(849,581,100,color.red)" is displayed on the code window X1.

[0062] The user intuitively understands the basic programming concept that to draw a circle, one simply needs to input the code (command) "drawcircle(849,581,100,color.red)", and that the arguments for this command are the coordinates of the center point, the radius, and the color. After inputting the code "drawcircle", a circle is actually drawn in the execution window X2 according to this code, allowing the user to confirm that the circle is drawn exactly as imagined by the code. Maintaining the visibility and ease of input in both the code window X1 and the execution window X2 makes it easier for the user to be aware of the relationship between the two windows. This control method allows the user to recognize the relationship between "code" and "execution result", in other words, "what kind of output is produced when you write code in a certain way".

[0063] Figure 16 is a screenshot showing the explanation of the function input. The user sets the penetration mode by touching the target selection area 162 (selection areas 128L1, 128R1) and the gesture mode by touching the gesture activation area 164 (selection areas 128L2, 128R2). If the user continues to touch selection area 128L1 for a predetermined time, for example 5 seconds, without entering any data, the display update unit 122 displays an explanation area 142 to explain the penetration input function activated by selection area 128L1. The same applies to selection area 128R1. If the user continues to touch the gesture activation area 164 (selection areas 128L2, 128R2), the display update unit 122 displays an explanation area 142 regarding gesture input.

[0064] If there are many selection areas 128 or a large number of types, users may be aware that invisible selection areas 128 exist in the four corners of the monitor screen 102, but may forget or have difficulty understanding which selection area 128 provides what function. By touching and holding an invisible selection area 128 at the edge of the monitor screen 102 for a while, the explanation area 142 is displayed, making it easier for users to easily check the functions of the selection areas 128, etc.

[0065] When the selection area 166 is touched, the display update unit 122 may immediately display the explanation area 142. However, in this case, the explanation area 142 would be displayed every time the user touches the selection area 166 on the monitor screen 102, so it is considered more preferable to display the explanation area 142 only when the selection area 166 is touched continuously for a predetermined time. Alternatively, the explanation area 142 may be displayed only when the user makes another input while touching the selection area 166. For example, the explanation area 142 may be displayed when the user touches the selection area 128L1 and then utters a predetermined word, such as "explanation".

[0066] Figure 17 shows the screen when using gesture input. The user can activate the gesture area G1 by touching the gesture activation area 164 (selection areas 128L2, 128R2). In Figure 17, the user is touching the selection area 128L2 with their left thumb. At this time, the layer management unit 124 activates the gesture area G1 and switches from normal mode to gesture mode. The gesture area G1 remains invisible even after activation. The display update unit 122 may change the display manner, such as changing the color of the entire screen, to make it easier for the user to understand that it is in gesture mode. Alternatively, an icon indicating that it is in gesture mode may be displayed in a predetermined area of ​​the monitor screen 102. The gesture input unit 116 detects all inputs in gesture mode as gesture inputs.

[0067] The gesture input unit 116 determines whether the user's finger movements on the monitor screen 102 correspond to one of the pre-registered gesture patterns. If a gesture pattern is found, processing corresponding to the gesture pattern is executed on the code window X1 (focus window). For example, as shown in Figure 17, if the user slides two fingers up and down simultaneously, the display update unit 122 scrolls the code window X1 up and down. The display update unit 122 may also be able to scroll the code window X1 up and down faster than usual based on the gesture input.

[0068] In addition, the following gesture patterns are registered: • When a user taps the monitor screen 102 with one finger, a context menu can be displayed at the tapped location on the code window X1. The context menu is typically the menu displayed when right-clicking with a mouse, and it lists various operation options. The user can select the menu they want to execute from the context menu. • When a user double-tap the monitor screen 102 with two fingers, the display update unit 122 enlarges or reduces the code window X1. • When a user pinches the monitor screen 102 with two fingers, the display update unit 122 enlarges or reduces the code window X1. • When a user slides two fingers left or right, the display update unit 122 changes the page of the code window X1. • When a user double-tap the monitor screen 102 with three fingers, the display update unit 122 enlarges the code window X1. The above are examples, and various other gesture patterns may be registered. For example, a wide variety of gesture patterns can be defined depending on the number or combination of fingers used, or depending on circular movement, linear movement, polygonal movement (e.g., N-shaped movement, W-shaped movement), etc.

[0069] When set to gesture mode, various gesture inputs can be made using the entire monitor screen 102. The user can perform various operations on the code window X1 by touching the selection area 128L2 with their left hand, which is supporting the tablet device 100, while drawing large gestures with their right hand across the entire monitor screen 102.

[0070] The tablet terminal 100 has been described above based on the embodiment. According to the tablet terminal 100 of this embodiment, when the code window X1 and the execution window X2 are displayed superimposed, data can be entered into the execution window X2 on the back side without having to perform a front-to-back operation to swap the front-to-back relationship between the code window X1 and the execution window X2.

[0071] Traditionally, when the code window X1 was displayed in the foreground and the execution window X2 in the background, data input into the execution window X2 required bringing the execution window X2 to the front. This operation moved the execution window X2 to the front and the code window X1 to the background. Bringing to the front required pressing and releasing a device such as a touch panel or mouse. With a mouse, this required clicking the "close button" or "minimize button" on the code window X1. In this case, a series of actions were required: moving the mouse pointer to the "close button," pressing the left mouse button, and then releasing it. When performing the bringing to the front operation, the user is aware of a layer swap operation: "by removing the obstructing code window X1 from the front, the execution window X2 from the background will move to the front." The same principle applies when using shortcut keys, as bringing to the front operation swaps the layers of windows.

[0072] On the other hand, in the tablet terminal 100 of this embodiment, the user can target the execution window X2 simply by touching the targeting selection area 162. For example, the user can touch the targeting selection area 162 with their left hand holding the tablet terminal 100, while inputting data into the execution window X2 on the back with their right hand. When they want to input data into the code window X1, they only need to lift their finger from the targeting selection area 162. Because the user selects the focus window with their left hand and inputs data with their right hand, it is easy for the user to continue working without feeling interrupted when changing the focus window. In particular, when working while frequently switching between the code window X1 and the execution window X2, the user only needs to repeatedly touch and detach the targeting selection area 162 with their free hand, which significantly improves operability when changing the focus window.

[0073] Furthermore, since the order of the code window X1 and the execution window X2 is maintained when the focus window is changed, the user does not need to be aware of changes in the layer structure. In other words, even when there are many windows and the focus window is switched one after another, there is no need to worry about "which window has moved to which layer". In this embodiment, the visibility of the execution window X2 (focus window) is further enhanced by hiding the code window X1 when the targeting selection area 162 is touched.

[0074] In normal mode, the code window X1 is displayed semi-transparently, allowing both the code window X1 and the execution window X2 to be viewed simultaneously. There is no need to place the code window X1 and the execution window X2 side by side, or to move or close the code window X1 in order to view the execution window X2.

[0075] When the user continues to touch the selection area 166, etc., the explanation area 142 is displayed. Even if the function determination area M1 is not visible, the user understands that they can input function selections by touching near the edge of the monitor screen 102. Since the explanation area 142 is displayed when the user continues to touch the edge of the monitor screen 102, the user can easily understand which function selection inputs can be made by touching where.

[0076] As an additional user assistance feature, the display update unit 122 may display information indicating the available selection areas or the role of each selection area. This information display may involve each selection area lighting up and displaying its function name. For example, when a certain amount of time has passed since the user made a function selection input without making any further input, in other words, when a predetermined amount of time has passed since the appearance of the explanation area 142, the display update unit 122 may activate such an assistance function.

[0077] The user can activate gesture area G1 by touching the gesture activation area 164 (selection areas 128L2, 128R2). When gesture area G1 is activated, the user can perform various gesture inputs by moving their finger across the entire monitor screen 102. Because gesture input can be performed using the entire monitor screen 102, high operability can be achieved even when the monitor screen 102 is small. In addition, since gesture input and data input are switched by touching and detaching the gesture activation area 164, accidental data input will not occur while gesture input is being performed. Similarly, operations for data input will not be mistakenly recognized as gesture input.

[0078] As described above, according to this embodiment, for example, a right-handed user can perform data entry work with their right hand and use their free left hand to change the mode as needed.

[0079] It should be noted that the present invention is not limited to the embodiments and modifications described above, and the components can be modified and implemented without departing from the spirit of the invention. Various inventions may be formed by appropriately combining the multiple components disclosed in the embodiments and modifications described above. In addition, some components may be deleted from all the components shown in the embodiments and modifications described above.

[0080] Although it has been explained that the learning software is executed on a single tablet device 100, some of the functions of the learning software or the tablet device 100 may be implemented by other devices such as a server.

[0081] [Differentiation] Figure 18 is a conceptual diagram illustrating focus control for three or more windows. In Figure 18, six layers are composed of the function determination area M1, the gesture area G1, and four windows W1 to W4. The upper left of the function determination area M1 is set as the gesture activation area 164, and the lower edge is set as four targeting selection areas 162: W1 selection area 150, W2 selection area 152, W3 selection area 154, and W4 selection area 156. When the gesture activation area 164 is touched, the gesture area G1 is activated.

[0082] Window W1 is associated with selection area W1 150, window W2 with selection area W2 152, window W3 with selection area W3 154, and window W4 with selection area W4 156. When the user touches selection area W1 150, window W1 can be set as the focus window. Similarly, when the user touches selection area W2 152, window W2 can be set as the focus window. The same applies to selection areas W3 154 and W4 156.

[0083] The layer management unit 124 does not change the layer even when the target window is changed. Therefore, the order of windows W1 to W4 is maintained. For example, suppose window W1 is a web screen, window W2 is an editing screen for word processing software, window W3 is an editing screen for graphic editing software, and window W4 is an editing screen for spreadsheet software. If the user is working while switching between the four software programs as needed, they can simply switch the focus window as needed using the target selection area 162. Since the order of the layers is not changed, there is no need to worry about which layer the desired window is on, even when there are multiple windows.

[0084] Furthermore, the selected window may be set as the focus window on the condition that part or all of the selected window from windows W1 to W4 is obscured by any window in the foreground. In other words, when the foreground is not obscured, the focus change due to the function selection input may be treated as invalid. For example, suppose window W3 is selected by the W3 selection area 154. If part of window W3 is obscured by window W1 or window W2 in the foreground, the layer management unit 124 sets window W3 as the focus window. On the other hand, if the foreground of window W3 is not obscured for reasons such as both window W1 and window W2 being closed, the window W3 may not be changed to the focus window even if the W3 selection area 154 is touched.

[0085] In this embodiment, the code window X1 is displayed semi-transparently, allowing both the code window X1 and the execution window X2 to be viewed simultaneously. As a modification, the code window X1 may be displayed normally (opaque).

[0086] In this embodiment, the selection area 166 was described as being arranged at the four corners of the function determination area M1 (monitor screen 102). As a modification, multiple selection areas 166 may be arranged on the left side of the function determination area M1, or on the bottom side. In addition, the arrangement of the selection areas 166 can be arbitrarily set to the top side, right side, etc.

[0087] In this embodiment, the learning software was described as being run on a tablet device 100, but this learning software may also be run on other mobile computers such as smartphones or laptop PCs. Similarly, the learning software may be run on a stationary computer such as a desktop PC.

[0088] In this embodiment, it was described that the code window X1 is set as the focus window in normal mode. As a variation, the execution window X2 on the back side may be set as the focus window in normal mode, and the code window X1 on the front side may be changed to be the focus window in special mode.

[0089] Other methods besides touching the targeting selection area 162 may be used to switch to penetration mode. For example, the user may switch to penetration mode by double-clicking the mouse or by pressing a predetermined key on the keyboard. In addition, the user may switch to penetration mode by voice input, or by aligning their gaze with a predetermined area on the monitor screen 102. The user may also switch to penetration mode using a foot device such as a pedal. Similarly, data input may be performed not only by touch, but also by mouse, keyboard, voice, etc.

[0090] In this embodiment, it has been explained that the system is set to penetration mode only for the duration that the target selection area 162 is touched. As a variation, the layer management unit 124 may continue the penetration mode for a predetermined valid period, for example, 3 seconds, after the target selection area 162 is touched. If a penetration input is made during the valid period, the valid period may be extended by another 3 seconds. If no input is received for 3 seconds, the layer management unit 124 may return from penetration mode to normal mode.

[0091] In this embodiment, the code window X1 on the front is described as being displayed semi-transparently. As a variation, the display update unit 122 may display the code window X1 normally (opaque). The display update unit 122 may change the code window X1 to semi-transparent or transparent display on the condition that the code window X1 and the execution window X2 overlap. Alternatively, when the code window X1 and the execution window X2 overlap, the display update unit 122 may set only the overlapping area of ​​the code window X1 to semi-transparent or transparent display.

[0092] In this embodiment, the layer management unit 124 was described as not swapping the order of the code window X1 and the execution window X2. As a modification, the layer management unit 124 may conditionally swap the order of the code window X1 and the execution window X2. When a data input operation is performed on the exposed area of ​​the execution window X2, in other words, the area of ​​the execution window X2 that is not obscured by the code window X1, the layer management unit 124 may swap the order of the execution window X2 and the code window X1. For example, when point P1 (exposed area 134) in Figure 11 is touched, the layer management unit 124 may display the execution window X2 in front of the code window X1 and set the execution window X2 as the focus window.

[0093] In through-mode, when the user touches point P1 in the exposed area 134, the layer management unit 124 may maintain the front-to-back relationship between the code window X1 and the execution window X2. When the user touches the exposed area 134 of the execution window X2 (back), the front-to-back relationship may be maintained in through-mode, while the execution window X2 may be moved to the front in normal mode.

[0094] In this embodiment, it was described that when the pass-through mode is set, the semi-transparent code window X1 is hidden (made completely transparent). As a variation, both the code window X1 and the execution window X2 may be displayed normally. When the pass-through mode is set, the display update unit 122 may increase the transparency of the code window X1 to improve the visibility of the execution window X2 (focus window).

[0095] Additionally, when the system is set to pass-through mode, the transparency of the execution window X2 may be reduced. For example, suppose both the code window X1 and the execution window X2 are displayed semi-transparently. When pass-through mode is selected, the visibility of the execution window X2 (the focus window) may be improved by reducing its transparency.

[0096] The same applies when there are three or more windows, as shown in Figure 18. As an example, windows W1 to W4 are all displayed semi-transparently. When the user focuses on window W2, reducing the transparency of window W2 makes it easier for the user to recognize that window W2 is the focused window. At this time, the transparency of window W1, which is in the foreground, may be increased.

[0097] The rear window may be a window equipped with an input interface such as buttons, for example, a dialog box or a software keyboard. When the user wants to operate a dialog box, they should touch the target selection area 162 (selection areas 128L1, 128R1). In this case, the user may operate the rear window, which acts as a control panel, using through-input, and then return to the main window on the front side and operate it using normal input.

[0098] In this embodiment, the user transitions to penetration mode by touching the targeting selection area 162, and then performs data input operations in that state. As a modification, the user may dynamically select whether the code window X1 or the execution window X2 is the focus window depending on the settings when performing data input operations.

[0099] For example, the touch panel may have a function to detect touch pressure. In this case, the layer management unit 124 may set the code window X1 as the focus window when lightly touched, and the execution window X2 as the focus window when strongly touched. The operation detection unit 110 may also detect which finger is touching the panel based on the area of ​​the touching finger. For example, the code window X1 may become the focus window when touched with the index finger, and the execution window X2 may become the focus window when touched with the little finger.

[0100] The operation detection unit 110 may distinguish between finger and stylus pen touches based on differences in capacitance. In this case, the code window X1 may be set as the focus window when touched with a finger, and the execution window X2 may be set as the focus window when touched with a stylus pen.

[0101] Multiple types of stylus pens may be provided, and the operation detection unit 110 may identify each of them. A window may be associated with each of the multiple stylus pens. In this case, by using multiple stylus pens, it becomes possible to input data into multiple windows. For example, software can be conceivable in which a painting is created by layering windows for underpainting, sketching, rough drawing, and finishing, similar to an oil painting. In this case, the user may simultaneously operate the four semi-transparent windows while switching between stylus pens (types of brushes). The painting may then be completed by layering the four semi-transparent windows.

[0102] Gesture input may be enabled during penetration mode. For example, when the user touches only the targeting selection area 162, the code window X1 becomes the target of gesture input. On the other hand, when the user touches both the targeting selection area 162 and the gesture enablement area 164 simultaneously, the execution window X2 may be operated by the user's gesture input.

[0103] The through-input described in this embodiment can be used for purposes other than learning software. For example, two types of windows may be provided in a computer game. One is a game field window, and a menu window for selecting equipment is provided behind it. In this case, the user may operate the front window as the target while using through-input to operate the menu window as needed. Alternatively, multiple windows may be provided to correspond to multiple game fields. The game may be played simultaneously in the front and back game fields, and the user may progress through the game by switching the focus between the two windows.

[0104] In SNS (Social Networking Service) software, a pop-up window may appear while the user is typing in the chat window, causing data entry in the chat window to be mistakenly recognized as input into the pop-up window. To address this, in normal mode, the chat window may be set as the focus window. Even if a pop-up window appears in front of the chat window, the chat window in the background remains the focus window, preventing input into the pop-up window during a chat. When the user wishes to switch to inputting into the pop-up window, they can touch selection area 166 to set the pop-up window as the focus window. In this case as well, the relative positions of the pop-up window and the chat window are maintained. This control method prevents the user from having their input interpreted as input into a suddenly appearing pop-up window when they intended to input into the chat window. [Explanation of Symbols]

[0105] 100 Tablet terminal, 102 Monitor screen, 104 User interface processing unit, 106 Data processing unit, 108 Data storage unit, 110 Operation detection unit, 112 Output unit, 114 Function input unit, 116 Gesture input unit, 118 Data input unit, 120 Display control unit, 122 Display update unit, 124 Layer management unit, 126 Communication unit, 128L1 Selection area, 128L2 Selection area, 128R1 Selection area, 128R2 Selection area, 132 Overlay area, 134 Exposure area, 135 Start button, 136 Code menu, 138 Circle drawing button, 140 Circle, 142 Explanation area, 150 W1 Selection area, 152 W2 Selection area, 154 W3 Selection area, 156 W4 Selection area, 162 Targeting selection area, 164 Gesture activation area, 166 Selection area, 300 Processor, 302 First bus, 304 Memory, 308 Bridge, 310 Second bus, 312 Storage, 314 Input devices, 316 Output devices, 318 Peripherals, G1 Gesture area, M1 Function detection area, X1 Code window, X2 Execution window

Claims

1. The processor comprises, A first area for displaying a first image, which is to be displayed as the programming code, is placed in the foreground, and a second area for displaying a second image, which is to be displayed as the program execution screen, is placed behind the first area, and these are displayed on the monitor screen. The system accepts touch operations on an invisible predetermined area of ​​the monitor screen that designates the second area, and input operations from the user on the monitor screen for data input. When the input operation is detected while the touch operation is being detected, the first image is hidden while keeping the first area in front of the second area, the input operation is accepted as input to the second area, the second image is changed based on the input operation, and the programming code of the first image is updated in response to the change in the second image. When the detection of the touch operation is complete, the first image is redisplayed. Information processing device.

2. The processor is configured to display the program code on a monitor screen by placing a first area for displaying a first image at the front, and a second area for displaying the program execution screen for displaying a second image at the back of the first area. The processor receives touch operations on an invisible predetermined area of ​​the monitor screen that designates the second area, and input operations from the user on the monitor screen for data input. When the input operation is detected while the touch operation is being detected, the processor will hide the first image while keeping the first area in front of the second area, accept the input operation as input to the second area, change the second image based on the input operation, update the programming code of the first image in response to the change in the second image, and redisplay the first image when the detection of the touch operation is finished. program.

3. The processor displays the program code on a monitor screen by placing a first area for displaying a first image at the front and a second area for displaying a second image behind the first area. The processor receives a touch operation on a predetermined invisible area of ​​the monitor screen that designates the second area, and an input operation from the user on the monitor screen for data input. When the processor detects an input operation while detecting the touch operation, it hides the first image while keeping the first area in front of the second area, accepts the input operation as input to the second area, changes the second image based on the input operation, updates the programming code of the first image in response to the change in the second image, and redisplays the first image when the detection of the touch operation ends. Information processing methods.

4. The system comprises a server and a terminal device, and the processor of the terminal device is A first area for displaying a first image, which is to be displayed as the programming code, is placed in the foreground, and a second area for displaying a second image, which is to be displayed as the program execution screen, is placed behind the first area, and these are displayed on the monitor screen. The system accepts touch operations on an invisible predetermined area of ​​the monitor screen that designates the second area, and input operations from the user on the monitor screen for data input. When the input operation is detected while the touch operation is being detected, the first image is hidden while keeping the first area in front of the second area, the input operation is accepted as input to the second area, the second image is changed based on the input operation, and the programming code of the first image is updated in response to the change in the second image. When the detection of the touch operation is complete, the first image is redisplayed. system.

Images