Information processing system, information processing device, and control method

Abstract

To improve an operational feeling in drawing input.SOLUTION: A touch sensor overlaps with a display and detects a contact position with an input device. A controller samples the contact position, and a host system generates a trajectory of the contact position, and the controller causes the display to display a line segment from a contact position of a previous sample to a contact position of a current sample before acquiring a part of the trajectory corresponding to the contact position of the current sample from the contact position of the previous sample. The present embodiment can be implemented in any form such as an information process system, an information processing apparatus, a control method or the like.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present application relates to an information processing system, an information processing device and a control method, for example, for displaying drawn graphics. [Background technology]

[0002] An electrophoretic display is characterized by being made of a flexible material having plasticity, and is also called an electronic paper display (EPD) or an electronic ink display (E-Ink). An EPD has the advantage of consuming less power than other types of display devices such as a liquid crystal display (LCD) or an organic electroluminescence display (OLED). Patent Document 1 describes how the use of an EPD as a display device in information processing devices such as various portable information processing terminals, wearable information processing terminals, electronic POP (Point of Purchase advertising), electronic shelf labels, and electronic posters makes it possible to reduce power consumption. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-64421 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, EPDs are so-called "slow devices." The refresh rate of EPDs is typically about 8 Hz. In contrast, the refresh rate of LCDs and OLEDs is about 60 Hz to 120 Hz. When displaying a figure drawn arbitrarily using an input device, the visible rendering latency from operation to display may be significantly perceived by the user in EPDs. Such a latency may impair the user experience (UX), particularly the sense of operation, in handwritten input. In LCDs or OLEDs, it may be possible to predict and display the input trajectory to compensate for the latency, and erase the part with a significant error, but this is not realistic for EPDs. As exemplified in FIG. 4, the error occurring in the predicted trajectory tends to be large and is not immediately erased, so the prediction error is significantly visually recognized, which may give the user a sense of incongruity. [Means for solving the problem]

[0005] The present application has been made to solve the above-mentioned problems, and an information processing system according to one aspect of the present application includes a host system, a controller, a touch sensor, and a display, wherein the touch sensor overlaps the display and detects a contact position with an input device, the controller samples the contact position, the host system generates a trajectory of the contact position, and the controller causes the display to display a line segment from the contact position of the previous sample to the contact position of the current sample before acquiring a portion of the trajectory that corresponds to the contact position of the current sample from the contact position of the previous sample.

[0006] In the above information processing system, the controller may display the line segment with a width narrower than that of the trajectory.

[0007] In the above information processing system, the controller may display the line segment in the same line type as the trajectory.

[0008] In the above information processing system, the controller may erase a portion of the line segment that deviates from the trajectory.

[0009] In the above information processing system, the display may be an electrophoretic display.

[0010] The above-mentioned information processing system may further include the input device, wherein the input device acquires drawing control information indicating whether or not drawing is required from the host system, outputs the drawing control information to the controller, and the controller determines whether or not the line segment needs to be displayed based on the drawing control information.

[0011] In the above information processing system, the drawing control information may include drawing mode information indicating a drawing mode for representing the trajectory, and the controller may display the line segment in accordance with the drawing mode.

[0012] An information processing device according to a second aspect of the present application includes a host system, a controller, a touch sensor, and a display, the touch sensor overlaps the display and detects a contact position with an input device, the controller samples the contact position from the touch sensor, the host system generates a trajectory of the contact position, and the controller causes the display to display a line segment from the contact position of the previous sample to the contact position of the current sample before acquiring a portion of the trajectory that corresponds to the contact position of the current sample from the contact position of the previous sample.

[0013] A control method according to a third aspect of the present application is a control method in an information processing system including a host system, a controller, a touch sensor, and a display, wherein the touch sensor overlaps the display, the touch sensor detects a contact position with an input device, the controller samples the contact position, the host system generates a trajectory of the contact position, and the controller causes the display to display a line segment from the contact position of the previous sample to the contact position of the current sample before acquiring a portion of the trajectory that corresponds to the contact position of the current sample from the contact position of the previous sample. Effect of the Invention

[0014] According to the embodiment of the present application, the operational feel during drawing input is improved. [Brief description of the drawings]

[0015] [Figure 1] 1 is a block diagram showing an example of a functional configuration of an information processing system according to a first embodiment. [Diagram 2] 1 is a block diagram showing a schematic configuration example of an information processing device according to a first embodiment. [Diagram 3] FIG. 11 is an explanatory diagram showing an example of a display of a path created by handwriting input. [Figure 4] FIG. 11 is an explanatory diagram showing an example of a predicted trajectory of handwritten input; [Diagram 5] 5 is a flowchart illustrating an example of a drawing process according to the first embodiment. [Figure 6] FIG. 11 is a schematic block diagram illustrating an example of a functional configuration of an information processing system according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] <First embodiment> Hereinafter, embodiments of the present application will be described with reference to the drawings. First, a configuration example of an information processing system S1 according to a first embodiment of the present application will be described. Fig. 1 is a schematic block diagram showing a functional configuration example of the information processing system S1 according to this embodiment. The information processing system S1 includes an information processing device 10 and a display device 40. The information processing device 10 and the display device 40 are connected wirelessly or by wire so as to be able to transmit and receive various data. The information processing device 10 is an example of an electronic device having a computer system. The information processing device 10 may be realized in any form, such as a personal computer, a tablet terminal, or a mobile phone. The information processing device 10 may be configured integrally with the display device 40, or may be configured as hardware separate from the display device 40.

[0017] The information processing device 10 acquires pen data from the display device 40. The pen data is data indicating a contact position sampled at a predetermined period. The contact position is a position on the display surface of a display 46 (described later) that is contacted by an input device 50. The information processing device 10 generates a trajectory of the contact position based on a time series consisting of the contact position for each sample. The information processing device 10 outputs display data representing the generated trajectory to the display device 40.

[0018] The display device 40 is an example of an electronic device equipped with a touch sensor 44 and a display 46. In this embodiment, an EPD is used as the display 46. The touch sensor 44 and the display 46 overlap each other so that the detection surface of the touch sensor 44 is covered by the display surface of the display 46. The display device 40 can be configured as a so-called flexible display.

[0019] The touch sensor 44 detects (samples) the position on the detection surface where the input device 50 touches at a predetermined period, and generates pen data indicating the detected position as the touch position. The display device 40 outputs the generated pen data to the information processing device 10. The display device 40 displays on the display 46 a line drawing based on the contact position of the previous sample and the contact position of the current sample for each sample (pre-rendering). After that, the display device 40 displays on the display 46 a trajectory indicated in the display data input from the information processing device 10 (actual rendering). Furthermore, when performing the pre-rendering, the display device 40 erases the line drawing displayed on the display 46. In this application, the "current sample" refers to the latest sample at that time, and the "previous sample" refers to the sample immediately before the current sample, that is, the next newest sample.

[0020] The input device 50 is held by a user and used for inputting or editing data such as characters, symbols, and figures. When one end of the input device 50 comes into contact with the detection surface of the touch sensor 44, a figure formed based on the contact position is detected. The input device 50 is not limited to electronic devices, and may be any instrument as long as it can detect the contact position with the detection surface. The input device 50 is typically an operation medium having a pen shape, and may be configured as a writing device. The input device 50 may have any form such as a stylus, a digital pen, or the like. The input device 50 may or may not have a haptic feedback function.

[0021] Next, an example of the hardware configuration of the information processing device 10 will be described. The information processing device 10 includes a processor 11, a main memory 12, a flash memory 13, a PCH 21, an audio system 24, a microphone 25, a speaker 26, a baseband chip 27, a communication unit 28, and an input / output unit 29.

[0022] The processor 11 controls the overall functions of the information processing device 10. For example, one or more CPUs (Central Processing Units) are applied as the processor 11. The processor 11 executes a predetermined program, cooperates with a main memory 12 and other hardware, and performs the functions of a host system 100 (described later). In this application, the processor 11 or other hardware executing processing instructed by instructions written in a program may be referred to as "executing a program" or "running a program."

[0023] The main memory 12 is a writable memory used as a working area for the processor 11, i.e., an area for reading programs to be executed and various setting data, and an area for writing processing data acquired by executing the programs. The main memory 12 is configured to include, for example, a plurality of DRAM (Dynamic Random Access Memory) chips. The executable programs include an OS, various device drivers for controlling peripheral devices, various services / utilities, applications, and the like.

[0024] The processor 11 and the main memory 12 function as a system device constituting a computer system, i.e., a host system, which is the core of the information processing device 10. The computer system of the information processing device 10 includes a system device as hardware, and software such as an OS and a schedule task.

[0025] Various programs and data are stored in the flash memory 13. The various programs include, for example, firmware, device drivers, services / utilities, and applications. These programs are executed by the processor 11. The stored data includes data to be processed by the processor 11 and data generated by the processing. The generated data may include not only final data, but also intermediate data used in a step subsequent to a certain step.

[0026] The PCH (Platform Controller Hub) 21 includes one or more controllers and can be connected to a plurality of devices to input and output various data. The PCH 21 includes, for example, one or a combination of bus controllers such as USB, Serial ATA (Advanced Technology Attachment), SPI (Serial Peripheral Interface) bus, PCI (Peripheral Component Interconnect) bus, PCI-Express bus, and LPC (Low Pin Count). The devices to which the PCH 21 is connected include, for example, an audio system 24, a baseband chip 27, and an input / output unit 29, which will be described later.

[0027] The audio system 24 performs processes such as processing, input, output, recording, playback, encoding, and decoding of audio signals. The audio system 24 includes, for example, an audio IC (Integrated Circuit). A microphone 25 and a speaker 26 are connected to the audio system 24. An audio signal can be input to the audio system 24 from the processor 11, the microphone 25, or a baseband chip 27. The audio system 24 reads out the audio signal recorded therein. The audio system 24 can output the acquired audio signal to the speaker 26 or the processor 11.

[0028] The microphone 25 picks up sounds arriving at the microphone 25 and outputs audio signals of the picked up sounds to the audio system 24. The speaker 26 presents sound based on the audio signal input from the audio system 24 .

[0029] The baseband chip 27 is a dedicated IC for controlling communication using the communication unit 28. The baseband chip 27 realizes communication using, for example, a public wireless communication system such as 4G (fourth generation wireless communication system) or 5G (fifth generation wireless communication system), or an in-house wireless communication network defined in IEEE802.11. The baseband chip 27, under the control of the processor 11, connects to other devices via a communication network using the communication unit 28 so as to be able to transmit and receive various types of data, and transmits and receives various types of data.

[0030] The communication unit 28 is a wireless communication module for connecting to a communication network wirelessly or by wire. The communication unit 28 may include an antenna for transmitting and receiving radio waves used for wireless communication. The communication unit 28 can use, for example, a wireless communication method defined in IEEE802.11.

[0031] The input / output unit 29 includes an interface for wirelessly or wired connection to other devices (mainly the display device 40) so that various data can be input / output using a predetermined input / output method. The input / output unit 29 can use an input / output method defined in, for example, USB (Universal Serial Bus) 3.0, 3.1, etc.

[0032] Next, an example of the hardware configuration of the display device 40 according to the present embodiment will be described. The display device 40 includes a controller 42 , a touch sensor 44 , a display 46 and an input / output unit 48 .

[0033] The controller 42 executes predetermined firmware to perform processes for implementing the functions of the display device 40 and processes for controlling the functions. The controller 42 includes, for example, a timing controller. The controller 42 displays on the display 46 a display screen indicated by display data input from the information processing device 10 via the input / output unit 48 (drawing process). The controller 42 specifies a signal value for each pixel specified by the display data, and causes the pixel to emit light at a luminance corresponding to the specified signal value. The controller 42 samples the contact position indicated by the detection signal input from the touch sensor 44 at every predetermined sampling period, and acquires pen data indicating the contact position for each sample. The controller 42 outputs the acquired pen data to the information processing device 10 using the input / output unit 48.

[0034] The touch sensor 44 detects a position where another object touches the touch sensor. The other object that touches the touch sensor is mainly an input device 50. The touch sensor 44 has a sheet-like shape that is much smaller than its width or length. A detection surface is set on most of the surface of the touch sensor 44. The touch sensor 44 has a plurality of detection elements, and each detection element is arranged at a different position within the detection surface. The touch sensor 44 detects an area of ​​the detection surface that has been in contact with another object at each predetermined sampling period, and determines a representative point (e.g., reception) of the detected area as the contact position.

[0035] The touch sensor 44 generates pen data indicating the determined contact position, and outputs the generated pen data to the controller 42 using a predetermined input / output method. As the input / output method, for example, a serial communication method such as an I2C (Inter-Integrated Circuit) method can be used. In response to this, the controller 42 includes a serial communication controller, obtains a detection signal indicating the presence or absence of contact for each pixel, determines an area in which pixels determined to be in contact are distributed as a contact area, and determines the center of gravity of the contact area as a contact position. The touch sensor 44 may employ any operating principle, such as a resistive film method or a capacitive method.

[0036] The display 46 displays a pattern instructed by the display data under the control of the controller 42. The display 46 has a sheet-like shape. The display 46 has a plurality of pixels, and each pixel is arranged at a different position on the display surface. Each pixel emits light at a luminance instructed by the controller 42. The display 46 includes, for example, a TTL (Transistor Transistor Logic) circuit. The TTL is used to supply a driving power corresponding to a signal value to each pixel. Each pixel emits light at a luminance corresponding to the supplied driving power. In response to this, the controller 42 includes a TTL controller, and applies an electric signal having a potential corresponding to the signal value of each pixel to the TTL.

[0037] The input / output unit 48 has an interface that connects wirelessly or wired to other devices so that various data can be input and output using a predetermined input / output method. The other devices that are the input source and output destination are mainly the information processing device 10. The input / output unit 48 uses the same input / output method as the other devices that are the input source and output destination. In contrast, the controller 42 has a USB controller, and obtains input data from the input source using the input / output unit 48 in accordance with the USB regulations, generates output data in accordance with the regulations, and outputs the generated output data to the output destination.

[0038] Next, an example of the functional configuration of the information processing system S1 according to this embodiment will be described. Fig. 2 is a block diagram showing an example of the schematic configuration of the information processing system S1 according to this embodiment. The information processing device 10 includes a host system 100 and an input / output unit 29. The host system 100 includes a pen data acquisition unit 102, a drawing processing unit 104, a post-processing unit 106, and a drawing setting unit .

[0039] The pen data acquisition unit 102 waits for pen data to be input from the display device 40 via the input / output unit 29. The pen data acquisition unit 102 outputs the acquired pen data to the drawing processing unit 104. The drawing processing unit 104 receives pen data from the pen data acquisition unit 102. The drawing processing unit 104 identifies the contact position for each sample indicated by the pen data. The drawing processing unit 104 uses a predetermined function to generate a curve based on the contact positions for each sample up to the current sample as a trajectory. As the predetermined function, a function that passes through a plurality of contact positions and gives a curve that sequentially connects these contact positions can be used. For example, the pen data acquisition unit 102 generates a Bezier curve by interpolating the coordinates between the contact positions using a linear combination of Bernstein bases. The pen data acquisition unit 102 may generate a spline curve by interpolating the coordinates between the contact positions for each section between two contact positions that are in sequential order using a polynomial.

[0040] The drawing processing unit 104 generates display data that expresses the generated trajectory using a predetermined drawing mode, and outputs the generated display data to the display device 40 via the input / output unit 29. This allows the display device 40 to display the generated trajectory (standard drawing). As the drawing mode, any one of distinction between drawing and erasing, line type, color, line width, etc., or any combination thereof can be specified. The drawing mode may be set in the drawing processing unit 104 in advance, or may be specified by the drawing setting unit 108. The display of the trajectory by standard drawing is delayed compared to the display of the line drawing by temporary drawing (described later). The drawing processing unit 104 outputs the generated display data to the post-processing unit 106.

[0041] Since the generated trajectory is made of curved lines, it does not necessarily overlap completely in space with the line segment connecting adjacent samples by the provisional drawing (see FIG. 3). Therefore, the post-processing unit 106 specifies, as unnecessary figures, the part of the line drawing up to the current sample that is not covered by the generated trajectory and deviates, that is, the part where the line segment does not overlap spatially with the trajectory and protrudes, as a post-processing. The post-processing unit 106 can specify the area occupied by the trajectory using the display data input from the drawing processing unit 104. The post-processing unit 106 can specify the line drawing displayed by the provisional drawing based on the pen data input from the pen data acquisition unit 102. From the contact position of each sample indicated in the pen data, for each pair of two adjacent contact positions, a line segment with the two contact positions as both ends is specified, and a figure formed by connecting the individual line segments corresponds to the line drawing by the provisional drawing. The post-processing unit 106 may not generate a line drawing based on the contact position, but may acquire an existing line drawing. The post-processing unit 106 may obtain, for example, from the display control unit 422 of the display device 40, line drawing data indicating the line drawing displayed on the display 46 during temporary drawing.

[0042] The post-processing unit 106 outputs unnecessary figure data indicating the erasure of the identified unnecessary figure to the display device 40 via the input / output unit 29. This makes it possible to erase unnecessary figures that deviate from the trajectory from among the temporary drawing figures displayed on the display device 40 (unnecessary figure erasure). The period of the post-processing and unnecessary figure erasure may be longer than the sampling period of the contact position. The period of the post-processing is set to be equal to the refresh rate of the display 46 (for example, about 60 Hz). However, it is preferable that the period of the post-processing is shorter than the presentation time required for visual recognition of the visual information.

[0043] The drawing setting unit 108 specifies a drawing mode based on operation data input from various input devices, and sets the specified drawing mode to the drawing processing unit 104. The drawing mode includes one or more element items, and is specified by one of a plurality of candidate items for each element item. The drawing setting unit 108, for example, displays a predetermined drawing setting screen on the display 46 in which screen components representing the respective candidate items are arranged. The drawing setting unit 108 can select a drawing mode by specifying a candidate item corresponding to a screen component that includes a position designated by the operation data within a display area while the drawing setting screen is being displayed. The drawing setting unit 108 may apply pen data acquired from the display device 40 as the operation data. The element items include, for example, a distinction between drawing and erasing, a line type, a color, and a line width. The drawing setting unit 108 sets drawing mode information indicating the specified drawing mode to the display device 40. The drawing mode set to the display device 40 can be applied to the display of a line segment in a temporary drawing. The drawing mode information is also called brush information. A drawing mode may be a characteristic of the writing displayed by a physical writing implement, and the characteristic or drawing mode may be distinguished by the type of writing implement (e.g., ballpoint pen, highlighter, brush, marker, etc.), and erasure may be specified as an "eraser."

[0044] The controller 42 of the display device 40, the touch sensor 44, the display 46, and the input / output unit 48 execute predetermined firmware and cooperate with memory and other hardware to perform the functions of a display control unit 422. The display control unit 422 executes processing related to drawing based on the pen data. The display control unit 422 samples the detection position where the input device 50 touches the detection surface of the touch sensor 44 at a constant sampling period, and acquires pen data indicating a time series of the contact position. The display control unit 422 generates a line drawing including a line segment from the contact position of the previous sample to the contact position of the current sample for each sample. The display control unit 422 displays the generated line drawing on the display surface of the display 46 (provisional drawing). In FIG. 3, x marks exemplify each sample of the contact position. The x mark closest to the tip T of the input device 50 indicates the current sample. A line drawing formed by sequentially connecting the line segments between adjacent samples is displayed by the provisional drawing.

[0045] 2, the display control unit 422 transmits the acquired pen data to the information processing device 10 using the input / output unit 48 (pen data transfer). The transmitted pen data is used in the information processing device 10 mainly to generate a trajectory of the contact position. Display data representing the generated trajectory is input to the display control unit 422 from the information processing device 10 using the input / output unit 48. The display control unit 422 displays the trajectory represented in the display data on the display surface of the display 46 (standard drawing).

[0046] The display control unit 422 receives input of unnecessary figure data indicating unnecessary figures not covered by the trajectory from the displayed line segments from the information processing device 10 using the input / output unit 48. The display control unit 422 identifies the unnecessary figures indicated in the unnecessary figure data. The post-processing unit 106 erases the identified unnecessary figures from the line drawing displayed on the display 46 (unnecessary figure erasure). By erasing the unnecessary figures, it is possible to prevent the temporary drawing from causing a sense of incongruity to the user. When erasing the unnecessary figures, the post-processing unit 106 may restore the display of a partial background, which is a part of the background image that was displayed before the line drawing was displayed, in the area occupied by the unnecessary figures.

[0047] The display control unit 422 may receive drawing mode information from the information processing device 10 using the input / output unit 48. The display control unit 422 sets the drawing mode indicated by the drawing mode information and applies it as the drawing mode of the line drawing in the temporary drawing. Since the line drawing in the temporary drawing is displayed in the same manner as the trajectory in the standard drawing, the two are no longer clearly distinguishable from each other, and therefore, the sense of incongruity caused by the juxtaposition of the difference between the two can be eliminated or reduced.

[0048] Next, a display example of a trajectory by handwriting input will be described. FIG. 3 is an explanatory diagram showing a display example of a trajectory by handwriting input. FIG. 3 illustrates a trajectory and a line segment displayed on the display surface of the display 46 while the tip of the input device 50 is moved from left to right while touching the detection surface of the touch sensor 44. In the example of FIG. 3, in order to distinguish between the trajectory by standard drawing and the line segment by provisional drawing, both are displayed with different line widths. The distance from the tip T of the input device 50 to the right end of the displayed trajectory is caused by a processing delay (standard drawing delay) from the detection of the contact position through the generation of the trajectory to the display of the generated trajectory. In this embodiment, before the trajectory by standard drawing is displayed, the line segment Seg is displayed by provisional drawing. The processing delay (provisional drawing delay) from the detection of the contact position to the display of the line segment Seg by provisional drawing is shorter than the processing delay (standard drawing delay) from the detection of the contact position to the display of the trajectory by standard drawing, thereby shortening the distance between the tip T of the input device 50 and the end of the displayed line segment Seg, thereby improving the operability. Since both the trajectory displayed by standard drawing and the line segment Seg displayed by temporary drawing are based on a common contact position, most of the line segment Seg is overwritten by the trajectory. In the example of FIG. 3, the width of the line segment Seg by temporary drawing is narrower than the width of the trajectory by standard drawing. Since the trajectory is made of curved lines, it is possible to avoid or suppress the generation of unnecessary figures that constitute the portion of the line segment Seg that protrudes from the trajectory, compared to when the width of the line segment Seg is equal to the trajectory. In addition, by making the line type of the line segment Seg by temporary drawing and the trajectory by standard drawing the same, it is possible to eliminate or reduce the sense of incongruity caused by the difference between the two.

[0049] Next, an example of the drawing process according to the present embodiment will be described with reference to a flowchart shown in FIG 5. (Step S102) The display control unit 422 of the display device 40 detects the contact position of the input device 50 with the touch sensor 44 at every predetermined sampling period, and obtains pen data indicating the detected contact position. (Step S104) The display control unit 422 causes the display 46 to display a line drawing (line segment Seg) including a line segment connecting the contact position of the previous sample to the contact position of the current sample (provisional drawing).

[0050] (Step S106) The display control unit 422 outputs the acquired pen data to the host system 100 of the information processing device 10. (Step S108) The pen data acquiring unit 102 of the host system 100 acquires the pen data from the display device 40. (Step S110) The drawing processing unit 104 generates a curve based on the contact position of each sample indicated in the acquired pen data as a trajectory, and generates display data that represents the generated trajectory in the set drawing mode (drawing process). The drawing processing unit 104 outputs the generated display data to the display device 40. (Step S112) Display data is input from the information processing device 10 to the display control unit 422 of the display device 40, and the trajectory indicated in the input display data is displayed on the display 46 (standard drawing). That is, the display control unit 422 performs the provisional drawing in step S104 from the pen data indicating the contact position of the pen by the same sampling, and also executes the standard drawing in step S112.

[0051] (Step S114) The post-processing unit 106 identifies, among the line segment Seg displayed in the provisional drawing, a portion that does not overlap with the locus displayed in the standard drawing as an unnecessary figure. The post-processing unit 106 may output unnecessary figure data indicating the erasure of the identified unnecessary figure to the display device 40 (post-processing). The processing of this step and the next step may be executed only when the line segment Seg in the provisional drawing is not completely covered by the locus in the standard drawing. (Step S116) The display control unit 422 receives the unnecessary graphic data from the information processing device 10 and erases the unnecessary graphic from among the graphics displayed on the display 46 (unnecessary graphic erasure). Then, the process returns to step S102.

[0052] <Second embodiment> Next, a configuration example of an information processing system S1 according to a second embodiment of the present application will be described. In the following description, differences from the above embodiment will be mainly described. Configurations and processes common to the above embodiment will be denoted by common reference numerals, and the description thereof will be used unless otherwise specified.

[0053] The display control unit 422 of the controller 42 causes the display 46 to provisionally draw a line drawing based on the contact position indicated by the pen data acquired from the touch sensor 44. If the contact position is included in a predetermined drawing area to be drawn, the display control unit 422 executes the provisional drawing, and if the contact position is not included in the drawing area, the display control unit 422 does not execute the provisional drawing. The drawing area may differ depending on each display screen. Although the drawing process may be instructed by an application executed by the host system 100 of the information processing device 10, access to the controller 42 is usually not taken into consideration. If an appropriate drawing area is not set in the display control unit 42, there is a risk that it will not be possible to accurately determine whether or not provisional drawing is required.

[0054] Therefore, in the information processing system S1 according to the present embodiment, the controller 42 of the display device 40 acquires drawing control information indicating whether or not temporary drawing is required from the information processing device 10 via the input device 50. An input / output function (Haptic API) related to a haptic presentation function that vibrates the input device 50 during drawing may be used to transmit the drawing control information. The haptic presentation function is a function that vibrates in response to contact with the touch sensor 44. The input device 50 vibrates based on vibration control information input from the information processing device 10. The vibration control information includes at least information indicating whether or not vibration is required. Whether or not vibration is required corresponds to whether or not the contact position is included in the drawing area. In addition, the drawing mode may be selected based on a position (for example, a screen component such as a button) designated on a setting screen. The setting of the position may also differ depending on each application.

[0055] 6 is a schematic block diagram showing an example of a functional configuration of an information processing system S1 according to this embodiment. The information processing system S1 includes an information processing device 10, a display device 40, and an input device 50. In the description of this embodiment, the input device 50 is mainly a device pen having a tactile sense providing function. The vibration control information may include information indicating vibration characteristics of the input device 50. Each vibration characteristic may be associated with a drawing mode of a trajectory to be displayed on the display 46. The vibration characteristic may be specified by the type of physical writing implement to be simulated in that drawing mode.

[0056] The host system 100 of the information processing device 10 executes an application and determines whether or not the contact position indicated in the pen data acquired from the display device 40 is included in the drawing area set by the application. The application to be executed and the execution of the application are instructed according to a user's operation. The host system 100 performs drawing processing based on the drawing area contact position and does not perform drawing processing based on the contact position outside the drawing area. The host system 100 outputs drawing control information indicating whether or not drawing processing is required to the input device 50. The host system 100 may select one drawing mode from a plurality of predetermined drawing modes according to a user's operation, and output drawing mode information indicating the selected drawing mode to the input device 50 by including it in the drawing control information. When performing drawing processing according to an application or selecting a drawing mode, the host system 100 may call the pen data acquisition unit 102, the drawing processing unit 104, the post-processing unit 106, and the drawing setting unit 108 and cause them to perform their processing.

[0057] The input device 50 includes a device control unit 52 , a vibration unit 55 , and an input / output unit 58 . The device control unit 52 controls the functions of the input device 50. The device control unit 52 includes, for example, an MCU (Micro Controller Unit) and executes a predetermined program to realize its functions. The MCU includes a processor, a memory, and an input / output interface.

[0058] The device control unit 52 receives drawing control information from the information processing device 10 via the input / output unit 58 , relays the input drawing control information, and outputs it to the display device 40 . The device control unit 52 acquires a drive signal for vibrating the vibration unit 55 based on the input drawing control information. The device control unit 52 outputs the acquired drive signal to the vibration unit 55.

[0059] For example, reference vibration waveform information is set in the device control unit 52. The reference vibration waveform information is information indicating a reference vibration waveform having vibration characteristics that are a predetermined standard. The device control unit 52 refers to the reference vibration waveform information, and when drawing control information (e.g., transmission API) indicating a drawing request (a vibration request) is input, the device control unit 52 outputs a drive signal indicating the reference vibration waveform to the vibration unit 55. When drawing control information (e.g., stop API) indicating a drawing not request (a vibration not request) is input, the device control unit 52 outputs a drive signal indicating the reference vibration waveform to the vibration unit 55. do not .

[0060] The device control unit 52 may have pre-set vibration waveform information for each drawing mode for each drawing mode. The vibration waveform information for each drawing mode is information indicating a vibration waveform having vibration characteristics corresponding to each drawing mode. The device control unit 52 refers to the vibration waveform information for each drawing mode, identifies the drawing mode specified by the drawing control information, and outputs a drive signal indicating the vibration waveform for each drawing mode related to the specified drawing mode to the vibration unit 55. The drawing mode may be specified using, for example, a waveform ID (wave ID). The waveform ID can also be regarded as identification information indicating the vibration waveform for each drawing mode.

[0061] The vibration unit 55 vibrates according to the drive signal input from the device control unit 52. The vibration unit 55 includes, for example, a digital-to-analog converter (DAC), an amplifier, and an actuator. The DAC converts the digital drive signal input from the device control unit 52 into an analog drive signal. The DAC outputs the converted drive signal to the amplifier. The amplifier adjusts the amplitude of the drive signal input from the DAC, and outputs the drive signal with the adjusted amplitude to the actuator. The actuator generates vibration according to the drive signal input from the amplifier. The actuator includes, for example, a piezoelectric vibrator such as a piezo element.

[0062] The input / output unit 58 includes an input / output interface for wirelessly or wiredly inputting and outputting various data to and from other devices. The other devices are mainly the information processing device 10 and the display device 40. The input / output unit 58 enables communication with the information processing device 10 and the display device 40 using, for example, a wireless communication method defined in IEEE802.15.1.

[0063] The display control unit 422 of the display device 40 determines whether or not temporary drawing is required according to drawing control information input via the input device 50. When the display control unit 422 determines that temporary drawing is required, it performs temporary drawing based on the contact position indicated in the pen data, and when it determines that temporary drawing is not required, it does not perform temporary drawing. When the drawing control information includes drawing mode information, the display control unit 422 performs temporary drawing according to the drawing mode indicated by the drawing mode information.

[0064] Therefore, even if a drawing area is not set on the display device 40 by an application, it is possible to accurately determine whether or not temporary drawing is required based on the drawing control information notified via the input device 50. Also, temporary drawing can be executed according to the drawing mode included in the drawing control information. Note that the input device 50 may omit the vibration unit 55 and may not have a haptic feedback function. Even in this case, the device control unit 52 relays the drawing control information output from the information processing device 10 to the display device 40.

[0065] As described above, the information processing system S1 according to this embodiment includes the host system 100, the controller 42, the touch sensor 44, and the display 46. The touch sensor 44 overlaps with the display 46 and detects a contact position with the input device 50. The controller 42 samples the contact position, and the host system 100 generates a trajectory of the contact position. Before acquiring a portion of the generated trajectory that corresponds to the contact position of the current sample from the contact position of the previous sample, the controller 42 causes the display 46 to display a line segment from the contact position of the previous sample to the contact position of the current sample. The display 46 may be an electrophoretic display. The information processing system S1 may be configured as an information processing device 10 including the host system 100 and the display device 40. According to this configuration, before the portion of the trajectory corresponding to the contact position of the current sample from the contact position of the previous sample is acquired, a line segment from the contact position of the previous sample to the contact position of the current sample is provisionally displayed on the display 46 (provisional drawing). The delay time required for displaying the line segment from the contact position of the previous sample to the contact position of the current sample (standard drawing) can be reduced to be less than the delay time required for displaying the portion of the trajectory corresponding to the contact position of the current sample from the contact position of the previous sample. This improves the operability of drawing input. In addition, since the line drawing related to the provisional drawing and the trajectory related to the standard drawing are based on a common contact position, the difference between them and the trajectory generated based on the existing contact position can be reduced more than the trajectory predicted based on the past contact position.

[0066] The controller 42 may display the line segments in the provisional drawing with a narrower width than the trajectory in the standard drawing. Since the trajectory is generally a curved line, if the line segment of the temporary drawing has the same width, there will be areas where the two do not overlap. By making the line segment of the temporary drawing narrower in width than the trajectory, the area not covered by the trajectory can be eliminated or reduced. The discomfort felt by the user by the area not covered by the trajectory can be eliminated or reduced.

[0067] The controller 42 may display the line segments related to the provisional drawing in the same line type as the trajectory related to the standard drawing. By making the display mode common to the trajectory, the line segments related to the temporary drawing become less noticeable, so that it is possible to reduce the sense of discomfort felt by the user by the line segments related to the temporary drawing.

[0068] The controller 42 may erase any portion of the line segment displayed on the display 46 that deviates from the displayed trajectory. According to this configuration, by erasing the portion of the line segment related to the provisional drawing that is not covered by the trajectory related to the standard drawing and extends beyond the trajectory, it is possible to eliminate any sense of discomfort felt by the user by the display of that portion.

[0069] The input device 50 may obtain drawing control information indicating whether drawing is required from the host system 100, and output the drawing control information to the controller 42, and the controller 42 may determine whether or not it is required to display the line segment based on the drawing control information. According to this configuration, the controller 42 can determine whether or not temporary drawing based on the pen data is necessary, based on the drawing control information notified from the host system 100 via the input device 50. Therefore, even if the program executed by the host system 100 cannot know whether or not the pen data is related to temporary drawing, the controller 42 can determine whether or not temporary drawing is necessary. In addition, in transmitting the drawing control information, a procedure similar to that for the control information related to the tactile presentation function can be used to achieve economical realization.

[0070] The drawing control information may include drawing mode information indicating a drawing mode for expressing a trajectory, and the controller 42 may display the line segment according to that drawing mode. According to this configuration, the controller 42 receives the drawing mode from the drawing control information notified from the host system 100 via the input device 50. This makes it possible to more reliably realize temporary drawing using the drawing mode instructed by the host system 100. In transmitting the drawing mode, a similar procedure to that for the control information related to the tactile sense provision function is used, thereby making it possible to realize the drawing mode economically.

[0071] Although the embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above embodiment, and the present invention also includes designs that do not deviate from the gist of the present invention. The configurations described in the above embodiment can be combined in any combination. [Explanation of symbols]

[0072] S1...information processing system, 10...information processing device, 29...input / output unit, 40...display device, 42...controller, 44...touch sensor, 46...display, 48...input / output unit, 52...device control unit, 55...vibration unit, 58...input / output unit, 100...host system, 102...pen data acquisition unit, 104...drawing processing unit, 106...post-processing unit, 108...drawing setting unit, 422...display control unit,

Claims

1. A host system, a controller, a touch sensor, an electrophoretic display and an input device are provided, the touch sensor overlies the electrophoretic display and detects a position of contact with the input device; The controller samples the touch position; The host system generates a trajectory of the touch location; determining whether or not drawing is required based on whether or not the contact position is included in a drawing area set by the application being executed, and outputting drawing control information indicating whether or not drawing is required to the controller via the input device; The controller: determining whether or not a line segment from the touch position of the previous sample to the touch position of the current sample should be displayed on the electrophoretic display based on the drawing control information; when it is determined that display is necessary, before acquiring a portion of the trajectory from the contact position of the previous sample to the contact position of the current sample, a line segment from the contact position of the previous sample to the contact position of the current sample is displayed on the electrophoretic display; The input device vibrates when a drawing control signal indicating a drawing request is input. Information processing system.

2. The controller displays the line segment with a width narrower than that of the locus. The information processing system according to claim 1 .

3. The controller displays the line segment in the same line style as the trajectory. The information processing system according to claim 1 .

4. The controller erases a portion of the line segment displayed on the electrophoretic display that deviates from the locus. The information processing system according to claim 2 .

5. the drawing control information includes drawing mode information indicating a drawing mode for expressing the trajectory, The controller causes the line segment to be displayed in accordance with the drawing mode. The information processing system according to claim 1 .

6. A host system, a controller, a touch sensor and an electrophoretic display are provided, the touch sensor overlies the electrophoretic display and detects a position of contact with an input device; The controller samples the touch location from the touch sensor; The host system generates a trajectory of the touch location; determining whether or not drawing is required based on whether or not the contact position is included in a drawing area set by the application being executed, and outputting drawing control information indicating whether or not drawing is required to the controller via the input device; The controller: determining whether or not a line segment from the touch position of the previous sample to the touch position of the current sample should be displayed on the electrophoretic display based on the drawing control information; when it is determined that display is necessary, before acquiring a portion of the trajectory from the contact position of the previous sample to the contact position of the current sample, a line segment from the contact position of the previous sample to the contact position of the current sample is displayed on the electrophoretic display; The input device vibrates when a drawing control signal indicating a drawing request is input. Information processing device.

7. A host system, a controller, a touch sensor, an electrophoretic display and an input device are provided, 1. A control method for an information processing system in which the touch sensor overlaps the electrophoretic display, comprising: The touch sensor detects a contact position with the input device; The controller samples the touch position; The host system generates a trajectory of the touch location; determining whether or not drawing is required based on whether or not the contact position is included in a drawing area set by the application being executed, and outputting drawing control information indicating whether or not drawing is required to the controller via the input device; The controller: determining whether or not a line segment from the touch position of the previous sample to the touch position of the current sample should be displayed on the electrophoretic display based on the drawing control information; when it is determined that display is necessary, before acquiring a portion of the trajectory from the contact position of the previous sample to the contact position of the current sample, a line segment from the contact position of the previous sample to the contact position of the current sample is displayed on the electrophoretic display; The input device vibrates when a drawing control signal indicating a drawing request is input. Control methods.

Images