Display device, display method, and program

Abstract

To improve operability in display of characters through voice recognition.SOLUTION: A display comprises: a voice processing unit that, when a first touch operation to a screen is detected, receives input of voice and converts the input voice into characters through voice recognition; and a display control unit performs control of displaying the characters obtained through the conversion at a position on the screen where the first touch operation is received. When a second touch operation to the characters displayed through the voice recognition is detected, the voice processing unit receives the second touch operation as an operation to correct the characters.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a display device, a display method, and a program. [Background technology]

[0002] A display device is disclosed that accepts voice input, converts the input voice into characters by voice recognition, and displays the characters.

[0003] For example, Patent Document 1 discloses an input display device that determines display attributes for characters related to text data of a voice received by a voice receiving means, and displays the characters based on the determined display attributes. The display attributes include, for example, the display start and end positions, font size, vertical or horizontal writing, etc., of a character or character string to be displayed. Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional technology, when displaying characters using voice recognition, operations such as specifying the timing to start voice recognition, specifying the position to display the characters, and correcting the displayed characters are complicated, resulting in low operability.

[0005] The disclosed technology aims to improve operability in displaying characters through voice recognition. [Means for solving the problem]

[0006] The disclosed technology includes a voice processing unit that, when a first touch operation on a screen is detected, receives a voice input and converts the input voice into characters by voice recognition, and a display control unit that performs control to display the converted characters at a position on the screen where the first touch operation is received. When a second touch operation on a character displayed by the voice recognition is detected, the voice processing unit receives the second touch operation as a correction operation for the character. When a first touch operation on the screen is detected, the operation mode for the correction function is switched to a correction ON mode, and when the operation mode for the correction function is the correction ON mode and the second touch operation on a character displayed by the voice recognition is detected, the second touch operation is accepted as a correction operation for the character, and when a certain time has elapsed since the start of the correction ON mode, the operation mode for the correction function is switched to a correction OFF mode. It is a display device. Effect of the Invention

[0007] The operability of displaying characters through voice recognition can be improved. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of an overall configuration of a display device. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device. [Diagram 3] FIG. 2 is a diagram showing an example of a perspective view of a pen. [Figure 4] FIG. 2 is a diagram illustrating an example of functions of a display device. [Diagram 5] FIG. 11 is a sequence diagram showing an example of a flow of audio processing. [Figure 6] FIG. 11 is a first diagram showing an example of a display screen related to audio processing. [Figure 7] FIG. 13 is a second diagram showing an example of a display screen related to audio processing. [Figure 8] FIG. 11 is a third diagram showing an example of a display screen related to audio processing. [Figure 9] FIG. 11 is a first diagram showing another configuration example of the display device. [Figure 10] FIG. 11 is a second diagram showing another configuration example of the display device. [Figure 11] FIG. 11 is a third diagram showing another configuration example of the display device. [Figure 12] FIG. 4 is a fourth diagram showing another configuration example of the display device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] First, the terminology used in this embodiment will be described. Handwritten data is data that displays a sequence of coordinate points on a display where the user has continuously moved the input tool as a trajectory. A series of operations in which the user presses the input tool against the display, continuously moves it, and then releases it from the display is called a stroke, and data handwritten using strokes is called stroke data. Stroke data can also be said to be a coordinate group that indicates a sequence of coordinate points.

[0011] The handwritten data includes one or more stroke data. Specifically, the handwritten data is a collection of multiple stroke data in which the time from the end of a stroke to the start of the next stroke, i.e., the time from the release of the input means from the display to the contact, is shorter than a predetermined time. Therefore, the handwritten data includes a coordinate group indicated by the stroke data included in the handwritten data.

[0012] The input means is a means for inputting handwritten data by specifying coordinates on the display. Examples of the input means include a pen, a human finger or hand, and a rod-shaped member.

[0013] Next, the configuration of the display device will be described.

[0014] FIG. 1 is a diagram showing an example of an overall configuration of a display device.

[0015] 1(a) shows, as an example of the display device 2, a display device 2 used as a horizontally long electronic whiteboard hung on a wall.

[0016] (b) shows a display device 2 used as a vertically long electronic whiteboard suspended on a wall.

[0017] (c) shows the display device 2 placed flat on a desk 230. The display device 2 is about 1 cm thick, so there is no need to adjust the height of the desk even if it is placed flat on a normal desk. In addition, it can be easily moved.

[0018] The display device 2 is provided with a display 220 as an example of a display unit. A user U can use a pen 2500 to handwrite (also referred to as input or drawing) characters or the like on the display 220.

[0019] FIG. 2 is a diagram illustrating an example of a hardware configuration of the display device.

[0020] The display device 2 has the configuration of an information processing device or a computer. The display device 2 includes a central processing unit (CPU) 201, a read only memory (ROM) 202, a random access memory (RAM) 203, and a solid state drive (SSD) 204.

[0021] The CPU 201 controls the overall operation of the display device 2. The ROM 202 stores the CPU 201 and programs used to drive the CPU 201, such as an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201. The SSD 204 stores various data, such as programs for the display device 2.

[0022] The display device 2 also includes a display controller 213, a touch sensor controller 215, a touch sensor 216, a display 220, a power switch 227, a tilt sensor 217, a serial interface 218, a speaker 219, a microphone 221, a wireless communication device 222, an infrared I / F 223, a power control circuit 224, an AC adapter 225, and a battery 226.

[0023] The display controller 213 controls and manages the screen display in order to output an output image to the display 220, etc. The touch sensor 216 detects that the pen 2500, the user's hand, etc. (the pen and the user's hand serve as input means) have touched the display 220. The touch sensor 216 also receives a pen ID.

[0024] The touch sensor controller 215 controls the processing of the touch sensor 216. The touch sensor 216 inputs and detects coordinates. Specifically, for example, in the case of an optical type, two light receiving and emitting devices installed at both ends of the upper side of the display 220 emit a plurality of infrared rays in parallel to the display 220. Then, the two light receiving and emitting devices receive light that is reflected by a reflecting member installed around the display 220 and returns along the same optical path as the optical path of the light emitted by the light receiving element.

[0025] The touch sensor 216 outputs position information of the infrared rays radiated by the two light receiving and emitting devices that are blocked by the object to the touch sensor controller 215. The touch sensor controller 215 identifies the coordinate position that is the contact position of the object. The touch sensor controller 215 also has a communication unit 215a and can wirelessly communicate with the pen 2500.

[0026] For example, if communication is performed according to a standard such as Bluetooth (registered trademark), a commercially available pen can be used. If one or more pens 2500 are registered in advance in the communication unit 215a, the user can communicate with the display device 2 without performing connection settings to allow the pen 2500 to communicate with the display device 2.

[0027] The power switch 227 is a switch for switching ON / OFF the power of the display device 2. The tilt sensor 217 is a sensor for detecting the tilt angle of the display device 2. The display device 2 can automatically change the thickness of characters, etc., depending on the installation state measured by the tilt sensor 217.

[0028] The serial interface 218 is an external communication interface such as a USB, and inputs information from the outside. The speaker 219 outputs sound. The microphone 221 inputs sound. The wireless communication device 222 communicates with a terminal carried by the user, and relays a connection to the Internet, for example. The wireless communication device 222 communicates according to standards such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), but may communicate according to any standard. The wireless communication device 222 forms an access point, and when the user sets an SSID (Service Set Identifier) ​​and password obtained by the user in a terminal carried by the user, the terminal can connect to the wireless communication device 222 forming the access point.

[0029] The wireless communication device 222 has, for example, the following two access points formed. a. Access point → Internet b. Access point → Internal network → Internet Access point a is for users outside the company. Terminals connected to access point a cannot access the company network, but can use the Internet. Access point b is for users inside the company. Terminals connected to access point b can use the company network and the Internet.

[0030] The infrared I / F 223 detects the display device 2 arranged adjacently. By utilizing the linearity of infrared rays, only the display device 2 arranged adjacently can be detected. It is preferable that one infrared I / F 223 is provided on each side, and it is possible to detect in which direction of a display device 2 another display device 2 is arranged. This expands the screen, and previously handwritten information (handwritten information of another page, with the area of ​​one display 220 being one page) can be displayed on the adjacent display device 2.

[0031] The power supply control circuit 224 controls an AC adapter 225 and a battery 226 which are power sources for the display device 2. The AC adapter 225 converts AC common to commercial power sources into DC.

[0032] If the display 220 is so-called electronic paper, it consumes little or no power to maintain the image display, and therefore can be driven by a battery 226. This makes it possible to use the display device 2 for purposes such as digital signage even in places where it is difficult to connect to a power source, such as outdoors.

[0033] The display device 2 further includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 201 shown in FIG.

[0034] The touch sensor 216 is not limited to an optical type, and may include various detection means such as a capacitance type detection means that identifies the contact position by detecting a change in capacitance, a resistive film type detection means that identifies the contact position by a voltage change between two opposing resistive films, or an electromagnetic induction type detection means that identifies the contact position by detecting electromagnetic induction caused by a contact object touching the display unit. The touch sensor 216 may be of a type that does not require an electronic pen to detect the presence or absence of a touch with the tip of the pen. In this case, a fingertip or a pen-shaped stick can be used to perform a touch operation. The pen 2500 does not have to be in the shape of a long, thin pen.

[0035] FIG. 3 is a diagram showing an example of a perspective view of a pen.

[0036] A pen 2500 that has a built-in power source and can send commands to the display device 2 is called an active pen (a pen that does not have a built-in power source is called a passive pen). The pen 2500 in Fig. 3 has one physical switch at the tip of the pen, one at the end of the pen, and two on each side of the pen. The switch at the tip of the pen is for writing, the end of the pen is for erasing, and the switch on the side of the pen is for allocating user functions. The pen 2500 of this embodiment has a non-volatile memory and stores a pen ID that is unique to other pens.

[0037] A pen with a switch can reduce the number of steps a user must take to operate the display device 2. Although the term "pen with a switch" primarily refers to an active pen, in the case of the electromagnetic induction method, even a passive pen without a built-in power supply can generate power using only an LC circuit, so it includes not only active pens but also passive pens using the electromagnetic induction method. Pens with a switch that use optical, infrared, and capacitive methods other than the electromagnetic induction method are all active pens.

[0038] The hardware configuration of the pen 2500 is similar to that of a general control system equipped with a communication function and a microcomputer. The input method of the coordinates of the pen 2500 includes an electromagnetic induction method and an active electrostatic coupling method. The pen 2500 may also have functions such as pen pressure detection, tilt detection, and a hover function (displaying the cursor before the pen touches the screen).

[0039] FIG. 4 is a diagram illustrating an example of functions of the display device.

[0040] The display device 2 includes a contact position detection unit 21, a drawing data generation unit 22, a character recognition unit 23, a contact processing unit 24, a voice processing unit 25, a display control unit 26, a data recording unit 27, a network communication unit 28, an operation reception unit 29, and a memory unit 30.

[0041] Each unit of the display device 2 is realized by any of the components shown in FIG. 2 operating in response to an instruction from the CPU 201 in accordance with a program loaded from the SSD 204 onto the RAM 203.

[0042] The contact position detection unit 21 detects the coordinates of the position where the tip of the pen 2500 touches the display 220. The contact position detection unit 21 is realized by a touch sensor controller 215 and a touch sensor 216. The contact position detection unit 21 is an example of a receiving unit that receives input of handwritten data by an input unit.

[0043] The drawing data generating unit 22 acquires the coordinates where the tip of the pen 2500 touches the display 220 from the contact position detecting unit 21 multiple times at short time intervals, and generates drawing data showing a connected image by interpolating the multiple acquired coordinates. That is, the drawing data generating unit 22 generates drawing data showing an image based on handwritten data.

[0044] The character recognition unit 23 performs character recognition processing on one or more stroke data (handwritten data) handwritten by the user, and converts it into a character code. The character recognition unit 23 recognizes characters (not only Japanese but also multiple languages ​​such as English), numbers, symbols (%, $, &, etc.), figures (lines, circles, triangles, etc.), etc. in parallel with the user's pen operation. Various algorithms have been devised for the recognition method, but details will be omitted since known techniques can be used in this embodiment.

[0045] When the contact position detection unit 21 detects the position where the tip of the pen 2500 touches the display 220, the contact processing unit 24 executes processing defined according to the operation mode of each function. Specifically, the operation mode for the voice recognition function is switched between a voice recognition ON mode and a voice recognition OFF mode. Also, the operation mode for the correction function for correcting characters by voice recognition is switched between a correction ON mode and a correction OFF mode. When the contact processing unit 24 detects a touch operation on the display 220 in the voice recognition OFF mode and the correction OFF mode, it switches to the voice recognition ON mode and the correction ON mode.

[0046] Here, a touch operation is an operation of touching the tip of the pen 2500 to the display 220 for a short time equal to or shorter than a reference time. In contrast, an operation of touching the tip of the pen 2500 to the display 220 for a long time exceeding the reference time is called a long press operation. The reference time that serves as a criterion for determining whether an operation is a touch operation or a long press operation is preset, for example, one second. This reference time may be changeable.

[0047] When the operation mode for the correction function is the correction ON mode, upon detecting a touch operation on a character by voice recognition, the contact processing unit 24 requests the voice processing unit 25 to start correcting the character. Then, upon detecting a touch operation on a correction candidate displayed on the screen under the control of the voice processing unit 25, the contact processing unit 24 notifies the voice processing unit 25 of the completion of selection of the correction candidate.

[0048] Furthermore, the contact processing unit 24 switches the operation mode of the correction function to the correction OFF mode after a certain period of time (hereinafter, referred to as Tb) has elapsed since the start of the correction ON mode.

[0049] The contact processing unit 24 may manage the operation modes for the voice recognition function or the correction function separately each time voice recognition is performed. That is, if the correction ON mode continues after switching to the voice recognition OFF mode, and then if the voice recognition ON mode is switched to by a touch operation on another position, the operation modes for the two correction functions become the correction ON mode simultaneously. In this way, the contact processing unit 24 manages each touch operation to start voice recognition as an individual voice recognition function, and also manages the operation modes for the corresponding correction functions individually in association with the voice recognition functions.

[0050] The voice processing unit 25 executes various processes related to voice input according to the operation mode of each function. Specifically, when the operation mode for the voice recognition function is the voice recognition ON mode, the voice processing unit 25 accepts voice input via the microphone 221, converts it into characters by performing voice recognition processing, and passes the converted characters to the display control unit 26.

[0051] Furthermore, when the voice processing unit 25 is requested by the contact processing unit 24 to start correcting characters, the voice processing unit 25 passes the correction candidates to the display control unit 26 .

[0052] When the operation mode for the voice recognition function is the voice recognition ON mode, the voice processing unit 25 switches the operation mode for the voice recognition function to the voice recognition OFF mode after a certain time (hereinafter, Ta) has elapsed since the start of the voice recognition ON mode. When the operation mode for the voice recognition function is the voice recognition OFF mode, the voice processing unit 25 does not perform any particular processing even if it receives voice input via the microphone 221.

[0053] The display control unit 26 displays on the display 220 an image based on the handwritten data, an operation menu for the user to operate, characters to be passed to the voice processing unit 25, correction candidates, and the like.

[0054] The data recording unit 27 records in the storage unit 30 handwritten data handwritten on the display device 2, character strings converted by character recognition or voice recognition, information indicating the break positions of phrases in voice recognition, and the like.

[0055] The network communication unit 28 is connected to a network such as a LAN (Local Area Network), and transmits and receives data to and from other devices via the network.

[0056] The operation reception unit 29 receives operations by input means such as a pen 2500 on operation buttons and the like displayed on the display 220 .

[0057] The storage unit 30 is constructed in the SSD 204, the RAM 203, or the like, and stores the above information recorded by the data recording unit 27.

[0058] FIG. 5 is a sequence diagram showing an example of the flow of audio processing.

[0059] When the pen tip of the pen 2500 touches the display 220 of the display device 2 by a user's operation (step S101), the contact processing unit 24 detects a touch operation (first touch operation) at the contact position detected by the contact position detection unit 21. Then, the contact processing unit 24 switches the operation mode of the voice recognition function to a voice recognition ON mode.

[0060] The contact processing unit 24 notifies the voice processing unit 25 that the operation mode for the voice recognition function has been switched to the voice recognition ON mode (i.e., voice recognition has started) together with information indicating the contact position (step S102). Upon receiving the notification of the start of voice recognition, the voice processing unit 25 stores the information indicating the contact position in the storage unit 30 via the data recording unit 27 as the start coordinates for starting display of characters (step S103).

[0061] Next, the contact processing unit 24 switches the operation mode for the correction function to the correction ON mode (step S104). Then, the contact processing unit 24 starts a timer to detect the passage of a certain period of time (Ta and Tb) from the start of the voice recognition ON mode and the correction ON mode (step S105).

[0062] Next, when the voice processing unit 25 receives voice input from the user (step S106), it executes voice recognition processing and converts it into characters. Here, the voice processing unit 25 stores the delimitation positions of phrases or words by voice recognition and candidates of characters to be converted into each phrase or word in the memory unit 30 via the data recording unit 27. It is desirable that the candidates of characters to be stored include not only the first candidate but also the second candidate, the third candidate, etc.

[0063] Then, the voice processing unit 25 transfers the converted characters to the display control unit 26 and requests drawing of the characters (step S107). The display control unit 26 controls the display 220 to display the transferred characters at the position of the start coordinates stored in the storage unit 30 (step S108).

[0064] The voice processing unit 25 and the display control unit 26 execute the same processes as steps S106-S108 every time they receive voice input from the user (steps S109-S111).

[0065] When the timer detects the elapse of a fixed time (Ta), the voice processing unit 25 switches the operation mode for the voice recognition function to the voice recognition OFF mode (step S112). Note that Ta and Tb are set in advance, and Ta < Tb is set. Therefore, here, the operation mode for the correction function is not switched to the correction OFF mode.

[0066] Next, when the tip of the pen 2500 touches the character drawn on the display 220 by the user's operation (step S113), the contact processing unit 24 detects a touch operation (second touch operation) at the contact position detected by the contact position detection unit 21. Then, the contact processing unit 24 requests the voice processing unit 25 to start the correction process of the character drawn at the contact position detected by the contact position detection unit 21 (step S114).

[0067] The voice processing unit 25 sets the word including the character displayed at the position touched by the tip of the pen 2500 as the correction target. Note that the voice processing unit 25 may specify the correction target by other methods. For example, the word displayed at the position of the line (strikethrough) written by the tip of the pen 2500 may be set as the correction target. Also, the word displayed in the area surrounded by the line written by the tip of the pen 2500 may be set as the correction target.

[0068] Note that the voice processing unit 25 determines the break between words or phrases based on the break positions of the phrases or words stored in the storage unit 30 during voice recognition. For example, for the input of the voice "じゅうようじこうのせつめい", when the break positions of the words "じゅうよう", "じこう", "の", and "せつめい" are stored during voice recognition, based on the stored break positions, the word "じゅうよう" is set as the correction target.

[0069] The voice processing unit 25 may determine the breaks between words or phrases by other methods. For example, the voice processing unit 25 may determine the breaks between words by judging whether a character string including, for example, one to four characters centered around the touched character is suitable as a word. For example, if the touched character is "yo" in "juuyou", the voice processing unit 25 may determine the appropriate word "juuyou" as the break between words from among break candidates such as "you", "uuyou", "uuyou", and "juuyou".

[0070] Next, the voice processing unit 25 transfers the correction candidates to the display control unit 26 and requests display (step S115). The voice processing unit 25 sets as correction candidates, among the character candidates stored in the storage unit 30 at the time of voice recognition, candidates other than the one currently displayed (usually the first candidate) (for example, the second candidate, the third candidate, etc.). For example, when information indicating that "weight" was determined as the first candidate, "important" as the second candidate, "weight-dependent" as the third candidate, and "medium-weight" as the fourth candidate at the time of voice recognition is stored in the storage unit 30, the voice processing unit 25 sets as correction candidates, among these, "important", "weight-dependent" and "medium-weight", excluding the first candidate "weight" to be corrected. The voice processing unit 25 may also determine correction candidates by other methods. For example, the voice processing unit 25 may output a word or phrase close to the word or phrase for which a break has been determined, using a voice recognition engine.

[0071] The display control unit 26 performs control to display the passed correction candidates (step S116).

[0072] Next, when the pen tip of the pen 2500 touches the correction candidate drawn on the display 220 by the user's operation (step S117), the contact processing unit 24 detects a touch operation (third touch operation) at the contact position detected by the contact position detection unit 21. The contact processing unit 24 notifies the voice processing unit 25 of the completion of selection of the correction candidate drawn at the contact position detected by the contact position detection unit 21 (step S118).

[0073] The voice processing unit 25 passes the corrected character string to the display control unit 26 and requests it to be drawn (step S119). The display control unit 26 controls the drawing of the passed character string (step S120).

[0074] When the timer detects that a certain period of time (Tb) has elapsed, the contact processing unit 24 switches the operation mode for the correction function to a correction OFF mode (step S121).

[0075] FIG. 6 is a first diagram showing an example of a display screen related to audio processing.

[0076] Fig. 6(a) shows the display 220 in a state where the tip of the pen 2500 touches the voice input start position (Fig. 5; step S101). Fig. 6(b) shows the display 220 in a state where the voice input "Explanation of important matters" has been input (Fig. 5; step S106). Note that the display 220 shows "weight" instead of "important" as a result of the voice recognition process.

[0077] Fig. 6(c) shows the display 220 in a state where the tip of the pen 2500 touches the correction location (Fig. 5; step S113). Fig. 6(d) shows the display 220 in a state where correction candidates are displayed (Fig. 5; step S116). Fig. 6(e) shows the display 220 in a state where the correction is completed (Fig. 5; step S120).

[0078] According to the display device 2 of this embodiment, when a first touch operation on the screen is detected, the voice processing unit 25 accepts voice input and converts the input voice into characters by voice recognition. This allows voice recognition to be started with a simple operation. Furthermore, the display control unit 26 performs control to display the converted characters at the position on the screen where the first touch operation was received. This allows the location where the voice recognition result is to be displayed to be specified with a simple operation.

[0079] Furthermore, in the display device 2 according to the present embodiment, when a second touch operation on a character displayed by voice recognition is detected, the voice processing unit 25 accepts the second touch operation as a character correction operation. This allows the result of voice recognition to be corrected by a simple operation.

[0080] In the case where a touch operation is distinguished from a long press operation, a touch operation is an operation that can obtain a quicker response than a long press operation, and each of the above-mentioned operations is an intuitive operation. Therefore, it is possible to improve the operability in character display by voice recognition.

[0081] The correction ON mode ends after a certain period of time (Tb). This limits the time during which the above-mentioned touch operation is treated as a correction to the time when the operation is likely to be performed. If such a limitation is not imposed, even when a new character is to be input, the operation is likely to be treated as a correction. This further improves operability.

[0082] The voice processing unit 25 may switch the operation mode of the voice recognition function to a voice recognition OFF mode when a phrase boundary is detected by voice recognition or a certain period of time has elapsed since a voice pause.

[0083] In addition, the contact processing unit 24 may switch the operating mode for the correction function to the correction OFF mode after a certain period of time has elapsed since switching from the voice recognition ON mode to the voice recognition OFF mode, or after a period of time corresponding to the amount of characters converted by voice recognition has elapsed.

[0084] Furthermore, the display device 2 may display a menu or the like indicating switching to the correction OFF mode or the voice recognition OFF mode, and the switching may be performed by an explicit operation of the user.

[0085] FIG. 7 is a second diagram showing an example of the display screen related to the voice processing.

[0086] As shown in Fig. 7(d), in the process of step S115 in Fig. 5, the voice processing unit 25 may request the display control unit 26 to control displaying a frame line for accepting handwritten input instead of displaying correction candidates. In this case, when handwritten input is accepted, the character recognition unit 23 recognizes characters from the input handwritten data and requests the display control unit 26 to control displaying the characters as corrected characters.

[0087] FIG. 8 is a third diagram showing an example of the display screen related to the voice processing.

[0088] As shown in Fig. 8(d), the voice processing unit 25 accepts voice input instead of displaying correction candidates in the process of step S115 in Fig. 5. In this case, when voice input is accepted, the voice processing unit 25 executes voice recognition processing on the input voice, requests the display control unit 26 to control displaying the corrected characters, and stores the result in the storage unit 30.

[0089] FIG. 9 is a first diagram showing another configuration example of the display device.

[0090] In Fig. 9, a projector 411 is installed on the upper side of a normal whiteboard 413. This projector 411 corresponds to the display device 2. The normal whiteboard 413 is not a flat panel display integrated with a touch panel, but a whiteboard on which a user directly writes by hand with a marker. The whiteboard may be a blackboard, and may be any flat surface large enough to project an image.

[0091] Projector 411 has an ultra-short focal length optical system and can project an image with little distortion from a distance of about 10 cm onto whiteboard 413. This image may be transmitted from PC 400-1 connected wirelessly or via a wire, or may be stored in projector 411.

[0092] A user writes by hand on the whiteboard 413 using a dedicated electronic pen 2501. The electronic pen 2501 has a light-emitting part, for example at the tip, that is switched on and emits light when the user presses it against the whiteboard 413 to write by hand. The wavelength of the light is near-infrared or infrared, so it is invisible to the user's eyes. The projector 411 has a camera that captures the light-emitting part, analyzes the image, and identifies the direction of the electronic pen 2501. The electronic pen 2501 also emits sound waves along with the light emission, and the projector 411 calculates the distance based on the arrival time of the sound waves. The position of the electronic pen 2501 can be identified based on the direction and distance. Handwritten data is drawn (projected) at the position of the electronic pen 2501.

[0093] Since the projector 411 projects the menu 430, when the user presses a button with the electronic pen 2501, the projector 411 identifies the pressed button by the position of the electronic pen 2501 and the ON signal of the switch. For example, when the save button 431 is pressed, the handwritten data (coordinate point sequence) handwritten by the user is saved in the projector 411. The projector 411 saves the handwritten information in a predetermined server 412 or USB memory 2600. The handwritten information is saved for each page. Since the information is saved as coordinates rather than as image data, the user can re-edit it. However, in this embodiment, the menu 430 does not need to be displayed because an operation command can be called up by handwriting.

[0094] FIG. 10 is a second diagram showing another configuration example of the display device.

[0095] The display device 2 includes a terminal device 600, an image projection device 700A, and a pen operation detection device 810.

[0096] The terminal device 600 is connected by wire to the image projection device 700A and the pen operation detection device 810. The image projection device 700A projects image data input by the terminal device 600 onto the screen 800.

[0097] The pen operation detection device 810 communicates with the electronic pen 820 and detects the operation of the electronic pen 820 in the vicinity of the screen 800. Specifically, the electronic pen 820 detects coordinate information indicating a point pointed to by the electronic pen 820 on the screen 800 (the detection method may be the same as that in FIG. 9 ), and transmits the coordinate information to the terminal device 600.

[0098] The terminal device 600 generates image data of handwritten data input by the electronic pen 820 based on the coordinate information received from the pen operation detection device 810, and causes the image projection device 700A to draw an image of the handwritten data on the screen 800.

[0099] Furthermore, the terminal device 600 generates superimposed image data indicating a superimposed image obtained by combining the background image projected by the image projection device 700A and the image of the handwritten data input by the electronic pen 820.

[0100] FIG. 11 is a third diagram showing another configuration example of the display device.

[0101] In the example of FIG. 11, the display device 2 includes a terminal device 600, a display 800A, and a pen operation detection device 810.

[0102] The pen operation detection device 810 is disposed near the display 800A, detects coordinate information indicating a point indicated by the electronic pen 820A on the display 800A (the detection method may be the same as that in FIG. 26), and transmits the coordinate information to the terminal device 600. In the example of FIG. 28, the electronic pen 820A may be charged by the terminal device 600 via a USB connector.

[0103] Based on the coordinate information received from the pen operation detection device 810, the terminal device 600 generates image data indicative of an image of handwritten data input by the electronic pen 820A, and displays the image data on the display 800A.

[0104] FIG. 12 is a fourth diagram showing another configuration example of the display device.

[0105] In the example of FIG. 12, the display device 2 includes a terminal device 600 and an image projection device 700A.

[0106] The terminal device 600 performs wireless communication (Bluetooth (registered trademark) or the like) with the electronic pen 820B to receive coordinate information of a point indicated by the electronic pen 820B on the screen 800. The coordinate information may be obtained by the electronic pen 820B reading minute position information formed on the screen 800, or the coordinate information may be received from the screen 800.

[0107] Then, the terminal device 600 generates image data of an image of the handwritten data input by the electronic pen 820B based on the received coordinate information, and causes the image projection device 700A to project the image of the handwritten data.

[0108] Furthermore, the terminal device 600 generates superimposed image data indicating a superimposed image obtained by combining the background image projected by the image projection device 700A and the image of the handwritten data input by the electronic pen 820.

[0109] As described above, each of the above-described embodiments can be applied to various system configurations.

[0110] The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0111] The character string is stored as character codes, and the handwritten data is stored as coordinate point data in the display device 2. The data can also be stored in various storage media or in a storage device on a network, and later downloaded from the display device 2 for reuse. The display device 2 to be reused may be any display device, or may be a general information processing device. Therefore, the user can reproduce the handwritten content on a different display device 2 and continue a meeting, etc.

[0112] In this embodiment, the coordinates of the pen tip are detected by a touch panel, but the coordinates of the pen tip may be detected by ultrasonic waves. The pen emits light and also emits ultrasonic waves, and the display device 2 calculates the distance based on the arrival time of the ultrasonic waves. The position of the pen can be identified based on the direction and distance. The projector draws (projects) the trajectory of the pen as stroke data.

[0113] Moreover, the configuration example in FIG. 4 and the like is divided according to main functions in order to facilitate understanding of the processing by the display device 2. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing by the display device 2 can be divided into more processing units according to the processing contents. Also, it can be divided so that one processing unit includes more processes.

[0114] Furthermore, part of the processing performed by the display device 2 may be performed by a server connected to the display device 2 via a network.

[0115] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a conventional circuit module designed to execute each function described above.

[0116] Each function of the above-described embodiment can be realized by an application program that runs on an OS (Operating System) that defines the operation of the CPU 201 included in the display device 2.

[0117] The device to which the present embodiment is applied may be any display device having a function of operating an object by touch operation, and is not limited to an electronic whiteboard. The display device to which the present embodiment is applied may be, for example, a PJ (Projector), an output device such as digital signage, a HUD (Head Up Display) device, a network home appliance, an automobile (Connected Car), a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game machine, a PDA (Personal Digital Assistant), a wearable PC, etc.

[0118] Although the present invention has been described above based on the embodiments, the present invention is not limited to the requirements shown in the above embodiments. These points can be changed without departing from the spirit of the present invention, and can be appropriately determined according to the application form. [Explanation of symbols]

[0119] 2 Display device 21 Contact position detection unit 22 Drawing data generation unit 23 Character recognition section 24 Contact treatment section 25 Audio Processing Unit 26 Display control section 27 Data Recording Section 28 Network Communication Department 29 Operation reception section 30 Storage section [Prior art documents] [Patent documents]

[0120] [Patent Document 1] JP 2012-088969 A

Claims

1. a voice processing unit that receives a voice input and converts the input voice into text by voice recognition when a first touch operation on the screen is detected; a display control unit that performs control to display the converted character at a position on the screen where the first touch operation is received, When a second touch operation on a character displayed by the voice recognition is detected, the voice processing unit accepts the second touch operation as a correction operation of the character, and when a first touch operation on the screen is detected, the voice processing unit switches an operation mode for a correction function to a correction ON mode, and when the operation mode for the correction function is the correction ON mode, when the second touch operation on a character displayed by the voice recognition is detected, the voice processing unit accepts the second touch operation as a correction operation of the character, and when a certain time has elapsed since the start of the correction ON mode, switches the operation mode for the correction function to a correction OFF mode. Display device.

2. when a second touch operation on the character displayed by the voice recognition is detected, the display control unit displays a correction candidate for the character. The display device according to claim 1 .

3. when a second touch operation on the character displayed by the voice recognition is detected, the voice processing unit accepts a handwritten input as a correction to the character. The display device according to claim 1 .

4. when a second touch operation on the character displayed by the voice recognition is detected, the voice processing unit accepts a voice input as a correction to the character and converts it into the corrected character by voice recognition. The display device according to claim 1 .

5. 1. A computer-implemented method comprising: receiving a voice input when a first touch operation on the screen is detected, and converting the input voice into text by voice recognition; performing control to display the converted character at a position on the screen where the first touch operation is received; when a second touch operation on a character displayed by the voice recognition is detected, the second touch operation is accepted as a correction operation of the character; when a first touch operation on the screen is detected, an operation mode for a correction function is switched to a correction ON mode; when the operation mode for the correction function is the correction ON mode, when the second touch operation on a character displayed by the voice recognition is detected, the second touch operation is accepted as a correction operation of the character; and when a certain time has elapsed since the start of the correction ON mode, the operation mode for the correction function is switched to a correction OFF mode. Display method.

6. On the computer, receiving a voice input when a first touch operation on the screen is detected, and converting the input voice into text by voice recognition; performing control to display the converted character at a position on the screen where the first touch operation is received; when a second touch operation on a character displayed by the voice recognition is detected, accepting the second touch operation as a correction operation of the character; when a first touch operation on the screen is detected, switching an operation mode for a correction function to a correction ON mode; when the operation mode for the correction function is the correction ON mode, when the second touch operation on a character displayed by the voice recognition is detected, accepting the second touch operation as a correction operation of the character; and when a certain time has elapsed since the start of the correction ON mode, switching the operation mode for the correction function to a correction OFF mode; A program for executing.

Images