Display device, display method, and display system

Abstract

To provide a display device for recording an external image displayed thereby as a video image.SOLUTION: A display device capable of displaying an externally fed external image is provided, comprising a handwriting data input unit for receiving an input of handwriting data, a display control unit for displaying the external image and the handwriting data, and a recorded data generation unit configured to record the external image as a video image according to the handwriting data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a display device, a display method, and a display system. [Background technology]

[0002] Display devices such as electronic whiteboards are known, which display handwritten data drawn by a user using a special electronic pen or finger on a touch panel display. Unlike conventional whiteboards, handwritten data can be saved as electronic data, and the display device can be connected to an external device such as a PC (Personal Computer) to display images displayed by the external device.

[0003] A technology has been devised that allows a display device to save an image from an external device as a still image (see, for example, Patent Document 1). Patent Document 1 discloses a display system that allows the touch panel function to be easily used while the external device is connected to the display device. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technologies have had the problem of making it difficult to record external images displayed on a display device as video. For example, when an external device transmits video to a display device, a user must repeatedly save still images one by one in order to save the video on the display device. Even if such operations are performed, frame drops often occur, and the saved still images often do not appear to be video.

[0005] In view of the above-mentioned problems, an object of the present invention is to provide a display device that records an external image displayed on the display device as a moving image. [Means for solving the problem]

[0006] In view of the above problem, the present invention provides a method for converting an external image input from an external device into a digital still image. On the displaya handwritten data input unit that accepts input of handwritten data; A part of the display The external image and displaying a display control unit that displays the handwritten data; and a recording data generation unit that records the external video as a video based on the handwritten data. an instruction generation unit that generates instructions to start and stop recording for the recording data generation unit based on the handwritten data; , a superimposed image generating unit that generates a superimposed image in which the handwritten data is superimposed on the external video image; and, the instruction generation unit generates an instruction to the recording data generation unit to start or stop recording of the external video only when the handwritten data is superimposed on the external video. It is characterized by: [Effects of the Invention]

[0007] It is possible to provide a display device that records an external image displayed on the display device as a moving image. [Brief explanation of the drawings]

[0008] [Figure 1] 10A and 10B are diagrams illustrating an operation method when the display device stores a video. [Figure 2] 1 is an example of an overall configuration diagram of a communication system. [Figure 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device. [Figure 4] 3 is a diagram illustrating an example of a configuration of each image layer displayed by a display device. FIG. [Figure 5] FIG. 10 is a diagram illustrating an example of an external video display. [Figure 6] FIG. 2 is an example of a functional block diagram illustrating functions of the display device, divided into blocks. [Figure 7] 10 is a diagram schematically illustrating conversion information stored in a conversion information storage unit used by an instruction generation unit. FIG. [Figure 8] FIG. 1 is a diagram illustrating an example of a neural network in which L layers from the input layer to the output layer are fully connected. [Figure 9] FIG. 1 is an example of a sequence diagram illustrating an overall operation of a communication system. [Figure 10] 10 is an example of a flowchart illustrating a process in which the display device generates an instruction to start recording, starts recording, and generates an instruction to stop recording, and stops recording. FIG. [Figure 11]FIG. 1 is a diagram illustrating an example of the configuration of a display system. [Figure 12] 12 is a diagram showing an example of a functional block diagram of a display device and an information processing device in the display system of FIG. 11. FIG. [Figure 13] 1 is an example of a sequence diagram illustrating an overall operation of a communication system including an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, as an example of an embodiment for carrying out the present invention, a display device and a display method performed by the display device will be described with reference to an embodiment.

[0010] <Outline of the video saving method of this embodiment> An outline of the process in which the display device saves a video from an external device such as a PC will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining an operation method in which the display device 2 of this embodiment saves a video.

[0011] As shown in Fig. 1(a), when the external image displayed by an external device such as a PC is a moving image, the moving image can be displayed on the display device 2. In Fig. 1, a moving image of a soccer match is displayed.

[0012] As shown in FIG. 1(b), the user can write by hand on the video. In FIG. 1(b), the user inputs handwritten data 101 of circling a player of interest. The display device 2 detects this handwritten data 101 of the circle as handwritten data for starting recording. Therefore, the display device 2 starts recording the video.

[0013] In Fig. 1(c), the player of interest is moving, and the user indicates the player's movement with arrows and a box drawn around it in handwritten data 102. The display device 2 detects this boxed handwritten data 102 as handwritten data indicating the end of recording. Therefore, the display device 2 ends recording of the video.

[0014] In the past, when a user wanted to play a video on a PC and trace the movements of players or the ball by hand while the video was being displayed on the display device 2, they had to save each still image one by one, which was a cumbersome operation. Also, frames were dropped, making it impossible for the display device 2 to acquire consecutive frames like a video.

[0015] In contrast to this, in this embodiment, handwritten data is treated as command input, making it possible to start and stop recording, so that the same handwritten data can be used to start and stop recording and to save the trajectories that trace the movements of players and the ball.

[0016] <Terminology> The input means may be any means that allows handwriting by specifying coordinates on a touch panel, such as a pen, a human finger or hand, or a rod-shaped member.

[0017] A stroke is a series of operations in which a user presses an input device against a display, moves it continuously, and then releases it from the display. Stroke data is information displayed on a display based on the trajectory of coordinates input by the input device. Stroke data may be interpolated as appropriate. Handwritten data is data that has one or more strokes. Handwritten input refers to the input of handwritten data by a user.

[0018] The text data that is generated by character recognition of handwritten data is one or more characters that can be handled by a computer. The actual content of text data is character codes. Characters include numbers, alphabets, and symbols.

[0019] The text data may include stamps displayed as fixed characters such as "Done", marks, figures such as circles and stars, straight lines, and the like.

[0020] Saving an external video image transmitted from an external device and displayed on the display device 2 is sometimes called "importing" or "capturing." The external device may be any device other than the display device 2, such as a PC, a removable memory, or a server on a network.

[0021] A video is a series of continuous images that appear to move, as opposed to still images such as photographs. A video that appears to move naturally to the human eye is 24 fps (frames per second), but in this embodiment, a frame rate of less than 24 fps is acceptable, as long as it is at least a few fps that generally appears to be a video.

[0022] <System configuration example> Fig. 2 is a diagram showing the overall configuration of the communication system of this embodiment. Note that, for the sake of simplicity, Fig. 2 shows only two display devices 2a and 2b and accompanying electronic pens 4a and 4b, but three or more display devices 2 and electronic pens may be used.

[0023] 2, the communication system 1 includes a plurality of display devices 2a, 2b, a plurality of electronic pens 4a, 4b, USB memories 5a, 5b, notebook personal computers (PCs) 6a, 6b, video conference terminals 7a, 7b, and a PC 8. The display devices 2a, 2b, and the PC 8 are communicatively connected via a communication network 9. The display devices 2a, 2b are further provided with displays 3a, 3b, respectively.

[0024] Furthermore, the display device 2a can display an image drawn by an event generated by the electronic pen 4a (touching the display 3a with the tip or the butt of the electronic pen 4a). Note that the image displayed on the display 3a can also be changed based on an event generated not only by the electronic pen 4a but also by the user's hand Ha or the like (gestures such as zooming in, zooming out, turning a page, etc.).

[0025] A USB memory 5a can be connected to the display device 2a, and the display device 2a can read electronic files such as PDFs from the USB memory 5a and record electronic files to the USB memory 5a. A laptop PC 6a is connected to the display device 2a via a cable 10a1 capable of communication according to standards such as DisplayPort (registered trademark), DVI (Digital Visual Interface), HDMI (High-Definition Multimedia Interface, registered trademark), and VGA (Video Graphics Array). The display device 2a generates an event when the display 3a is touched, and transmits event information indicating this event to the laptop PC 6a in the same manner as events from input devices such as a mouse and keyboard. A video (television) conference terminal 7a is also connected to the display device 2a via a cable 10a2 capable of communication according to the above standards. The laptop PC 6a and the video conference terminal 7a may communicate with the display device 2a via wireless communication conforming to various wireless communication protocols such as Bluetooth (registered trademark).

[0026] Meanwhile, at another location where display device 2b is installed, similarly to the above, display device 2b equipped with display 3b, electronic pen 4b, USB memory 5b, notebook PC 6b, video conference terminal 7b, cable 10b1, and cable 10b2 are used. Furthermore, the image displayed on display 3b can be changed based on an event caused by the user's hand Hb or the like.

[0027] As a result, an image drawn on the display 3a of the display device 2a at one location is also displayed on the display 3b of the display device 2b at another location, and conversely, an image drawn on the display 3b of the display device 2b at another location is displayed on the display 3a of the display device 2a at the one location. In this way, the communication system 1 is capable of performing remote sharing processing to share the same image at remote locations, making it very convenient when used for remote conferences, etc.

[0028] In the following, any one of the multiple display devices 2 will be referred to as "display device 2." Any one of the multiple displays will be referred to as "display 3." Any one of the multiple electronic pens will be referred to as "electronic pen 4." Any one of the multiple USB memory sticks will be referred to as "USB memory stick 5." Any one of the multiple notebook PCs will be referred to as "notebook PC 6." Any one of the multiple video conference terminals will be referred to as "video conference terminal 7." Any one of the multiple user's hands will be referred to as "hand H." Any one of the multiple cables will be referred to as "cable 10."

[0029] In addition, in this embodiment, an electronic whiteboard is described as an example of the display device 2, but the present invention is not limited thereto. Other examples of the display device 2 may include an electronic billboard (digital signage), a telestrator used for sports or weather forecasts, or a remote image (video) diagnostic device. In addition, a notebook PC 6 is described as an example of an external device, but the present invention is not limited thereto. Other examples of the external device may include a terminal capable of supplying image frames, such as a desktop PC, a tablet PC, a PDA, a digital video camera, a digital camera, or a game console. Furthermore, communication networks include the Internet, a LAN (Local Area Network), a mobile phone communication network, and the like. In addition, in this embodiment, a USB memory is described as an example of a recording medium, but the present invention is not limited thereto. Other examples of the recording medium may include various recording media such as an SD card.

[0030] <Hardware configuration of display device> Next, the hardware configuration of the display device 2 of this embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing the hardware configuration of the display device 2. As shown in Fig. 3, the display device 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an SSD (Solid State Drive) 204, a network I / F 205, and an external device connection I / F (Interface) 206.

[0031] Of these, the CPU 201 controls the overall operation of the display device 2. The ROM 202 stores the CPU 201 and programs used to drive the CPU 201, such as an IPL (Initial Program Loader). The RAM 203 is used as a work area for the CPU 201.

[0032] The SSD 204 stores various data such as an OS and programs for the display device 2. This program may be an application program that runs on an information processing device equipped with a general-purpose OS (Windows (registered trademark), Mac OS (registered trademark), Android (registered trademark), iOS (registered trademark), etc.). Therefore, although the display device 2 is normally used as a general-purpose information processing device, when a user executes an application program, the user can write by hand on the display device 2 just as if it were a dedicated device.

[0033] The network I / F 205 controls communication with the communication network 9. The external device connection I / F 206 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory 5 and external devices (a microphone 240, a speaker 250, and a camera 260).

[0034] The display device 2 also includes a capture device 211, a GPU 212, a display controller 213, a contact sensor 214, a sensor controller 215, an electronic pen controller 216, a short-range communication circuit 219, an antenna 219a of the short-range communication circuit 219, a power switch 222, and selection switches 223.

[0035] Of these, the capture device 211 displays video information as still images or moving images on the display of the external notebook PC 6. The GPU (Graphics Processing Unit) 212 is a semiconductor chip that specializes in graphics. The display controller 213 controls and manages the screen display to output the output image from the GPU 212 to the display 3, etc. The contact sensor 214 detects contact of the electronic pen 4, the user's hand H, etc. on the display 3. The sensor controller 215 controls the processing of the contact sensor 214. The contact sensor 214 inputs and detects coordinates using an infrared blocking method. This coordinate input and coordinate detection method involves two light receiving and emitting devices installed at both ends of the upper side of the display 3 emitting multiple infrared rays parallel to the display 3, which are reflected by reflecting members installed around the display 3 and return along the same optical path as the light emitted by the light receiving element. The contact sensor 214 outputs the ID of the infrared light emitted by the two light receiving and emitting devices that has been blocked by an object to the sensor controller 215, and the sensor controller 215 identifies the coordinate position of the contact position of the object. The electronic pen controller 216 determines whether the pen tip or the pen tail has touched the display 3 by communicating with the electronic pen 4. The short-range communication circuit 219 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 222 is a switch for switching the power of the display device 2 on and off. The selection switches 223 are a group of switches for adjusting, for example, the brightness and color of the display 3.

[0036] The display device 2 further includes a bus line 210. The bus line 210 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 201 shown in FIG.

[0037] The contact sensor 214 is not limited to an infrared blocking type, and various detection means may be used, such as a capacitive touch panel that identifies the contact position by detecting a change in capacitance, a resistive film touch panel that identifies the contact position by a change in voltage between two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by an object touching the display unit. Also, the electronic pen controller 216 may determine whether or not the part of the electronic pen 4 that the user grips or other parts of the electronic pen have been touched, in addition to the pen tip and pen butt.

[0038] <Layer configuration> Fig. 4 is a configuration diagram of each image layer displayed by the display device 2. As shown in Fig. 4, the display device 2 displays one image by superimposing a UI image (A), a stroke image (B), an output image (C), and a background image (D).

[0039] The UI image (A) is a UI (user interface) image, and displays menu buttons and the like that are operated by the user.

[0040] The stroke image (B) is an image in which one or more stroke data (handwritten data) handwritten by the user is displayed.

[0041] The output image (C) is an image (still image or video) input by an external device (such as a laptop PC 6). External devices include servers and USB memory 5, and also include web pages and video on storage media.

[0042] The background image (D) is a pre-stored image that serves as the background when handwriting, such as a plain image, a grid display, etc. Note that a pre-stored image, such as a saved still image, can also be used as the background image.

[0043] The display device 2 can also selectively display any of four layers. For example, it can display any one or more of a UI image (A), a stroke image (B), an output image (C), or a background image (D).

[0044] <Example of external video display> Figure 5 is a diagram illustrating an example of the display of an external video. As shown in Figure 5(a), the display device 2 can display the external video 110 displayed in the output image (C) in a small size on a part of the display 3, or as shown in Figure 5(b), it can display it across the entire surface of the display 3. It is also preferable that the user be able to arbitrarily change the position of the external video 110 displayed in a small size on a part of the display 3 as shown in Figure 5(a).

[0045] The user can also handwrite strokes anywhere on the display 3, including the external image 110. In both Figures 5(a) and 5(b), there is handwritten data 101 in a circle at a position that overlaps the external image 110. In Figure 5(a), there is handwritten data 120 of "ABC" in an area where there is no external image 110.

[0046] Conventionally, when a user performs an operation to save an external video, the entire display 3 is saved as a single-page still image. In this embodiment, when the user saves a video, and the external video 110 is displayed on part of the display 3 as shown in FIG. 5(a), the user can select whether to save the entire display 3 or only the external video 110. This allows the size of video files, which tend to be large, to be reduced.

[0047] The user can set in advance using the UI image (A) whether to save the entire display 3 or only the external video 110. Alternatively, the user may set this by handwriting predetermined symbols using handwritten data or by voice input.

[0048] <About the function> Fig. 6 is an example of a functional block diagram illustrating functions of the display device 2. As shown in Fig. 6, the display device 2 has an external video input unit 81, a communication video input unit 82, an external video receiving unit 83, a superimposed image generating unit 84, a handwritten data input unit 85, a voice input unit 86, a handwritten data receiving unit 87, a recorded data generating unit 88, a display control unit 89, an instruction generating unit 90, and a storage unit 91. These functions of the display device 2 are functions or means realized, for example, by the CPU 201 shown in Fig. 3 executing a program loaded from the SSD 204 to the RAM 203.

[0049] The external video input unit 81 is realized by the execution of a program by the CPU 201 and the capture device 211, etc., and receives input of external video continuously transmitted by the notebook PC 6. The input external video is mainly an image of the screen displayed on the display of the notebook PC 6 itself, but content stored in the notebook PC 6 may also be transmitted to the display device 2.

[0050] The communication video input unit 82 is realized by the execution of a program by the CPU 201 and the network I / F 205, etc., and receives input of external video transmitted via communication such as wireless LAN, wired LAN, or Miracast (registered trademark).

[0051] The external video input unit 81 and the communication video input unit 82 input external video to the external video receiving unit 83. The external video receiving unit 83 inputs at least one of the external video input by the external video input unit 81 and the external video input by the communication video input unit 82 to the superimposed image generating unit 84.

[0052] The handwritten data input unit 85 is realized by the execution of a program by the CPU 201 and the contact sensor 214, etc., and receives the input of a sequence of coordinate points input by the input means. The handwritten data input unit 85 inputs the sequence of coordinate points to the handwritten data receiving unit 87.

[0053] The handwriting data receiving unit 87 creates stroke data based on the sequence of coordinate points input from the handwriting data input unit 85. The handwriting data receiving unit 87 interpolates the sequence of coordinate points to create stroke data of a set thickness and color. The handwriting data receiving unit 87 inputs the stroke data to the superimposed image generating unit 84 and the instruction generating unit 90.

[0054] The audio input unit 86 is realized by the execution of a program by the CPU 201 and the microphone 240, etc., and accepts input of audio data. The audio input unit 86 converts the audio input from the microphone 240 into audio data by PCM (pulse code modulation) encoding. The audio data may be a WAVE file or an MP3 file. The audio input unit 86 inputs the audio data to the superimposed image generation unit 84 and the instruction generation unit 90.

[0055] The superimposed image generating unit 84 superimposes (combines) the handwritten data input from the handwriting data receiving unit 87 on the external video input from the external video receiving unit 83 to create output data for the display 3, and inputs the output data to the recording data generating unit 88 and the display control unit 89. The superimposed image generating unit 84 also synchronizes the image in which the handwritten data is superimposed on the external video with the audio data. This allows the recording data generating unit 88 to save an explanation of the handwritten data together with the audio.

[0056] The recording data generating unit 88 converts the recording data into a video file and stores it in the storage unit 91. The display control unit 89 displays the output data on the display 3.

[0057] The instruction generation unit 90 generates an instruction to start or stop recording based on the handwritten data or voice data and the conversion information shown in Fig. 7. The instruction generation unit 90 inputs the instruction to start or stop recording to the recording data generation unit 88. The recording data generation unit 88 receives the instruction as a command and starts or stops recording.

[0058] 7 is a diagram showing the conversion information stored in the conversion information storage unit 93 used by the instruction generation unit 90. The conversion information is information that converts handwritten data or text data into an instruction to start or stop recording. There are two types of conversion information: one that directly converts handwritten data into an instruction, and one that converts text data converted from handwritten data or voice data into an instruction.

[0059] Figure 7(a) is an example of conversion information that directly converts handwritten data into instructions. In Figure 7(a), an instruction to start recording is associated with handwritten data in a circle, and an instruction to stop recording is associated with handwritten data in a square. In Figure 7(a), the shape of the handwritten data is shown for ease of explanation, but in reality the handwritten data is a sequence of coordinate points.

[0060] Such conversion information is generated, for example, by machine learning, which will be described later. Furthermore, the conversion information in FIG. 7(a) is an example, and any shape of handwritten data can be associated with an instruction to start or stop recording. For example, handwritten data that is easy for a user to write and that prevents the user from accidentally starting or stopping recording is used. In addition to a circle or a square, shapes such as a triangle or an arrow, numbers, or one or more alphabetic characters may also be used.

[0061] 7(b) is an example of conversion information for converting text data converted from handwritten data or voice data into instructions. In FIG. 7(b), the text data "ROKUGA" converted by character recognition is associated with an instruction to start recording, and the text data "Yame" is associated with an instruction to stop recording. Therefore, the user can start recording by handwriting "ROKUGA" and stop recording by handwriting "Yame."

[0062] In addition, in Figure 7(b), the instruction to start recording is associated with the text data "Rokugakaishi" converted by speech recognition, and the instruction to stop recording is associated with the text data "Rokugashuryo" (end of recording). Therefore, the user can start recording by saying "Rokugakaishi" and stop recording by saying "Rokugashuryo" (end of recording).

[0063] In FIG. 7, the text data converted by character recognition and the text data converted by voice recognition are separated, but the two do not need to be distinguished.

[0064] <Converting stroke data or voice data into instructions> Methods by which the display device 2 detects whether stroke data or voice data expresses a specific intent include using machine learning and pattern matching. Machine learning is a technology that allows a computer to acquire human-like learning capabilities. The computer autonomously generates algorithms necessary for judgments such as data identification from previously acquired training data and applies these algorithms to new data to make predictions. The learning method for machine learning may be supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, or deep learning, or may be a combination of these learning methods. Any learning method for machine learning is acceptable. Pattern matching is a technique in which a pattern to be identified is prepared in advance, and when searching data, the computer determines whether and where the specific pattern appears.

[0065] Figure 8 shows a neural network 11 in which L layers are fully connected from the input layer 31 to the output layer 33. A neural network with deep layers is called a DNN (Deep Neural Network). The layer between the input layer 31 and the output layer 33 is called an intermediate layer 32. The number of layers and the number of nodes 35 are merely examples.

[0066] Stroke data (sequences of coordinate points) are input to the input layer 31. Since the stroke data is input continuously, a certain number of sequences of coordinate points are input repeatedly. On the other hand, the number of nodes in the output layer 33 is the number of categories to be classified. For example, if classification is into characters, the number of nodes in English is 26, the number of the alphabet. Recognition is also possible on a word-by-word basis rather than on a character-by-character basis, in which case the number of nodes is the number of words. In the case of Japanese, the number of nodes in the output layer 33 is the number of hiragana, katakana, and kanji characters.

[0067] In the neural network 11, all nodes 35 in the l-1 layer are connected to one node 35 in the l-1 layer (l (l): 2 to L), and the product of the output z of the node 35 in the l-1 layer and the connection weight w is input to the node in the l layer.

[0068] An activation function is placed in each node of the lth layer, and the lth layer node 35 converts the input into a value between 0 and 1 using the activation function and outputs it to the l+1th layer node 35. This process is repeated from the input layer 31 to the output layer 33. Known activation functions include ReLU, tanh, and sigmoid. Note that the nodes of the input layer 31 are not activated; they only need to transmit feature vectors to the second layer.

[0069] In this embodiment, the output of the nodes in the output layer 33 is classified into characters, words, recording start, recording end, etc., and so a classification problem is dealt with in machine learning. Therefore, the activation function of the nodes in the output layer 33 may be a softmax function. The error between the output of each node in the output layer and the teacher signal (a one-hot vector where nodes with the correct classification are 1 and the rest are zero) is calculated using cross-entropy, and the connection weights w and bias b are updated (learned) so that the output of the output layer 33 approaches the teacher signal.

[0070] To classify continuous data such as stroke data and voice data, it is effective to apply a recurrent neural network (RNN), which is an extension of a neural network that can handle time-series data.

[0071] Furthermore, machine learning techniques include perceptron, support vector machine, logistic regression, naive Bayes, decision tree, random forest, etc., and are not limited to the techniques described in this embodiment.

[0072] <Operation procedure> FIG. 9 is an example of a sequence diagram showing the overall operation of the communication system 1.

[0073] S1: The notebook PC 6 connected to the display device 2 transmits an external video to the display device 2.

[0074] S2: The external video input unit 81 of the display device 2 receives input of an external video and inputs it to the external video receiving unit 83. The external video receiving unit 83 inputs the external video to the superimposed image generating unit 84. The external video is displayed on the display 3 by the display control unit 89.

[0075] S3: When the user writes by hand, the handwritten data input unit 85 accepts the input of handwritten data. The handwritten data input unit 85 inputs the handwritten data to the handwritten data receiving unit 87. The handwritten data receiving unit 87 inputs the handwritten data to the superimposed image generation unit 84 and the instruction generation unit 90.

[0076] S4: The superimposed image generation unit 84 superimposes the handwritten data on the external image and inputs the superimposed image to the display control unit 89. The display control unit 89 displays an image in which the handwritten data is superimposed on the external image on the display 3. The frames of the external image are updated one after another, and the superimposed image generation unit 84 repeats the process of superimposing all of the handwritten data up to that point on new frames of the external image.

[0077] FIG. 10 is an example of a flowchart illustrating a process in which the display device 2 generates an instruction to start recording, starts recording, and generates an instruction to end recording, and ends recording.

[0078] The instruction generation unit 90 monitors the handwritten data and the voice data, and determines whether handwritten data or text data (converted from the handwritten data and the voice data) for starting recording registered in the conversion information has been input (S11). Note that the instruction generation unit 90 may determine whether the handwritten data is superimposed on the external video, and only if it is superimposed, determine whether handwritten data for starting recording registered in the conversion information has been input. This prevents the display device 2 from starting or ending recording due to handwritten data other than the external video.

[0079] If the determination in step S11 is Yes, the instruction generating unit 90 generates an instruction to start recording that is associated with the handwritten data or text data, and inputs it to the recording data generating unit 88 (S12).

[0080] Next, the instruction generation unit 90 monitors the handwritten data and voice data, and determines whether handwritten data or text data (converted from the handwritten data and voice data) indicating the end of recording, which is registered in the conversion information, has been input (S13).

[0081] If the determination in step S13 is Yes, the instruction generating unit 90 generates an instruction to end recording associated with the handwritten data or text data, and inputs it to the recording data generating unit 88 (S14).

[0082] On the other hand, the recording data generating unit 88 determines whether an instruction to start recording or an instruction to end recording has been input (S21).

[0083] When a command to start recording is input, the recording data generation unit 88 starts recording a superimposed image in which the handwritten data is superimposed on the external video (S22). If the external video includes audio, the audio is also recorded. Furthermore, the audio collected by the microphone 240 is also recorded along with the superimposed image. In this way, the display device 2 can save the user's commentary on the external video together with the external video and the handwritten data.

[0084] The recording data generating unit 88 continues recording the superimposed image in which the handwritten data is superimposed on the external video until an instruction to end recording is input (S23).

[0085] The handwritten data or text data (converted from the handwritten data and audio data) for the start and end of recording may be the same. In this case, the instruction generation unit 90 generates an instruction to switch the recording state based on the handwritten data or text data for the start and end of recording. The recording data generation unit 88 starts recording if no recording is being performed, and stops recording if recording is in progress. In this way, the handwritten data (or text data) for the start and end of recording will be the same.

[0086] The recording data generation unit 88 may continuously record the superimposed image in which handwritten data is superimposed on the external video even if no instruction to start recording is input. This makes it possible to record from several seconds before the instruction to start recording. The recording data generation unit 88 continuously records the superimposed image in which handwritten data is superimposed on the external video, and overwrites the old superimposed image after, for example, five seconds have passed. In this way, the superimposed image from five seconds before always remains, so recording can start five seconds before the instruction to start recording.

[0087] Regarding the end of recording, the recording data generation unit 88 may end recording after a certain time has elapsed since an instruction to end recording is given. Alternatively, the recording data generation unit 88 may end recording after a certain time has elapsed since recording started, even if no instruction to end recording is given. The certain time may be set by the user, or may be the average or maximum value of past recording times. This can prevent recording from being left unattended.

[0088] <Modification> In the above embodiment, the display device 2 generates the instructions to start and stop recording by itself, but the instructions to start and stop recording may be issued by an information processing device on the network. Such an information processing device is called a server device.

[0089] FIG. 11 shows an example configuration of a display system 100. In the display system 100 of FIG. 11, a display device 2 and an information processing device 300 are communicably connected via a communication network 9. The information processing device 300 is one or more computers and may be compatible with cloud computing. The information processing device 300 may be located on-premise or on the internet. With the configuration as shown in FIG. 11, the display device 2 can transmit handwritten data and voice data to the information processing device 300 and receive instructions to start and stop recording from the information processing device 300.

[0090] Fig. 12 shows a functional block diagram of the display device 2 and the information processing device 300 in the display system 100 of Fig. 11. The explanation of Fig. 12 will mainly focus on the differences from Fig. 6. The display device 2 of Fig. 12 has a communication unit 74. The communication unit 74 transmits handwritten data and voice data to the information processing device 300 in real time, and receives instructions to start or stop recording from the information processing device 300.

[0091] The information processing device 300 has a communication unit 75 and an instruction generation unit 90. The communication unit 75 receives handwritten data and voice data from the display device 2 and transmits an instruction to start or stop recording to the display device 2. The function of the instruction generation unit 90 may be the same as that shown in FIG.

[0092] According to this configuration, instructions to start and stop recording can be given by the information processing device 300 on the network. Conversion information can be collected in the information processing device 300, which facilitates maintenance. 13 is an example of a sequence diagram showing the overall operation of a communication system having information processing device 300. The explanation of FIG. 13 will mainly focus on the differences from FIG. 9. The processing of steps S1 to S4 may be the same as in FIG. S5: The communication unit 74 of the display device 2 transmits the handwritten data to the information processing device 300. S6: The communication unit 75 of the information processing device 300 receives the handwritten data, and the instruction generation unit 90 generates an instruction to start or stop recording. This process may be the same as the flowchart on the left side of FIG. S7: The communication unit 75 of the information processing device 300 transmits to the display device 2 an instruction to start or end recording. S8: The communication unit 74 of the display device 2 receives the instruction to start or stop recording, and starts or stops recording according to the instruction. This process may be the same as the flowchart on the right side of FIG.

[0093] <Major Effects> As described above, the display device 2 of this embodiment can handle handwritten data as a command input and start and stop recording, so that the same handwritten data can be used to start and stop recording and to save the handwritten data for external video. Also, it is possible to start and stop recording using text data that has been recognized as handwritten data or voice data as a command input.

[0094] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0095] Although details are omitted in this embodiment, the video is saved in a video format such as MOV, AVI, WMV, or MPEG. It is preferable that a recording start time is associated with the video. The time of each frame constituting the video can be calculated using fps (frames per second). Furthermore, text data is saved as character codes, and handwritten data is saved as coordinate point data, along with time information, in the display device 2. It is preferable that at least the start time of writing the text data or handwritten data is saved. Therefore, when playing back the saved video, the display device 2 can display the text data or handwritten data at the same timing as when it was handwritten. The display device 2 may save the handwritten data or text data superimposed on the video as a video.

[0096] Furthermore, the display device 2 can store handwritten data, text data, and videos in various storage media or in a storage device on a network, and later download and reuse the data from the display device 2. The display device 2 to be reused can be any display device 2, even a general information processing device. Therefore, the user can reproduce handwritten content on a different display device 2 and continue a meeting, etc.

[0097] For example, in the present embodiment, an electronic whiteboard has been described as an example of the display device 2, but the electronic whiteboard may also be called an electronic whiteboard, an electronic information board, or the like. Furthermore, the present embodiment can be suitably applied to any information processing device having a touch panel. Examples of information processing devices equipped with a touch panel include output devices such as PJs (Projectors), digital signage, HUDs (Head Up Display) devices, industrial machinery, imaging devices, sound collection devices, medical equipment, network appliances, notebook PCs (Personal Computers), mobile phones, smartphones, tablet devices, game consoles, PDAs (Personal Digital Assistants), digital cameras, wearable PCs, and desktop PCs.

[0098] Furthermore, if the display device 2 is a projector, the display device 2 may detect the coordinates of the pen tip using ultrasound. In this case, the pen emits light and also emits ultrasound, and the display device 2 calculates the distance based on the time it takes for the ultrasound to reach the screen. The display device 2 identifies the direction by capturing an image of the emitted light with a camera. The display device 2 can identify the position of the pen based on the direction and distance. The projector draws (projects) the trajectory of the pen as stroke data. In this case, an external image is input to the projector, and the stroke data is superimposed on the external image.

[0099] In addition, the configuration examples in Figure 6 and the like are divided according to main functions to make it easier to understand the processing by the display device 2. The method of dividing the processing units and their names do not limit the present invention. The processing by the display device 2 can be divided into even more processing units depending on the processing content. Also, it can be divided so that one processing unit includes even more processes.

[0100] Furthermore, part of the processing performed by the display device 2 may be performed by a server connected to the display device 2 via a network.

[0101] Furthermore, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions. [Explanation of symbols]

[0102] 1. Communication Systems 2 Display device 3. Display 6. Laptop [Prior art documents] [Patent documents]

[0103] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-191169

Claims

1. A display device capable of displaying an external image input from an external device on a display, a handwritten data input unit that accepts input of handwritten data; a display control unit that displays the external image on a part of the display and the handwritten data on the display; a recording data generating unit that records the external video as a video based on the handwritten data; an instruction generation unit that generates instructions to start and stop recording for the recording data generation unit based on the handwritten data; a superimposed image generating unit that generates a superimposed image by superimposing the handwritten data on the external video; the instruction generation unit generates an instruction to the recording data generation unit to start or stop recording of the external video only when the handwritten data is superimposed on the external video. A display device characterized by:

2. The display device described in Claim 1, characterized in that the recording data generation unit starts recording the external video in response to the instruction to start recording, and ends recording the external video in response to the instruction to end recording.

3. The display device described in Claim 2, characterized in that the instruction generation unit generates an instruction to the recording data generation unit to start or stop recording the external video based on the handwritten data superimposed on the external video.

4. the superimposed image generation unit repeatedly superimposes the handwritten data on frames of the external video that change over time; 4. The display device according to claim 3.

5. 4. The display device according to claim 3, wherein the instruction generating unit converts the handwritten data into an instruction to start recording or stop recording based on conversion information in which the sequence of coordinate points of the handwritten data is associated with an instruction to start recording or stop recording.

6. Based on the conversion information in which the text data and the instruction to start or stop recording are associated, 4. The display device according to claim 3, wherein the instruction generating unit converts text data generated from the handwritten data by character recognition into an instruction to start or stop recording.

7. a voice input unit that accepts input of voice data from a user; 7. The display device according to claim 6, wherein the instruction generating unit converts text data generated from the voice data by voice recognition into an instruction to start or stop recording.

8. The display device according to claim 7 , wherein the recording data generating unit records the superimposed image and stores the audio data.

9. 9. The display device according to claim 3, wherein the recording data generating section records the superimposed images going back several seconds before the instruction to start recording is generated.

10. A display method for displaying an external image input from an external device on a display, comprising: a step of a handwritten data input unit accepting input of handwritten data; a display control unit causing the external image to be displayed on a part of the display and displaying the handwritten data on the display; a recording data generating unit recording the external video as a video based on the handwritten data; an instruction generation unit generating instructions to start and stop recording for the recording data generation unit based on the handwritten data; a superimposed image generating unit generating a superimposed image in which the handwritten data is superimposed on the external video; generating an instruction to the recording data generating unit to start or stop recording of the external video only when the handwritten data is superimposed on the external video; A display method comprising:

11. A display system capable of displaying an external image input from an external device on a display, a handwritten data input unit that accepts input of handwritten data; a display control unit that displays the external image on a part of the display and the handwritten data on the display; a recording data generating unit that records the external video as a video based on the handwritten data; an instruction generation unit that generates instructions to start and stop recording for the recording data generation unit based on the handwritten data; a superimposed image generating unit that generates a superimposed image by superimposing the handwritten data on the external video; the instruction generation unit generates an instruction to the recording data generation unit to start or stop recording of the external video only when the handwritten data is superimposed on the external video. A display system characterized by:

Images