Writing device, writing system, writing processing method, and program

Abstract

To solve the problem in conventional art that when performing writing operation using a writing device, information on stroke images that are generated in conjunction with the writing operation has to be stored in a device other than the writing device.SOLUTION: When an information processing device 5 that performs communication with a writing device 3 cannot communicate with the writing device 3, the writing device 3 detects a start position on a writing surface of the writing device 3 associated with a writing operation (Step S14-3-1), calculates coordinate information on the writing surface of the writing device 3 associated with the writing operation on the basis of the detected start position (Step S14-4), and stores the calculated coordinate information in a storage unit 3000 as information on a stroke image (Step S14-5-4).SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] The present invention relates to a writing device, a writing system, a writing processing method, and a program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there has been known a technique for recording and managing handwritten data by inputting data using a pen-type input device or an inputting operation into a pen tablet or the like.

[0003] For example, a technology is known for a pen tip coordinate input device that corrects the movement direction and movement amount of the pen tip on the xy plane to the movement direction and movement amount on the writing surface in order to improve the accuracy of detection of the amount of movement (see Patent Document 1).

[0004] There is also known technology relating to a pen tablet that acquires the position coordinates of a pen and records the position coordinate information in a storage device according to the state of communication with an external computer (see Patent Document 2). Summary of the Invention [Problem to be solved by the invention]

[0005] However, these conventional techniques have a problem in that, when performing a writing action using a writing device, information related to a stroke image generated in association with the writing action needs to be stored in a device other than the writing device. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the invention of claim 1 provides a writing device capable of generating a trajectory of its own device accompanying a writing action as a stroke image, comprising: a storage means for storing information relating to the stroke image; a detection means for detecting a start position of the writing device on the writing surface accompanying the writing action; a calculation means for calculating coordinate information of the writing device on the writing surface accompanying the writing action based on the detected start position; and a storage control means for storing the calculated coordinate information in the storage means as information relating to the stroke image when an information processing device that communicates with the writing device is not in a state capable of communicating with the writing device. Effect of the Invention

[0007] As described above, according to the present invention, when performing a writing action using a writing device, it is possible to store information related to the stroke image generated in conjunction with the writing action in the writing device itself. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of the overall configuration of a writing system according to an embodiment; [Diagram 2] FIG. 2 is a diagram illustrating an example of a configuration of a writing device according to an embodiment. [Diagram 3] 5A and 5B are diagrams illustrating an example of the configuration of a pressure sensor provided in the writing device according to the embodiment. [Figure 4] 4A and 4B are diagrams illustrating an example of the configuration of an optical sensor provided in the writing device according to the embodiment. [Diagram 5] 10A and 10B are conceptual diagrams illustrating an example of a method for determining coordinate information in a stroke image according to the embodiment. [Figure 6] 1A and 1B are image diagrams showing an example of a writing device according to an embodiment when used as a cursor, where FIG. 1A shows the movement of the writing device in association with a writing action, and FIG. 1B shows the movement of a cursor on an information processing device in response to the writing action of the writing device. [Figure 7] FIG. 2 is a diagram illustrating an example of a hardware configuration of a writing device according to an embodiment. [Figure 8] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device according to an embodiment. [Figure 9] FIG. 2 is a diagram illustrating an example of a functional configuration of a writing system according to an embodiment. [Figure 10] FIG. 4 is a conceptual diagram illustrating an example of a coordinate information management table according to the embodiment. [Figure 11] FIG. 4 is a conceptual diagram illustrating an example of a coordinate information management table according to the embodiment. [Figure 12] FIG. 13 is a conceptual diagram illustrating an example of a division information management table according to the embodiment. [Figure 13] 5 is a flowchart showing an example of an overall process in the writing device according to the first embodiment. [Figure 14] 11 is a flowchart showing an example of processing including a writing action determination process and a coordinate information storage process, among the overall processing in the writing device according to the first embodiment. [Figure 15] 11 is a flowchart showing an example of details of a writing action in progress determination process, part of the overall processing in the writing device according to the first embodiment. [Figure 16] 13 is a flowchart showing an example of details of a coordinate information storage process among the overall processing in the writing device according to the first embodiment. [Figure 17] 4 is a conceptual diagram showing an example of a state of a coordinate information management table according to the first embodiment. FIG. [Figure 18] 10 is a conceptual diagram showing an example of another state of the coordinate information management table according to the first embodiment. FIG. [Figure 19] 10 is a conceptual diagram showing an example of another state of the coordinate information management table according to the first embodiment. FIG. [Figure 20] 4 is a conceptual diagram showing an example of a state of a coordinate information management table in a coordinate information storage process according to the first embodiment. FIG. [Figure 21] 10 is a conceptual diagram showing an example of another state in the coordinate information storage process of the coordinate information management table according to the first embodiment. FIG. [Figure 22] 10 is a conceptual diagram showing an example of another state in the coordinate information storage process of the coordinate information management table according to the first embodiment. FIG. [Figure 23] 10 is a conceptual diagram showing an example of another state in the coordinate information storage process of the coordinate information management table according to the first embodiment. FIG. [Figure 24] 10 is a flowchart showing an example of details of a data amount reduction process among the overall processing in the writing device according to the first embodiment. [Diagram 25] 10A and 10B are conceptual diagrams illustrating an example of a state during a data amount reduction process of a coordinate information management table according to the first embodiment. [Figure 26] FIG. 11 is a sequence diagram showing an example of processing between a writing device and an information processing device according to the second embodiment. [Figure 27] FIG. 11 is a diagram showing an example of a handwriting data display screen in an information processing device according to a second embodiment. [Figure 28A] FIG. 11 is a diagram showing an example of a handwriting data replay display screen in an information processing device according to a second embodiment. [Figure 28B] FIG. 11 is a diagram showing another example of the writing data replay display screen in the information processing device according to the second embodiment. [Figure 28C] FIG. 11 is a diagram showing another example of the writing data replay display screen in the information processing device according to the second embodiment. [Figure 29] 11A and 11B are diagrams showing an example of a method for displaying handwriting data according to a second embodiment, in which (a) is a diagram showing an example of a case in which stroke images are drawn in an overlapping manner by a writing action using a writing device, (b) is an example of a conceptual diagram of decomposing the overlapping stroke images into individual stroke images, and (c) is an example of a display screen in which each of the decomposed stroke images is displayed on an information processing device so as not to overlap one another. [Diagram 30] FIG. 11 is a sequence diagram showing an example of processing between a writing device and an information processing device according to a modified example of the second embodiment. [Diagram 31]FIG. 13 is a sequence diagram showing an example of processing between a writing device and an information processing device according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and the description of overlapping parts will be omitted.

[0010] [First embodiment] [Overall configuration of the writing system] Fig. 1 is a diagram showing an example of the overall configuration of a writing system. As shown in Fig. 1, the writing system 1 has a writing device 3 capable of generating a trajectory of the writing device accompanying a writing action as a stroke image, and an information processing device 5 capable of communicating with the writing device. In the writing system 1, the writing device 3 and the information processing device 5 are connected to each other via a communication network 100.

[0011] The communication network 100 is a wired or wireless communication network for an unspecified number of communications, and is constructed by the Internet, an intranet, a LAN (Local Area Network), etc. The communication network 100 may include a wireless communication network such as 4G (4th Generation), 5G (5th Generation), WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution), etc. The writing device 3 and the information processing device 5 may be connected to each other by a dedicated local area network, etc.

[0012] The writing device 3 and the information processing device 5 may be connected by short-distance wireless communication such as Bluetooth (registered trademark, omitted below) instead of the communication network 100. In this case, the short-distance wireless communication may further utilize short-distance wireless communication by NFC (Near Field Communication). Furthermore, the writing device 3 and the information processing device 5 may be connected by a wireless communication network such as Wi-Fi (registered trademark, omitted below).

[0013] <Writing device> The writing device 3 is a device that is used by a user of the writing system 1 and is capable of generating a trajectory of the device in association with a writing action as a stroke image. The writing device 3 may be realized by an information processing device (computer system) for performing communication that is equipped with a general OS or the like, and is one of the components that configure the writing system 1. The writing device 3 also has a communication application for communicating with the information processing device 5 via the communication network 100 installed and stored in a storage means.

[0014] The writing device 3 may be shaped like a pen, a ring, or a fingerstall, and is a device that is held in a hand or with fingers, or worn on a hand or with fingers, upon which writing operations are performed.

[0015] <Information processing device> The information processing device 5 is used by a user who uses the writing system 1, and is a device capable of communicating with the writing device. The information processing device 5 is realized by an information processing terminal (computer system) for performing communication equipped with a general OS or the like, and is one of the components that configure the writing system 1. The information processing device 5 also has a communication application for communicating with the writing device 3, a drawing application for drawing a stroke image in which a writing operation has been performed with the writing device 3, and the like installed and stored in a storage means.

[0016] The information processing device 5 may be a communication terminal having a communication function, such as a commonly used PC (Personal Computer), a portable notebook PC, a mobile phone, a smartphone, a tablet terminal, or a wearable terminal (sunglasses type, wristwatch type, etc.). The information processing device 5 may further be a communication device or a communication terminal capable of operating browser software and various application software.

[0017] Furthermore, the information processing device 5 may notify (transmit) data (information) by push notification (transmission) to other communication terminals or the like connected via the communication network 100. In this case, the information processing device 5 performs the push notification via, for example, FCM (Firebase Cloud Messaging), which is an example of a push notification server.

[0018] ●About terminology● In this embodiment, in the case of a pen-shaped writing device (for example, a ballpoint pen) which is an example of the writing device 3, the writing action refers to both the writing action in which the pen tip comes into contact with the writing surface to draw a stroke image (line), and the movement action between the first and second strokes. In other words, the writing action refers to a state in which the writing action and the movement action are continuous and a series of characters or figures are drawn. Note that the writing device 3 in this embodiment does not include a so-called pen tablet type input device.

[0019] In this embodiment, a stroke image is an image that represents a path drawn by a writing action using the writing device 3.

[0020] In this embodiment, the writing-related information accompanying the writing action is information that represents any one of date and time information or time information when the writing action was performed, location information that represents the location where the writing device 3 is located, and person information related to a person who is in the same space as the user who uses the writing device 3. The above-mentioned writing-related information is also called metadata accompanying the writing action.

[0021] In this embodiment, a user refers to a person who uses the handwriting device 3 or an administrator who manages the users. In this embodiment, unless otherwise specified, the term "user" is used.

[0022] <Writing device configuration> FIG. 2 is a diagram showing an example of the configuration of a writing device according to an embodiment. As shown in FIG. 2, the writing device 3 includes an optical sensor 317 as a detection means for detecting two-dimensional coordinate information (coordinate information), and the optical sensor 317 calculates the amount of movement of the writing device 3 in the x-axis direction and the y-axis direction. The writing device 3 also includes a pressure sensor 316 for detecting the presence or absence of a writing instruction. A tilt detector, a correction circuit, and a friction coefficient detection circuit are connected to the pressure sensor 316. The writing device 3 also includes a calibration switch 333. The writing device 3 also includes an acceleration sensor 315 in a portion close to the device tip 332, which detects yaw, roll, pitch, etc., in order to analyze and identify a gesture made by a user using the writing device 3, for example. An integrating circuit is connected to the acceleration sensor 315.

[0023] <<Accelerometer configuration>> Acceleration sensor 315 is provided to detect acceleration in the x-axis and y-axis directions when device axis 331 is the z-axis, for example, and detects the movement direction and acceleration of device tip 332 in two directions, the x-axis and y-axis directions. The integration circuit samples the acceleration detected by acceleration sensor 315 at short time intervals, for example, and integrates it twice over time to detect the amount of movement of device tip 332 in two directions, the x-axis and y-axis directions. Note that detection of the amount of movement by acceleration sensor 315 may be performed in cases where the amount of movement cannot be obtained by optical sensor 317 depending on the form of the writing device, and the like, by detecting the amount of movement with acceleration sensor 315.

[0024] <<Pressure sensor configuration>> Fig. 3 is a diagram showing an example of the configuration of a pressure sensor provided in a writing device according to an embodiment. As shown in Fig. 3, a pressure sensor 316 is provided in the device tip 332 shown in Fig. 2, and detects acting forces in the z-axis direction at the device tip 332, and in the x-axis and y-axis directions perpendicular to the z-axis.

[0025] The tilt detector connected to the pressure sensor 316 detects the tilt direction and tilt angle of the writing device 3 based on the applied force detected by the pressure sensor 316. For example, the applied force detected by the pressure sensor 316 is represented by a vector F, and the vector F is detected with a solid angle of α with respect to the device axis 331 as shown in FIG. 3. If the writing device 3 is pressed against the writing surface 340 with a force of Fn, the pressure sensor 316 receives a reaction force of Fn from the writing surface 340. In other words, this force is perpendicular to the writing surface, and the pressure sensor 316 measures the direction of this force based on the pen coordinates, thereby making it possible to know the current tilt angle θ between the device axis 331 (z direction) and the writing surface. In addition, the current writing pressure can be obtained from the magnitude of the force. If the frictional force between the writing surface and the device tip 332 is not taken into consideration, F=Fn and α=90-θ. The tilt detector detects the tilt direction and tilt angle of the writing device 3 based on the applied force detected by the pressure sensor 316 as described above.

[0026] Since the device tip 332 is actually subjected to a friction force of μFn (μ is the dynamic friction coefficient), it is advisable to calculate the dynamic friction coefficient μ in advance, and have the tilt detector detect the tilt direction and tilt angle of the writing device 3 based on the previously measured friction coefficient and the acting force detected by the pressure sensor 316. The dynamic friction coefficient μ is given by the ratio of Fn to the friction force, and is a constant determined by the material of the writing device 3 and the material of the writing surface.

[0027] To obtain the above-mentioned dynamic friction coefficient in advance, for example, a user presses the calibration switch 333 to set a mode for measuring the friction coefficient, and moves the writing device 3 vertically to the writing surface. The pressure sensor 316 detects a reaction force Fn in the z-axis direction, and receives a force μFn in a direction perpendicular to the z-axis direction. The friction coefficient detection circuit obtains and stores the dynamic friction coefficient μ from the force measured by the pressure sensor 316.

[0028] In the case shown in FIG. 3, the friction force μFn is μFn=Fntanη, and the dynamic friction coefficient μ is expressed as μ=tanη. In addition, the reaction force |Fn| from the contact surface is |Fn|=|F|cosη from the stress F detected by the pressure sensor 316, and the tilt angle θ is θ=90-α+η, so the writing pressure and pen tilt can be easily obtained. For simplification, it is assumed in the figure that the tilt direction (the direction when the device axis 331 is projected onto the writing surface) and the writing direction (the direction in which the friction force acts) are on the same axis. It is rare that the friction force direction and the pen tilt direction are on the same axis, but since the acceleration sensor 315 detects the moving direction of the device tip 332, this information is used to detect and correct the direction of action of the friction force, making it possible to detect the tilt angle three-dimensionally. In this way, the tilt detector can accurately detect the tilt direction and tilt angle of the writing device 3 based on the friction coefficient measured in advance by the friction coefficient measurement circuit and the acting force detected by the pressure sensor 316.

[0029] <<Configuration of optical sensor>> Fig. 4 is a diagram showing an example of the configuration of an optical sensor provided in a writing device according to an embodiment. As shown in Fig. 4, the optical sensor 317 includes a light emitting element 317a such as a semiconductor laser and a light receiving element 317b such as a photodiode.

[0030] The light sensor 317 irradiates a light spot from the light emitting element 317a toward the writing surface, receives the light reflected from the writing surface with the light receiving element 317b (image sensor, etc.), and reads the light reflected from the writing surface as status information (such as a pattern) of the irradiated portion on the writing surface. Furthermore, the writing device 3 calculates the amount of movement in the x-axis and y-axis directions by continuously comparing the status information (such as a pattern) read by the light sensor 317 with the status information read by the subsequent writing operation. The writing device 3 can also calculate the relative position and coordinates based on the amount of movement. The detection result of the pressure detected by the pressure sensor 316 can be used to determine the stroke image from the position where the pressure is detected to the position where the pressure is not detected among the relative position and coordinates detected by the light sensor 317.

[0031] In addition, it is preferable that the optical sensor 317 has a variable installation angle, that is, the angle of incidence of the light emitted from the light-emitting element 317a on the writing surface or the angle of incidence of the light reflected on the writing surface to the light-receiving element 317b can be variable. For example, when a writing surface with high absorbance is to be detected, if the angle of the light-emitting element 317a such as a semiconductor laser and the angle of the light-receiving element 317b such as a photodiode with respect to the writing surface in FIG. 4 are arranged so that they are substantially the same, the light emitted from the light-emitting element 317a that is specularly reflected is likely to enter the light-receiving element 317b, resulting in a small signal-to-noise ratio. In contrast, if the installation angle of the optical sensor 317 is variable, the angle of the light-emitting element 317a such as a semiconductor laser and the angle of the light-receiving element 317b such as a photodiode with respect to the writing surface in FIG. 4 can be arranged so that they are different. In this way, the light sensor 317a is arranged so that the regular reflection light (light whose angle of incidence and angle of reflection are the same) of the light emitting element 317a is less likely to enter the light receiving element 317b, and the detection accuracy on the writing surface can be improved. Note that the above-mentioned configuration of the light sensor 317 is an example, and is not limited to this.

[0032] <How to obtain coordinate information in a stroke image> Fig. 5 is a conceptual diagram showing an example of a method for determining coordinate information in a stroke image according to an embodiment. Fig. 5 shows a case in which a stroke image is drawn in the order of (1) -> (2) -> (3) in association with a writing action using the writing device 3.

[0033] First, (1) shows the state in which a stroke image is drawn from the origin (0,0) of the XY coordinates to (x1,y1).

[0034] Next, (2) shows a schematic diagram of the trajectory of the writing device 3 when the user removes the writing device 3 from the writing surface after drawing the stroke image in (1) and moves the writing device 3 to the coordinates (x2, y2).

[0035] Furthermore, (3) shows the state in which the user again touches the writing device 3 to the writing surface from the coordinates (x2, y2) and moves the writing device 3 to (x3, y3), thereby drawing the next stroke image.

[0036] Note that each coordinate value in the XY coordinate system is not an absolute coordinate indicating a physical position on the writing surface associated with the writing action of the writing device 3, but a relative coordinate obtained by contact presence / absence information (pressure information) on the writing surface detected by the pressure sensor 316 of the writing device 3 and movement amount information of the writing device 3 detected by the optical sensor 317 of the writing device 3. More specifically, the writing device 3 generates, as the coordinate information associated with the writing action, a result obtained by repeatedly performing an addition process of the movement amount information of the writing device 3 in a state in which the writing device 3 is in contact with the writing surface and the movement amount information of the writing device 3 in a state in which the writing device 3 is separated from the writing surface. In particular, when the writing device 3 moves in (2), the writing device 3 converts the movement amount of the writing device 3 into a relative coordinate value.

[0037] Through the series of drawing process steps described above, the writing device 3 calculates coordinate information relating to a stroke image as shown in FIG. 5, and stores the calculated coordinate information in the writing device 3 itself as information relating to the stroke image.

[0038] <How to obtain coordinate information in a stroke image> FIG. 6 is an image diagram of a case where the writing device according to the embodiment is used as a cursor. FIG. 6(a) shows the movement of the writing device accompanying a writing action. At this time, the device tip 332 of the writing device 3 maintains a non-contact state in which it is not in contact with the writing surface. In this non-contact state, the user horizontally moves the writing device 3 from a certain position to the right by a predetermined distance. Note that, depending on the performance of the optical sensor 317, if the device tip 332 is within a predetermined distance (e.g., 3 cm) from the writing surface, the writing device 3 can obtain coordinate information of the writing device 3 itself.

[0039] On the other hand, FIG. 6(b) shows the movement of the cursor in the information processing device according to the writing operation of the writing device. The writing device 3 is connected to the information processing device 5 via the communication network 100, and after a predetermined communication is established, the writing device 3 itself is moved by the user while maintaining the non-contact state as shown in FIG. 6(a). That is, the writing device 3 is operated by the user like a mouse. In response to this operation, coordinate information obtained by the writing device 3 is transmitted to the information processing device 5 in real time. Then, the information processing device 5 converts the transmitted coordinate information into coordinate information on a display means such as a display of the information processing device 5, and then displays it on the display as a cursor synchronized with the movement of the writing device 3. This series of processes is executed as a cursor operation (operation by a mouse) on the information processing device 5 when predetermined coordinate information (coordinate data) is transmitted from the writing device 3, regardless of whether a writing instruction for the writing device 3 described later is received.

[0040] [Hardware configuration] Next, the hardware configuration of the device or apparatus constituting the writing system according to the embodiment will be described with reference to Figures 7 and 8. Note that components may be added or deleted from the hardware configuration of the device or apparatus shown in Figures 7 and 8 as necessary.

[0041] <Hardware configuration of the writing device> Fig. 7 is a diagram showing an example of a hardware configuration of a writing device according to an embodiment. As shown in Fig. 7, the writing device 3 is constructed by, for example, a computer, and includes a CPU 301, a ROM 302, a RAM 303, an EEPROM 304, a short-range communication I / F 308, an external device connection I / F 309, a network I / F 311, an acceleration sensor 315, a pressure sensor 316, an optical sensor 317, a switch 318, and a bus line 320.

[0042] Of these, the CPU 301 controls the operation of the entire writing device 3. The ROM 302 stores programs and the like used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301. The EEPROM 304 reads and writes various data such as applications under the control of the CPU 301. The short-range communication I / F 308 is a communication circuit for performing data communication with a communication device or a communication terminal having a wireless communication interface such as NFC (Near Field Communication), Bluetooth (registered trademark, omitted below), or Wi-Fi (registered trademark, omitted below). The external device connection I / F 416 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory (particularly, a microUSB memory) or the like.

[0043] The network I / F 311 is an interface for performing data communication with the information processing device 5 by utilizing the communication network 100. The acceleration sensor 335, the pressure sensor 336, and the light sensor 337 each have the configurations described above with reference to FIGS.

[0044] The switch 318 also serves as the above-mentioned calibration switch 333, and is a switch operated by the user in the writing device 3. The switch 318 is a switch having a button type, a knock type, or the like structure provided on, for example, a ballpoint pen. The switch 318 further has a function of changing various setting information transmitted by the information processing device 5 according to the number of times the user operates the switch 318 itself, the operation pattern (for example, the length of the operation), and the like. This change in setting information also includes a function of resetting information related to stroke images and writing-related information stored in the writing device 3 for a predetermined period. In this embodiment, the switch 318 functions as an example of an operating means.

[0045] The bus line 320 is an address bus, a data bus, or the like for electrically connecting the various components such as the CPU 301 .

[0046] <Hardware configuration of information processing device> Fig. 8 is a diagram showing an example of a hardware configuration of an information processing device according to an embodiment. As shown in Fig. 8, the information processing device 5 is constructed by, for example, a computer, and includes a CPU 501, a ROM 502, a RAM 503, an EEPROM 504, a HD (Hard Disk) 505, a HDD (Hard Disk Drive) controller 506, a display 507, a short-range communication I / F 508, a CMOS sensor 509, and an image sensor I / F 510. In this embodiment, the display 507 functions as an example of a display unit. The information processing device 5 further includes a network I / F 511, a keyboard 512, a pointing device 513, a media I / F 515, an external device connection I / F 516, a sound input / output I / F 517, a microphone 518, a speaker 519, and a bus line 520.

[0047] Among these, the CPU 501 controls the operation of the entire information processing device 5. The ROM 502 stores programs and the like used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The EEPROM 504 reads or writes various data such as applications under the control of the CPU 501. The HD 505 stores various data such as programs. The HDD controller 506 controls the reading or writing of various data from the HD 505 under the control of the CPU 501. Here, the information processing device 5 may have a hardware configuration equipped with an SSD (Solid State Drive) instead of the HD 505 and the HDD controller 206. The display 507 displays various information such as a cursor, a menu, a window, a character, or an image. In this embodiment, the display 507 functions as an example of a display means. The short-range communication I / F 508 is a communication circuit for performing data communication with a communication device or a communication terminal having a wireless communication interface such as NFC (Near Field Communication), Bluetooth (registered trademark; omitted below), or Wi-Fi (registered trademark; omitted below). The CMOS sensor 509 is a type of built-in imaging means that captures an image of a subject under the control of the CPU 501 to obtain image data or video data. The imaging means may be an imaging means constituted by a CCD (Charge Coupled Device) sensor or the like instead of a CMOS sensor. The imaging element I / F 510 is a circuit that controls the driving of the CMOS sensor 509.

[0048] The network I / F 511 is an interface for data communication using the communication network 100. The keyboard 512 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. Note that, instead of or in addition to the keyboard 512, an input means such as a touch panel for operating a predetermined button, icon, etc. may be used. The pointing device 513 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The media I / F 515 controls reading or writing (storing) data from and to a recording medium 514 such as a flash memory. The external device connection I / F 516 is an interface for connecting various external devices. In this case, the external device may be, for example, a USB (Universal Serial Bus) memory, etc. The sound input / output I / F 517 is a circuit for processing input / output of sound signals between a microphone 518 and a speaker 519 under the control of the CPU 501. The microphone 518 is a built-in circuit that converts sound into an electrical signal, and obtains information using the electrical signal by acquiring voice or sound waves emitted from an external speaker or the like. The speaker 519 is a built-in circuit that converts an electrical signal into physical vibrations to generate sounds such as music or voice. The bus line 520 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501.

[0049] The above program may be recorded in a computer-readable recording medium as an installable or executable file, or may be distributed by downloading via a network. Examples of recording media include CD-Rs (Compact Disc Recordable), DVDs (Digital Versatile Disks), Blu-ray Discs (Blu-ray is a registered trademark; omitted below), SD cards, and USB memories. The recording media may be provided domestically or internationally as a program product. For example, the writing device 3 realizes the writing processing method according to the present invention by executing the program according to the present invention.

[0050] [Functional configuration of the writing system] Next, the functional configuration of this embodiment will be described with reference to Fig. 9 to Fig. 12. Fig. 9 is a diagram showing an example of the functional configuration of the writing system according to the embodiment. Fig. 9 shows the communication terminal or device shown in Fig. 1 that is related to the processing or operation described later.

[0051] <Functional configuration of the writing device> Next, the functional configuration of the writing device 3 will be described with reference to FIG. 9. As shown in FIG. 9, the writing device 3 has a transmitting / receiving unit 31, a receiving / detecting unit 32, an acquiring unit 33, a calculating / converting unit 34, a judging unit 35, a generating unit 37, a processing unit 38, and a storing / reading unit 39. Each of these functional units is a function or means realized by operating any of the hardware resources shown in FIG. 7 by an instruction from the CPU 301 according to a program for the writing device 3 expanded from at least one of the ROM 302 and the EEPROM 304 to the RAM 303. The writing device 3 also has a storage unit 3000 (an example of a storage unit) constructed by at least one of the ROM 302 and the EEPROM 304 shown in FIG. 7. A communication program (communication application) for communicating with the information processing device 5 via the communication network 100 and the like are installed and stored in the storage unit 3000. Note that information related to a stroke image generated by a trajectory accompanying a writing operation is stored in the storage unit 3000.

[0052] <<Functional configuration of the writing device>> Next, each functional configuration of the writing device 3 will be described in detail. The transmission / reception unit 31 of the writing device 3 shown in FIG. 9 is mainly realized by the processing of the CPU 301 on the network I / F 311 and the near field communication I / F 308, and transmits and receives various data (or information) to and from the information processing device 5 via the communication network 100. When the information processing device 5 is in a state in which it can communicate with the writing device 3, the transmission / reception unit 31 transmits at least one of movement amount information indicating the amount of movement and coordinate information to the information processing device 5. The transmission / reception unit 31 also transmits at least one of the movement amount information and coordinate information, and writing-related information accompanying the writing operation to the information processing device 5. The transmission / reception unit 31 further receives setting information related to the writing operation transmitted by the information processing device 5. In this embodiment, the transmission / reception unit 31 functions as an example of at least one of the first transmission means and the reception means.

[0053] The reception detection unit 32 is realized by the CPU 301 processing various signals detected by the acceleration sensor 315, the pressure sensor 316, and the optical sensor 317, as well as signals generated by various operations received by the switch 318, and detects the amount of movement of the writing device 3. When the information processing device 5 that communicates with the writing device 3 is not in a state in which it can communicate with the writing device 3, the reception detection unit 32 detects the start position on the writing surface of the writing device 3 associated with the writing action. When the information processing device 5 is in a state in which it can communicate with the writing device 3, the reception detection unit 32 detects the amount of movement on the writing surface of the writing device 3 associated with the writing action. The reception detection unit 32 further detects the presence or absence of a writing instruction by the pressure sensor 316 provided in the writing device 3, and detects the amount of movement on the writing surface by the optical sensor 317 provided in the writing device 3. With this configuration, the writing device 3 calculates the relative position of the writing device 3 itself based on the amount of movement detected by the optical sensor 317. In this embodiment, the reception detection unit 32 functions as an example of a reception means and an example of a detection means.

[0054] The acquisition unit 33 is mainly realized by the processing of the CPU 301, and acquires various information received by the transmission / reception unit 31. In this embodiment, the acquisition unit 33 functions as an example of an acquisition means.

[0055] The calculation conversion unit 34 is mainly realized by the processing of the CPU 301, and calculates coordinate information obtained with the writing operation of the writing device 3. The calculation conversion unit 34 also corrects the movement amount of the writing device 3. The calculation conversion unit 34 also calculates coordinate information on the writing surface of the writing device 3 associated with the writing operation based on the detected start position on the writing surface. The calculation conversion unit 34 also calculates coordinate information on the writing surface of the writing device 3 based on the detected movement amount. The calculation conversion unit 34 further repeats the calculation of movement amount information when a part of the writing device 3 is in contact with the writing surface and the calculation of movement amount information when a part of the writing device 3 is separated from the writing surface based on contact presence / absence information (pressure information) on the writing surface of the writing device 3 and movement amount information on the writing surface of the writing device 3. Note that the contact presence / absence information (pressure information) in this embodiment is an example of writing instruction presence / absence information. In this embodiment, the calculation conversion unit 34 functions as an example of a calculation means and an example of a conversion means.

[0056] The determination unit 35 is mainly realized by the processing of the CPU 301, and performs various determinations in the writing device 3. In this embodiment, the determination unit 35 functions as an example of a determination means.

[0057] The generating unit 37 is mainly realized by the processing of the CPU 301, and generates movement amount information and contact presence / absence information accompanying a writing action on the writing device 3. In this embodiment, the generating unit 37 functions as an example of a generating means.

[0058] The processing unit 38 is mainly realized by the processing of the CPU 301 for the switch 318. The processing unit 38 also executes a data amount reduction process in the writing device 3. The processing unit 38 also executes a process for the switch 318 provided in the writing device 3. The processing unit 38 further operates various operations in the writing device 3, and reflects the received setting information in the writing device 3 according to at least one of the length of operation time and the operation pattern for the switch 318. In this embodiment, the processing unit 38 functions as an example of a second processing means.

[0059] The memory readout unit 39 is mainly realized by processing of the CPU 301 on at least one of the ROM 302 and the EEPROM 304, and stores various data (or information) in the memory unit 3000 and reads various data (or information) from the memory unit 3000. When the information processing device 5 that communicates with the writing device 3 is not in a state in which it can communicate with the writing device 3, the memory readout unit 39 stores the calculated coordinate information in the memory unit 3000 as information related to a stroke image. In this embodiment, the memory readout unit 39 functions as an example of a memory control means.

[0060] ●Coordinate information management table● Fig. 10 is a conceptual diagram showing an example of a coordinate information management table according to an embodiment. Note that the structure and contents of the data table described below are merely examples and are not limited thereto. A coordinate information management DB 3001 configured by a coordinate information management table as shown in Fig. 10 is constructed in the storage unit 3000. In the coordinate information management table, date and time information, switch ON / OFF, the presence or absence of writing instructions, and coordinate information are associated with each data ID and stored and managed as information related to a stroke image.

[0061] Among these, the data ID is identification information for identifying information related to a stroke image accompanying a writing action performed by the writing device 3. Here, the information related to the stroke image includes at least a plurality of pieces of coordinate information on the writing surface and a data ID (an example of identification information) for identifying each of the coordinate information. The date and time information is the date and time when the writing action was performed. The switch ON / OFF is a status flag indicating the ON (on) or OFF (off) state of the switch 318 provided on the writing device 3. The presence or absence of a writing instruction is a status flag indicating a state in which a load on the pressure sensor 316 in the writing device 3 or a signal from the pressure sensor 316 is detectable when a writing action is performed by a user. The coordinate information is each of the coordinate information of the X coordinate and the Y coordinate related to two-dimensional coordinates indicated by the XY coordinate. In the coordinate information management table, while a writing action is being performed by a user, the date and time information is updated according to the passage of time during which the writing action is being performed, and the coordinate information corresponding to the updated date and time information is updated (accumulated).

[0062] In the coordinate information management table, among the information related to the stroke image, coordinate information, switch ON / OFF information, and writing instruction presence / absence information may be managed as image information, and date and time information may be managed as writing-related information. However, there is no particular restriction on which information is to be image information or writing-related information. Also, in this embodiment, the writing instruction presence / absence state by the pressure sensor 316 may be prioritized over the switch ON / OFF state. In other words, even if there is no operation on the switch 318 (even if the switch is OFF), if the writing instruction is "present", it is possible to save the coordinate information (even if the switch 318 is forgotten to be pressed, it is possible to save the coordinate information).

[0063] In this embodiment, the coordinate information management table (coordinate information management DB 3001) functions as an example of a coordinate information management means.

[0064] <Functional configuration of information processing device> Returning to FIG. 9, the functional configuration of the information processing device 5 will be described. As shown in FIG. 9, the information processing device 5 has a transmission / reception unit 51, an operation reception unit 52, an acquisition unit 53, a display control unit 54, a determination unit 55, a registration unit 56, a generation unit 57, a processing unit 58, and a storage / reading unit 59. Each of these functional units is a function or means realized by operating any of the hardware resources shown in FIG. 8 by an instruction from the CPU 501 according to a program for the information processing device 5 expanded from at least one of the ROM 502, the EEPROM 504, the HD 505, and the recording medium 514 to the RAM 503. The information processing device 5 also has a storage unit 5000 constructed by at least one of the ROM 502, the EEPROM 504, the HD 505, and the recording medium 514 shown in FIG. 8. Furthermore, a communication program (communication application), a browser application, and the like for communicating with each device and each terminal via the communication network 100 are installed and stored in the storage unit 5000.

[0065] <<Functional configuration of information processing device>> Next, each functional configuration of the information processing device 5 will be described in detail. The transmitting / receiving unit 51 of the information processing device 5 shown in Fig. 9 is mainly realized by the processing of the CPU 501 on the network I / F 511 and the short-range communication I / F 508, and transmits and receives various data (or information) between each terminal and each device (writing device 3) via the communication network 100. The transmitting / receiving unit 51 also receives the coordinate information and writing-related information transmitted by the writing device 3. The transmitting / receiving unit 51 also transmits setting information related to the writing operation in the writing device 3 to the writing device 3. In this embodiment, the transmitting / receiving unit 51 functions as an example of at least one of the second transmitting means and the receiving means.

[0066] The operation acceptance unit 52 is mainly realized by the processing of the CPU 501, and for example, acquires a posted document (file) transmitted (posted) by the writing device 3. The operation acceptance unit 52 also acquires first option item information representing a first option item for selecting the details of the document transmitted by the writing device 3. In this embodiment, the operation acceptance unit 52 functions as an example of an acquisition unit.

[0067] The acquisition unit 53 is mainly realized by the processing of the CPU 501, and calculates and analyzes the breakdown of a plurality of documents in the information processing device 5. In this embodiment, the acquisition unit 53 functions as an example of a calculation means and an example of a processing means.

[0068] The display control unit 54 is mainly realized by the processing of the CPU 501 for the display 507 or other communication terminals connected via the communication network 100. The display control unit 54 controls the display of various screens and information (data) for the information processing device 5. The display control unit 54 also uses, for example, a Web browser to display a display screen generated by HTML or the like on the display 507 of the information processing device 5 or the display of other communication terminals connected to the information processing device 5 via the communication network 100. The display control unit 54 further starts and executes a Launcher that runs on an OS (Operating System) and an external application, thereby displaying various screens generated by an API (Application Programming Interface) provided by the OS on the display 507 or the display of other communication terminals. In this embodiment, the display control unit 54 functions as an example of a display control means.

[0069] The determination unit 55 is realized mainly by the processing of the CPU 501, and performs the determination processing of the entire information processing device 5. In this embodiment, the determination unit 55 functions as an example of a determination means.

[0070] The registration unit 56 is mainly realized by the processing of the CPU 501, and registers various information in the information processing device 5. In the present embodiment, the registration unit 56 functions as an example of a registration means.

[0071] The generation unit 57 is mainly realized by the processing of the CPU 501, selects various information in the information processing device 5, and generates various data including screen data related to various screens. In this embodiment, the generation unit 57 functions as an example of a generation means.

[0072] The processing unit 58 is mainly realized by the processing of the CPU 501, and executes various processes in the information processing device 5. The processing unit 58 also performs drawing processing of drawing data to be displayed on the display 507 or another communication terminal based on information related to the stroke image transmitted by the writing device 3 (image information (coordinate information, switch ON / OFF information, information on the presence or absence of writing instructions), writing-related information (various metadata including date and time information; hereinafter referred to as "writing-related information")). The processing unit 58 also divides the coordinate information representing the stroke image transmitted by the writing device 3 into a predetermined group based on the writing-related information and processes it. In this embodiment, the processing unit 58 functions as an example of a first processing means.

[0073] The storage / reading unit 59 is mainly realized by processing of the CPU 501 on at least one of the ROM 502, the EEPROM 504, the HD 505, and the recording medium 514, and stores various data (or information) in the storage unit 5000 and reads various data (or information) from the storage unit 5000. In this embodiment, the storage / reading unit 59 functions as an example of a storage / reading means.

[0074] ●Coordinate information management table● Fig. 11 is a conceptual diagram showing an example of a coordinate information management table according to an embodiment. Note that the structure and contents of the data table described below are merely examples and are not limited thereto. A coordinate information management DB 5001 configured by a coordinate information management table as shown in Fig. 11 is constructed in the storage unit 5000. In the coordinate information management table, various information including date and time information, position information, and related person information as writing-related information, as well as writing instruction presence / absence information and coordinate information as image information, are associated with each data ID and stored and managed as information related to a stroke image.

[0075] Among these, the position information as the writing-related information includes GPS (Global Positioning System) information for the writing device 3, beacon information, information related to other users who are in the vicinity of the user using the writing device 3 or in the same space (user name, user ID, etc.), and the like. As for the beacon information, it is possible for other users who are in the vicinity of the user or in the same space to share such information by communicating with each other. As a result, the information processing device 5 receives, stores, and manages information related to the writing action using the writing device 3 and information related to the user using the writing device 3 from the writing device 3 in association with the data ID or date and time information when the writing action was performed. In this way, the coordinate information management DB 3001 also stores and manages the position information. As a result, it is possible to manage the writing action in association with the user's experience, such as where the writing action was performed. In addition, the coordinate information management DB 3001 also stores and manages related party information. As a result, it is possible to manage the record of who the writing content was recorded with.

[0076] In this embodiment, the coordinate information management table (coordinate information management DB 5001) functions as an example of a coordinate information management means and an example of a metadata management means.

[0077] ●Split information management table● Fig. 12 is a conceptual diagram showing an example of a division information management table according to an embodiment. Note that the structure and contents of the data table described below are merely examples and are not limited thereto. A division information management DB 5002 configured by a division information management table as shown in Fig. 12 is constructed in the storage unit 3000. In the division information management table, a user ID, a user name, a schedule, and a data ID are stored and managed in association with each piece of date and time information.

[0078] Among these, the user ID is identification information for identifying a user who uses the writing device 3, such as an email address. The schedule indicates the schedule of a user indicated by a user name at each date and time. The data ID indicates a collection of image data (stroke image data) obtained in association with writing actions performed at a meeting or the like held according to the above-mentioned schedule. The data ID indicates, for example, that a collection of stroke images obtained at the XX meeting held on 2023 / 02 / 20 10:00-11:00 is managed as D0001-D1842.

[0079] In this embodiment, the division information management table (division information management DB 5002) functions as an example of a division information management means.

[0080] [Processing or Operation of the Embodiment] Next, each process or operation in the writing system according to the first embodiment will be described with reference to FIGS.

[0081] <Overall processing in writing device> FIG. 13 is a flowchart showing an example of the overall process in the writing device according to the first embodiment. Note that the flowchart shown in FIG. 13 is an example, and is not limited thereto. First, a user using the writing system 1 performs a writing action using the writing device 3. The writing surface in this writing action includes paper and a tablet terminal. The writing surface also includes the surface of a three-dimensional structure (desk, wall surface, etc.) having a flat or curved surface. In this case, there is no particular restriction on the three-dimensional structure. The writing surface further includes the surface of a human body (arm, palm, etc.), clothing worn on the human body, or a product such as a wristband. Thereby, the reception detection unit 32 detects the presence or absence of pressure on the tip of the writing device 3 (device tip 332) (step S11). Specifically, the reception detection unit 32 detects the pressure on the device tip 332 detected by the pressure sensor 316.

[0082] In step S11, the acceptance detection unit 32 monitors the writing action of the user to determine whether to save the movement of the writing device 3 detected by the optical sensor 317 as writing. For example, if the writing device 3 is a pen-shaped device, the acceptance detection unit 32 accepts it as a writing instruction when the pen tip (device tip 332) is in contact with a desk, a wall, or the like. In this way, the acceptance detection unit 32 performs detection that matches the shape of the writing device 3 in cooperation with the determination unit 35. Note that the acceptance detection unit 32 performs detection that allows determination based on pressure as well, such as accepting it as a writing instruction only when the pen tip (device tip 332) is pressed against a desk or the like with a certain amount of pressure or more.

[0083] Next, the reception detection unit 32 detects the amount of movement of the tip of the writing device 3 (step S12). Specifically, the reception detection unit 32 detects the amount of movement of the device tip 332 of the writing device 3 based on the signal detected by the optical sensor 317 in response to the movement of the writing device 3 accompanying the writing action. Note that the detection of the pressure on the device tip 332 and the amount of movement of the device tip 332 is based on the configuration and functions described in Figs. 3 and 4 above.

[0084] Next, the calculation conversion unit 34 corrects the amount of movement (step S13). The process here corresponds to, for example, a process of correcting the amount of movement of the device tip 332 to a value that can be used in the actual writing device 3 (tilt correction).

[0085] <Connection determination process, writing operation determination process, and coordinate information storage process> Next, the writing device 3 executes a writing operation determination process and a coordinate information storage process (step S14). The process of step S14 will be described in detail below.

[0086] <<Details of connection determination process, writing operation determination process, and coordinate information storage process>> Here, the details of the process of step S14 described above will be explained. Fig. 14 is a flowchart showing an example of the process including the process of determining whether writing is in progress and the process of saving coordinate information, among the overall processes in the writing device according to the first embodiment. Note that the flowchart shown in Fig. 14 is an example, and the present invention is not limited to this. First, the determination unit 35 of the writing device 3 determines whether there is a connection with the information processing device 5 (step S14-1). Specifically, the determination unit 35 determines whether there is a connection with the information processing device 5 via the transmission / reception unit 31.

[0087] If there is a connection with the information processing device 5 (step S14-1: YES), the generating unit 37 generates movement amount information and contact presence / absence information and exits this flow (step S14-2). The movement amount information and contact presence / absence information are information to be transmitted to the information processing device 5 in the process described later. The contact presence / absence information here is information similar to a status flag indicating the presence or absence of a writing instruction. The movement amount information and contact presence / absence information (writing instruction presence / absence information) generated in step S14-2 are stored in the corresponding items of the coordinate information management DB 3001 (see FIG. 10) by the memory reading unit 39. Thereafter, the movement amount information stored in the coordinate information management DB 3001 is transmitted to the information processing device 5 via the transmitting / receiving unit 31 of the writing device 3 by the process described later when the writing device 3 is connected to the information processing device 5.

[0088] At this time, the writing device 3 can perform cursor operations such as those shown in FIG. 6 on the information processing device 5 by transmitting movement amount information to the information processing device 5 regardless of whether a writing instruction has been received.

[0089] On the other hand, when there is no connection with the information processing device 5 (step S14-1: NO), the writing device 3 executes a process of determining whether writing is in progress (step S14-3). Specifically, in order to store writing information (image information) in the writing device 3 itself, the writing start position is set as a start coordinate (origin), coordinate information related to the writing coordinates is calculated based on the moving direction, the moving amount, and the like, and the calculated coordinate information is stored. As a method of determining whether writing is in progress or not, the user may turn on / off (ON / OFF) the switch 318. In this case, when writing, the user can turn on the switch 318 to cause the writing device 3 itself to record the movement of the writing device 3 in that state. This series of processes is, for example, the same operation as knocking a ballpoint pen. Details of the process in the above-mentioned step S14-3 will be described later.

[0090] <<Details of the process to determine whether writing is in progress>> FIG. 15 is a flowchart showing an example of details of the writing operation determination process among the overall processes in the writing device according to the first embodiment. The flowchart shown in FIG. 15 is an example, and is not limited thereto. First, the determination unit 35 of the writing device 3 determines whether there is a writing instruction (step S14-3-1). Specifically, the determination unit 35 reads and checks the status flag of the presence or absence of a writing instruction corresponding to the current date and time information managed in the coordinate information management DB 3001 (see FIG. 10) via the storage and reading unit 39. The status flag of the presence or absence of a writing instruction is detected by the reception detection unit 32 of the output of the pressure sensor described in FIG. 3, and the result of detecting the presence or absence of a writing instruction on the writing surface based on the detection result is stored and managed in real time as a flag of "present" or "absent". As a result, the process of step S14-3-1 is executed as a process of detecting the start position (writing instruction) on the writing surface of the writing device 3 accompanying the writing operation.

[0091] When it is determined that there is a writing instruction (step S14-3-1: YES), the memory readout unit 39 overwrites and reserves the writing action timer (step S14-3-2).

[0092] Next, the processing unit 38 continues the process assuming that the writing device 3 is writing, and exits this flow (step S14-3-3).

[0093] On the other hand, if it is determined that there is no writing instruction (step S14-3-1: NO), the determination unit 35 further determines whether the timer has been started (step S14-3-4).

[0094] When it is determined that the timer has not been started (step S14-3-4: NO), the determination unit 35 transitions to the process of step S14-3-3 described above.

[0095] On the other hand, if it is determined that the timer has been started (step S14-3-4: YES), the processing unit 38 continues the process assuming that the writing device 3 is not currently writing, and exits this flow (step S14-3-5).

[0096] As described above, a method of determining whether a series of writing actions is in progress may involve measuring the interval time of the state in which a writing instruction is present detected by the reception detection unit 32, and determining that a writing action is in progress if the interval time is shorter than a predetermined time set in advance. For example, if the predetermined time is set to one second, the determination unit 35 determines that a movement without a writing instruction for less than one second is also a writing action, and stores the coordinate information in the corresponding coordinate information item of the coordinate information management DB 3001 (see FIG. 10) via the storage and reading unit 39. The predetermined time may be set or changed by the user. This allows the writing device 3 to appropriately determine whether a writing action is in progress for each user.

[0097] It should be noted that the determination based on time is effective for one character or one figure. However, when a user writes while thinking, or when the writing speed differs depending on the user, the stroke image accompanying the writing action may not be saved. Therefore, the writing device 3 utilizes the fact that the writing device 3 moves from left to right on the writing surface or from top to bottom on the writing surface when writing a character. As a result, the writing device 3 may save the coordinate information as being in writing action when the writing device 3 moves to the right or bottom of the writing surface while no writing instruction is received. In addition, since there are cases where writing is done from right to left on the writing surface depending on the language, such as vertical writing in Japanese, the writing device 3 may be configured to be able to set a predetermined time as appropriate. By performing the above-mentioned processing, the writing device 3 can save information related to the stroke image as a series of writing contents even when the writing speed is slow, such as when the user writes while thinking. Furthermore, it is also possible to store the movement of each stroke to the beginning of writing as information related to the stroke image accompanying the writing action (the positional relationship between each writing action and each stroke image can be known).

[0098] 14, the calculation conversion unit 34 calculates coordinate information (step S14-4). Specifically, the calculation conversion unit 34 calculates coordinate information on the writing surface of the writing device 3 accompanying the writing action, based on the start position of the writing device 3 detected in association with the movement of the writing device 3 accompanying the writing action.

[0099] Next, the writing device 3 executes a coordinate information storage process and exits this flow (step S14-5). The process of the above-mentioned step S14-3 will be described in detail later.

[0100] <<Details of the coordinate information saving process>> FIG. 16 is a flowchart showing an example of details of the coordinate information storage process among the overall processes in the writing device according to the first embodiment. Note that the flowchart shown in FIG. 16 is an example, and the present invention is not limited to this. First, the acquisition unit 33 of the writing device 3 acquires current date and time information (step S14-5-1). Specifically, the acquisition unit 33 acquires the current date and time information after it is determined that a writing operation is in progress (there is a writing instruction) in the writing operation determination process in step S14-3. Here, the acquisition unit 33 can acquire date and time information obtained by, for example, an internal clock that drives the CPU 301 of the writing device 3, or date and time information obtained by connecting to the communication network 100 via the transmission / reception unit 31.

[0101] Next, the determination unit 35 determines whether or not there is stored coordinate information (step S14-5-2). Specifically, the determination unit 35 refers to the coordinate information management DB 3001 (see FIG. 10) and determines whether or not there is stored coordinate information for date and time information that corresponds to the current date and time information or the closest date and time information.

[0102] If it is determined that there is no stored coordinate information (step S14-5-2: NO), the memory readout unit 39 stores (stores) the date and time information (step S14-5-3). Specifically, the memory readout unit 39 stores (stores) new date and time information (current date and time information) in the date and time information item of the coordinate information management DB 3001.

[0103] Next, the memory readout unit 39 saves the coordinate information (step S14-5-4) and exits this flow. Specifically, the memory readout unit 39 saves (stores) the coordinate information (an example of information related to the stroke image) in the coordinate information corresponding to the current date and time information stored in step S14-5-3, and exits this flow.

[0104] On the other hand, if it is determined that there is stored coordinate information (step S14-5-2: YES), the determination unit 35 further determines whether the difference between the date and time information of the latest stored data and the current date and time information is smaller than a predetermined time (step S14-5-5). Specifically, if coordinate information has already been stored in the coordinate information management DB 3001, the determination unit 35 checks the date and time information associated with the coordinates, and determines whether the difference between the date and time information and the current date and time information is smaller than a predetermined time (e.g., 3 minutes).

[0105] If the difference with the current date and time information is less than the predetermined time (step S14-5-5: YES), the memory readout unit 39 executes the process of step S14-5-4 described above and exits this flow. Specifically, for example, if the difference with the current date and time information is within 3 minutes, the memory readout unit 39 determines that the amount of movement of the writing device 3 accompanying the currently detected writing action is a writing action related (continuous) to the amount of movement in the past, and additionally saves (stores) only the coordinate information in the coordinate information management DB 3001. This makes it possible to store, for example, the movement to the beginning of a stroke as information related to the stroke image accompanying the writing action when the writing action is not a single stroke.

[0106] On the other hand, if the difference with the current date and time information is not smaller than the predetermined time (step S14-5-5: NO), the memory readout unit 39 executes the processes of steps S14-5-3 and S14-5-4 described above, respectively, and exits this flow. Specifically, for example, if the difference with the current date and time information exceeds 3 minutes, the memory readout unit 39 additionally saves (stores) new date and time information and coordinate information in the coordinate information management DB 3001. In other words, in this case, it is determined that the amount of movement of the writing device 3 accompanying the currently detected writing action is a writing action that is unrelated to the amount of movement in the past.

[0107] As described above, in the coordinate information saving process, the saving (storing) process of the date and time information and the coordinate information managed in the coordinate information management DB 3001 is controlled based on the presence or absence of saved coordinate information and the difference between the date and time information of the latest saved data and the current date and time information. This is expected to improve the efficiency of the processing of information related to stroke images in the writing device 3 by omitting unnecessary saving (storing) processes.

[0108] ●Management status in the coordinate information management table● Next, the respective management states in the coordinate information management DB 3001 (see FIG. 10) of the data including the coordinate information obtained by the above-mentioned coordinate information storage process will be described.

[0109] Fig. 17 is a conceptual diagram showing an example of the state of the coordinate information management table according to the first embodiment. Fig. 17 shows the management state in the coordinate information management DB3001 after detection of movement with no difference in date and time information. Fig. 17 shows a state in which the start position of the writing action is set to the start coordinates (origin: (0,0)), coordinate information is calculated based on the movement direction, movement amount, etc., and the calculated coordinate information is stored.

[0110] It is known that the movement of the writing device 3 is detected in milliseconds, and multiple pieces of coordinate information accompanying the writing action can be stored during writing instructions in accordance with the time unit of detection. For example, when the start coordinate is (0,0), if (Δx, Δy) = (1,3) and (4,3) are successively detected as the movement of the writing device 3, the coordinate information corresponding to data ID = "D0047" is stored as (5,6), which is the sum of the components of the coordinate information of (1,3) and (4,3).

[0111] However, in the storage state of the handwriting information shown in Fig. 17, the positional relationship between each handwriting action (including a single-stroke writing) cannot be stored, and characters, figures, etc. cannot be accurately reproduced. Therefore, the handwriting device 3 manages the coordinate information management DB 3001 as shown in Fig. 18.

[0112] Fig. 18 is a conceptual diagram showing an example of another state of the coordinate information management table according to the first embodiment. The state shown in Fig. 18 is one of the states managed by execution of the writing action determination process in step S14-3 described above. As explained in the definition of a writing action, in Fig. 18, the date and time information, switch ON / OFF, and coordinate information of data IDs ("D0003" and "D0004") with no writing instructions are managed as a state in which writing action and moving action are consecutive and a series of characters or figures are being drawn.

[0113] Also, as shown in FIG. 18, movement without a writing instruction results in the start coordinates of the next writing instruction (for example, the coordinate information of data ID "D0004" and the coordinate information of data ID "D0005").

[0114] Fig. 19 is a conceptual diagram showing an example of another state of the coordinate information management table according to the first embodiment. Fig. 19 shows a case where data with no writing instructions is deleted from the state of Fig. 18. This is expected to improve the efficiency of the coordinate information storage process of the writing device 3.

[0115] Fig. 20 is a conceptual diagram showing an example of a state in the coordinate information storage process of the coordinate information management table according to the first embodiment. The state shown in Fig. 20 is one of the states stored and managed by the process of step S14-5-4, which is executed after step S14-5-5: YES in the coordinate information storage process described above. As shown in Fig. 20, the date and time information in data ID "D0002" is the same as the date and time information in data ID "D0001" (the difference is smaller than a predetermined time), so storage of the date and time information is omitted, and "-" is stored and managed instead.

[0116] Fig. 21 is a conceptual diagram showing an example of another state in the coordinate information storage process of the coordinate information management table according to the first embodiment. In Fig. 21, states (data IDs "D0003" and "D0004") after movement is detected with no difference in date and time information (for example, the difference is smaller than a predetermined time) are additionally stored and managed.

[0117] Fig. 22 is a conceptual diagram showing an example of another state in the coordinate information storage process of the coordinate information management table according to the first embodiment. Fig. 22 shows a state after detection of a movement with a difference in date and time information (for example, the difference is a predetermined time or more). In Fig. 22, date and time information four minutes after the date and time information corresponding to data ID "D0001" (date and time information corresponding to data ID "D0003") is additionally stored and managed.

[0118] FIG. 23 is a conceptual diagram showing an example of another state in the coordinate information storage process of the coordinate information management table according to the first embodiment. In the example shown in FIG. 22, a series of writing actions are distinguished in units of minutes, but in actual writing actions, the writing location and writing contents change in units of several seconds or milliseconds. The example shown in FIG. 23 takes into consideration the case where date and time (time) precision is required during writing actions. In this way, by storing date and time information only at the beginning of a series of writing actions and only time information thereafter, more detailed processing can be performed. For example, when only time information is stored, by storing it down to milliseconds, it becomes possible to obtain information such as writing speed, and it becomes possible to not only display but also reproduce or replay the stored writing information. A display example in the information processing device 5 related to this reproduction or replay will be described later.

[0119] <Data volume reduction process> 13, the writing device 3 executes the data amount reduction process and exits from this flow (step S15). Note that the process of step S15 will be described in detail below.

[0120] <<Details of data volume reduction process>> Here, the details of the process of step S15 described above will be explained. FIG. 24 is a flowchart showing an example of details of the data amount reduction process among the overall processes in the writing device according to the first embodiment. Note that the flowchart shown in FIG. 24 is an example, and the present invention is not limited to this. First, the acquisition unit 33 of the writing device 3 acquires current date and time information (step S15-1). Specifically, the acquisition unit 33 acquires the current date and time information after the coordinate information storage process of step S14. Here, the acquisition unit 33 can acquire date and time information acquired by an internal clock that drives the CPU 301 of the writing device 3, or date and time information acquired by connecting to the communication network 100 via the transmission / reception unit 31, for example.

[0121] Next, the determination unit 35 determines whether or not there is stored coordinate information (step S15-2). Specifically, the determination unit 35 refers to the coordinate information management DB 3001 (see FIG. 10) and determines whether or not there is stored coordinate information for date and time information that corresponds to the current date and time information or the closest date and time information.

[0122] If it is determined that there is no stored coordinate information (step S15-2: NO), the memory / read unit 39 stores (stores) the date and time information (step S15-3). Specifically, the memory / read unit 39 stores (stores) new date and time information (current date and time information) in the date and time information item of the coordinate information management DB 3001.

[0123] Next, the memory / read unit 39 stores the coordinate information (step S15-4) and exits this flow. Specifically, the memory / read unit 39 stores (stores) the coordinate information in the coordinate information corresponding to the current date and time information stored in step S15-3 and exits this flow.

[0124] On the other hand, if it is determined that there is stored coordinate information (step S15-2: YES), the determination unit 35 further determines whether the difference between the date and time information of the latest stored data and the current date and time information is smaller than a predetermined time (step S15-5). Specifically, if coordinate information has already been stored in the coordinate information management DB 3001, the determination unit 35 checks the date and time information associated with the coordinates, and determines whether the difference between the date and time information and the current date and time information is a predetermined time (e.g., 3 seconds).

[0125] If the difference with the current date and time information is smaller than the predetermined time (step S15-5: YES), the judgment unit 35 further judges whether the difference (amount of movement) of the latest stored data is similar (step S15-6). Specifically, the judgment unit 35 refers to the coordinate information management DB 3001 (see FIG. 10) via the memory readout unit 39, and judges whether there is information similar (close) to the amount of movement (coordinate information) of the latest stored data.

[0126] If it is determined that the difference (amount of movement) is not similar to the latest stored data (step S15-6: NO), the judgment unit 35 proceeds to the processing of step S15-4 described above, executes the processing of storing the coordinate information via the memory reading unit 39, and exits this flow.

[0127] On the other hand, if it is determined that the difference (movement amount) of the latest stored data is similar (step S15-5: YES), the calculation conversion unit 34 adds it to the coordinate information of the latest data (step S15-7) and exits this flow. Specifically, the calculation conversion unit 34 adds the current coordinate information to the coordinate information of the latest data in the coordinate information management DB 3001 via the storage and reading unit 39, and then exits this flow. In other words, the calculation conversion unit 34 updates (adds) the coordinate information of the latest data in the coordinate information management DB 3001 via the storage and reading unit 39.

[0128] Furthermore, in the process of step S15-5, if the difference with the current date and time information is not smaller than the predetermined time (step S15-5: NO), the memory readout unit 39 executes the processes of steps S15-3 and S15-4 described above, respectively, and exits this flow. Specifically, for example, if the difference with the current date and time information exceeds 3 seconds, the memory readout unit 39 additionally saves (stores) new date and time information and coordinate information in the coordinate information management DB 3001 and exits this flow. In other words, in this case, it is determined that the amount of movement of the writing device 3 accompanying the currently detected writing action is a writing action unrelated to past movement amounts, and therefore is not subject to data amount reduction.

[0129] As described above, in the coordinate information saving process, based on the presence or absence of saved coordinate information and the difference between the date and time information of the latest saved data and the current date and time information, a process is performed to reduce the amount of data related to the date and time information and the coordinate information managed in the coordinate information management DB 3001. By doing so, it is expected that an increase in the amount of data in the storage means of the writing device 3 can be avoided by reducing unnecessary data, and it is also expected that the writing device 3 can be made smaller and less expensive.

[0130] ●Management status in the coordinate information management table● FIG. 25 is a conceptual diagram showing an example of a state in the data amount reduction process of the coordinate information management table according to the first embodiment. FIG. 25 shows a management state in the coordinate information management DB 3001 after detecting a movement with no difference in date and time information. FIG. 25 shows a state in which the start position of the writing action is set as the start coordinate (origin: (0,0)), coordinate information is calculated based on the movement direction and amount of movement, and the calculated coordinate information is stored. Here, as an example, a case where the coordinate information corresponding to the data ID "D0002" as the stored information is in a state of (x,y)=(1,3) will be described. In this state, a new movement amount (2,6) is detected, but if there is no difference in the date and time information and the movement amount is similar, it is determined that the value is similar even if it is different from the previous movement amount, and the result of adding it to the coordinate information corresponding to the data ID "D0002" is displayed. Therefore, the coordinate information corresponding to the data ID "D0002" is newly stored and managed as (3,9) obtained by adding (2,6) to (1,3). The term "similar" here refers to the result of judging a direction (vector) determined by the amount of movement expressed as (x, y), for example.

[0131] In the above example, if the stroke is not determined to be a series of writing actions, the coordinates are saved as the origin (0,0). However, it is possible to change the position of the origin so that the stroke is displayed in a position that is easy to see in the drawing application being used.

[0132] Note that each process described so far in the first embodiment may be executed by the information processing device 5 when the handwriting device 3 is connected to the information processing device 5 and is in a communicable state.

[0133] [Major Effects of the First Embodiment] As described above, according to this embodiment, when the information processing device 5 that communicates with the writing device 3 is not in a state in which it can communicate with the writing device 3, the writing device 3 detects the start position of the writing action on the writing surface of the writing device 3 (step S14-3-1), calculates coordinate information on the writing surface of the writing device 3 associated with the writing action based on the detected start position (step S14-4), and stores the calculated coordinate information in the storage unit 3000 as information related to the stroke image (step S14-5-4). This provides the effect that, when a writing action is performed using the writing device, information related to the stroke image generated in association with the writing action can be stored in the writing device itself.

[0134] Second Embodiment Next, a second embodiment will be described. In the second embodiment, a case will be described in which the writing device 3 and the information processing device 5 can communicate with each other while connected via a communication network 100 or a wireless communication interface such as Bluetooth. Note that the second embodiment is similar to the system configuration, hardware configuration, and functional configuration of the writing system described in the first embodiment, and therefore description thereof will be omitted.

[0135] <Processing for displaying stroke images of a writing device on an information processing device> FIG. 26 is a sequence diagram showing an example of processing between the writing device and the information processing device according to the second embodiment. First, in order to establish communication between the writing device 3 and the information processing device 5, an application is started and communication is established by a predetermined communication protocol (step S21). Note that the communication establishment may be performed by an existing method using a predetermined session ID or the like. Hereinafter, the second embodiment will be described on the assumption that the writing device 3 and the information processing device 5 maintain a state in which they can communicate with each other.

[0136] Next, the transmitting / receiving unit 51 of the information processing device 5 transmits a handwriting information acquisition request to the writing device 3 (step S22). As a result, the transmitting / receiving unit 31 of the writing device 3 receives the handwriting information acquisition request that the information processing device 5 wants to transmit. At this time, the handwriting information acquisition request may include, for example, information that identifies the writing device 3. Furthermore, the transmitting / receiving unit 51 of the information processing device 5 may transmit setting information related to the writing action in the writing device 3 to the writing device 3 in the process of step S22. As a result, the writing device 3 can set and reflect the setting information related to the writing action transmitted by the information processing device 5 in its own device by starting a dedicated application installed in itself. Note that the writing device 3 may be used by each of multiple users in the writing system 1. That is, multiple writing devices 3 may exist.

[0137] Next, the acquisition unit 33 of the writing device 3 acquires the stored data (step S23). Specifically, the acquisition unit 33 reads and acquires the data ID, image information (coordinate information, switch ON / OFF information, information on the presence or absence of writing instructions), and writing-related information (date and time information) stored and managed in the coordinate information management DB 3001 (see FIG. 10) via the storage and reading unit 39.

[0138] Next, the transmitting / receiving unit 31 transmits a handwriting information acquisition response to the information processing device 5 as a response to the handwriting information acquisition request of step S22 (step S24). As a result, the information processing device 5 receives the handwriting information acquisition response transmitted by the writing device 3. At this time, the handwriting information acquisition response includes a data ID, coordinate information as image information, switch ON / OFF information, information on the presence or absence of a writing instruction, and date and time information as an example of writing-related information. Note that the processes of steps S22 to S24 described above may be executed, for example, by being triggered by an operation on a button or the like provided on the writing device 3 or an instruction operation from the display 507 of the information processing device 5.

[0139] Next, the registration unit 56 of the information processing device 5 registers the coordinate information and the writing-related information (step S25). Specifically, the registration unit 56 registers the data ID received in step S24, the coordinate information as image information, the switch ON / OFF information, the writing instruction presence / absence information, and the date and time information as the writing-related information, in the corresponding items of the coordinate information management DB 5001 (see FIG. 11) as information related to the stroke image. At this time, the registration unit 56 may further obtain the position information of the writing device 3 and the related person information of the user who uses the writing device 3 by using a GPS signal, a beacon signal, etc., and register them in the corresponding items of the coordinate information management DB 5001.

[0140] Next, the processing unit 58 divides the handwriting information (step S26). Specifically, the processing unit 58 searches the division information management DB002 (see FIG. 12) using the date and time information managed in the coordinate information management DB5001 (see FIG. 11) as a search key for the information related to the stroke image registered in step S25, thereby reading out the corresponding data ID. As a result, the processing unit 58 may treat the image data corresponding to the read data ID as divided image data related to the stroke image obtained for each schedule. As a result, the information processing device 5 divides the stroke image data for each related information regarding the writing action performed on the writing device 3, thereby making it possible to display the stroke images without overlapping each other.

[0141] Next, the acquisition unit 53 acquires the coordinate information and the writing-related information (step S27). Specifically, the acquisition unit 53 reads and acquires the writing information managed in the coordinate information management DB 5001 (see FIG. 11) and divided in step S26, i.e., the coordinate information and the writing-related information (metadata) corresponding to the data ID summarized in a predetermined schedule or the like, via the storage and reading unit 39. As the metadata, various data such as the conference name, conference location, participants, conference recording data managed in association with the conference, and audio recording data that can be acquired from the schedule can be utilized. Depending on the writing-related information that can be added by the writing device 3, it is also possible to utilize information recorded by other writing devices used at the site.

[0142] Next, the processing unit 58 converts the amount of movement into stroke data (step S28). Specifically, the processing unit 58 converts the amount of movement of the handwritten information into stroke data to be actually drawn.

[0143] Next, the display control unit 54 displays the stroke data (step S29). Specifically, the display control unit 54 displays the stroke data converted in step S28 on the display 507. Note that the method of converting the movement amount into stroke data can be, for example, by connecting each movement amount with a single line.

[0144] The above-mentioned steps S27 to S29 are repeatedly executed for each divided piece of handwritten information (loop processing). This makes it possible to process stroke data for each schedule, such as a meeting or a conference, separately and display stroke images drawn during the execution of other schedules without mixing them.

[0145] In the writing system according to this embodiment, for example, when the processes of steps S22 and S24 described above are executed, other devices may exist between the writing device 3 and the information processing device 5. In other words, each piece of information (data) transmitted and received between the writing device 3 and the information processing device 5 may be configured to be transmitted and received once via another device. The above-mentioned configuration can be applied even when other processing steps exist between the writing device 3 and the information processing device 5.

[0146] ●Screen display example● FIG. 27 is a diagram showing an example of a writing data display screen in the information processing apparatus according to the second embodiment. In FIG. 27, a writing data display screen 5111 is displayed by the display control unit 54. In the writing data display screen 5111, for example, a meeting name as schedule information and participants who participated in the meeting are displayed. In addition, date and time information at a certain point in time is also displayed, and a stroke image written at that date and time is displayed. Note that the stroke image may include not only an image written at the moment of that date and time, but also an image at a date and time (time) several seconds before that date and time. This is for displaying a meaningful character, information, etc. As a result, a stroke image written earlier than a date and time (time) at a certain point in time is displayed. A confirmation button 5151 is further displayed on the writing data display screen 5111. As a result, after the user confirms the contents of the writing data, the user can operate the confirmation button 5151 to transition to another screen.

[0147] ●Screen display example (replay display)● Here, an example will be described in which stroke images accompanying a writing action are replay-displayed at a predetermined time interval, and continuous stroke images are displayed like a video playback screen. FIG. 28A is a diagram showing an example of a writing data replay display screen in an information processing device according to a second embodiment. In FIG. 28A, a writing data replay display screen 5121 is displayed by a display control unit 54. The writing data replay display screen 5121 has the same display content as the above-mentioned writing data display screen 5111. However, the date and time information in the writing data replay display screen 5121 changes at a predetermined time interval, and stroke images corresponding to this change are displayed.

[0148] ●Screen display example (replay display)● Fig. 28B is a diagram showing another example of the handwriting data replay display screen in the information processing device according to the second embodiment. In Fig. 28B, the handwriting data replay display screen 5121 is displayed by the display control unit 54, and the date and time information shows a state 300 ms later (2023 / 06 / 12 10:08:15:600) than the date and time of Fig. 28A. This allows the user to check the state in which the stroke image has been drawn for 300 ms.

[0149] ●Screen display example (replay display)● Fig. 28C is a diagram showing another example of the handwriting data replay display screen in the information processing device according to the second embodiment. In Fig. 28C, the display control unit 54 displays the handwriting data replay display screen 5121, but the date and time information shows a state 300 ms later (2023 / 06 / 12 10:08:15:900) than the date and time of Fig. 28B. This allows the user to confirm the state in which the stroke image has been drawn for another 300 ms.

[0150] When performing the above-mentioned replay display, the smaller the change in the date and time information (every 300 ms in FIG. 28), the more realistic the replay display of the writing action can be confirmed.

[0151] <Processing for drawing and displaying multiple pieces of handwritten information without overlapping> Here, the processing of stroke images in the information processing device 5 will be described. The handwriting information stored in the writing device 3 is not necessarily one batch (a series of strokes only). If the information processing device 5 performs drawing processing while the writing device 3 is transmitting stroke data to the information processing device 5, the stroke images accompanying the writing action will be displayed overlapping. Therefore, the information processing device 5 performs processing to change the drawing area, page, etc. on the display 507 of the information processing device 5 for each divided piece of handwriting information. By performing such processing, the stroke images accompanying the writing action can be displayed without overlapping.

[0152] FIG. 29 is a diagram showing an example of a handwriting data display method according to the second embodiment, and FIG. 29(a) is a diagram showing an example of a case where stroke images are drawn overlapping each other by a writing action using a writing device. As shown in FIG. 29(a), the writing device 3 according to the embodiment generates a stroke image according to the writing action of the user, but the user writes without being aware of the writing surface. Therefore, there are cases where the writing action is repeated at almost the same place on the writing surface. As a result, a state where each stroke image is drawn overlapping each other as shown in FIG. 29(a) is generated.

[0153] Therefore, in the information processing device 5, it is necessary to decompose the overlapped stroke images into independent stroke images. FIG. 29(b) is an example of a conceptual diagram of decomposing (separating) overlapping stroke images drawn into each stroke image. FIG. 29(b) shows an example of a method of decomposing two different stroke images and determining their display areas. The information processing device 5, for example, calculates the vertical length: v1, v2 and the horizontal length: h1, h2 of each stroke image from the maximum and minimum values ​​of the coordinate information of the writing device 3. The state shown in FIG. 29(b) is a display area that is a rectangle larger than the area surrounded by the calculated v1, h1, and v2, h2. Here, the reason for making the rectangle larger than the area surrounded by each of the vertical and horizontal lengths is, for example, to ensure visibility when the stroke image is displayed on the information processing device 5. The information processing device 5 may calculate a non-overlapping area from a vector amount representing each stroke image.

[0154] FIG. 29(c) is an example of a display screen in which the decomposed stroke images are displayed on the information processing device so as not to overlap each other. The information processing device 5 displays on the display 507 of the information processing device 5 the stroke images decomposed in FIG. 29(b) and rectangles for displaying the stroke images obtained as a result of calculation. This allows the user to check a series of letters, numbers, etc. imagined by a writing action on the display 507 of the information processing device 5. When the stroke images are displayed separately (shifted), they may be displayed while maintaining the positional relationship of the images generated in FIG. 29(b) as shown in FIG. 29(c), or the ends (for example, the left ends) of the rectangles may be aligned and displayed. The separation (decomposition) process for the stroke image data described above may be executed in the process of step S29 (display of stroke data) described in FIG. 26. In this way, by decomposing (separating) a plurality of stroke image data associated with a writing action performed on the writing device 3, it becomes possible to display the stroke images without overlapping each other, improving visibility.

[0155] [Major Effects of the Second Embodiment] As described above, according to this embodiment, stroke image information associated with a writing action performed with the writing device 3 is transmitted to the information processing device 5 together with writing-related information (step S24), the transmitted stroke image information is divided based on schedule information or the like (step S26), and each divided stroke image is displayed on the display 507 (step S29). This provides the advantage that the user can clearly check the contents of a memo written on the palm of the hand using the writing device 3 in a certain schedule via the information processing device 5.

[0156] [Modification of the second embodiment] Next, a modified example of the second embodiment will be described. In the modified example of the second embodiment, the system configuration, hardware configuration, functional configuration, and each data table in the second embodiment can be shared, so that the description of these will be omitted.

[0157] The main difference between the second embodiment and the modified example of the second embodiment is whether or not a conversion process of the movement amount into position coordinate information is performed on the information processing device 5 side. This point will be described below.

[0158] FIG. 30 is a sequence diagram showing an example of processing between a writing device and an information processing device according to a modified example of the second embodiment. In the modified example of the second embodiment, the writing information acquisition response in step S34 and the conversion process of the position coordinate information of the movement amount in step S35 are different. Specifically, the image information included in the writing information acquisition response in step S34 is not coordinate information but information on the movement amount of the writing device 3. As a result, a conversion process (step S35) to the position coordinate information of the movement amount executed by the information processing device 5 is added. In this way, in the modified example of the second embodiment, the movement amount information of the writing device 3 is transmitted to the information processing device 5, and the conversion process to the coordinate information of the movement amount is performed on the information processing device 5 side.

[0159] Note that the processes in steps S31 to S33 in Fig. 30 are similar to the processes in steps S21 to S23 in Fig. 26 described in the second embodiment, and therefore their explanations will be omitted. Furthermore, the processes in steps S36 to S40 in Fig. 30 are similar to the processes in steps S25 to S29 in Fig. 26 described in the second embodiment, and therefore their explanations will be omitted.

[0160] In the writing system according to this embodiment, for example, when the processes of steps S32 and S34 described above are executed, other devices may exist between the writing device 3 and the information processing device 5. In other words, each piece of information (data) transmitted and received between the writing device 3 and the information processing device 5 may be configured to be transmitted and received once via another device. The above-mentioned configuration can be applied even when other processing steps exist between the writing device 3 and the information processing device 5.

[0161] [Major Effects of the Modification of the Second Embodiment] As described above, according to this embodiment, the writing device 3 transmits information about the amount of movement of the writing device 3 accompanying the writing action to the information processing device 5 (step S34), and the information processing device 5 converts the received information about the amount of movement of the writing device 3 into coordinate information (step S35). This makes it possible to omit the coordinate conversion process in the writing device 3, and an effect is achieved in that a reduction in the cost of the writing device 3 can be expected.

[0162] Third embodiment Next, a third embodiment will be described. In the third embodiment, a case will be described in which a stroke image accompanying a writing action on the writing device 3 is displayed in real time on the information processing device 5 side in a state in which the writing device 3 and the information processing device 5 are connected via a communication network 100 or a wireless communication interface such as Bluetooth. Note that the third embodiment is similar to the system configuration, hardware configuration, and functional configuration of the writing system described in the first embodiment, and therefore description thereof will be omitted.

[0163] <Processing for displaying stroke images of a writing device on an information processing device> Fig. 31 is a sequence diagram showing an example of processing between the writing device and the information processing device according to the third embodiment. First, in order to establish communication between the writing device 3 and the information processing device 5, an application is started and communication is established by a predetermined communication protocol (step S41). Note that the communication establishment may be performed by an existing method using a predetermined session ID or the like. Hereinafter, the second embodiment will be described on the assumption that the writing device 3 and the information processing device 5 maintain a state in which they can communicate with each other.

[0164] Next, the reception detection unit 32 of the writing device 3 receives a writing action (step S42). Specifically, the reception detection unit 32 receives the writing action of the writing device 3 from information such as the tilt direction and tilt angle of the writing device 3 detected by the pressure sensor 316 provided in the writing device 3.

[0165] Next, the transmitting / receiving unit 31 transmits the handwriting information to the information processing device 5 (step S43). As a result, the transmitting / receiving unit 51 of the information processing device 5 receives the handwriting information transmitted by the handwriting device 3. At this time, the handwriting information includes a data ID, coordinate information as image information, switch ON / OFF information, information on the presence or absence of a handwriting instruction, and date and time information as handwriting-related information.

[0166] Next, the registration unit 56 of the information processing device 5 registers the coordinate information and the writing-related information (step S44). Specifically, the registration unit 56 registers the data ID received in step S43, the coordinate information as image information, the switch ON / OFF information, the writing instruction presence / absence information, and the date and time information as an example of the writing-related information, in the corresponding items of the coordinate information management DB 5001 (see FIG. 11). At this time, the registration unit 56 may further obtain the position information of the writing device 3 and the related person information of the user who uses the writing device 3 by using a GPS signal, a beacon signal, etc., and register them in the corresponding items of the coordinate information management DB 5001.

[0167] The above-mentioned processes in steps S42 to S44 are repeatedly executed for each writing action (loop process). As a result, while the user is writing, the stroke image and image information associated with the writing action, and writing-related information (information related to the stroke image) are continuously transmitted to the information processing device 5.

[0168] Thereafter, the process of step S45 and the loop process of steps S46 to S48 are similar to the process of step S37 and the loop process of steps S38 to S40 described above.

[0169] This allows the user to check the stroke image displayed on the information processing device 5 in real time while performing a writing action using the writing device 3.

[0170] In the writing system according to this embodiment, for example, when the processing of step 43 described above is executed, other devices may exist between the writing device 3 and the information processing device 5. In other words, each piece of information (data) transmitted and received between the writing device 3 and the information processing device 5 may be configured to be transmitted and received once via another device. The above-mentioned configuration can be applied even when other processing steps exist between the writing device 3 and the information processing device 5.

[0171] In the third embodiment, it is also possible to perform processing to prevent overlapping of stroke images as shown in Fig. 29. As a result, even when stroke images are displayed on the information processing device 5 in real time while a writing operation is being performed by the writing device 3, the stroke images do not overlap and can be recognized as meaningful information by the information processing device 5.

[0172] [Major Effects of the Third Embodiment] As described above, according to this embodiment, the writing device 3 transmits writing information associated with a writing action to the information processing device 5 in real time while the writing action is being performed (step S43), and the information processing device 5 repeatedly registers the coordinate information and writing-related information included in the received writing information (step S44). This provides an effect that, in addition to the effect of the first embodiment, the user can check the stroke image displayed on the information processing device 5 in real time while performing the writing action.

[0173] [Supplementary description of the embodiment] Each function of the above-mentioned embodiment can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a device programmed to execute each function by software, such as a processor implemented by an electronic circuit. This device refers to, for example, a processor, an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), a SOC (System on a chip), a GPU (Graphics Processing Unit), and a conventional circuit module designed to execute each function described above.

[0174] In the above-described embodiment, the writing system 1 may be constructed to perform various judgments in the above-described various processes by machine learning (learning using artificial intelligence: AI). This eliminates the need to use corresponding data tables.

[0175] Here, machine learning is a technology for making a computer acquire a human-like learning ability, and refers to a technology in which a computer autonomously generates an algorithm required for judgment such as data identification from learning data that is previously loaded, and applies this to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning. The learning method for machine learning may further be a combination of these learning methods, and any learning method for machine learning is acceptable.

[0176] So far, we have described a writing device, writing system, writing processing method, and program according to one embodiment of the present invention, but the present invention is not limited to the above-described embodiment, and can be modified within the scope of what a person skilled in the art can imagine, such as adding, changing, or deleting other embodiments, and as long as the functions and effects of the present invention are achieved in any aspect, it is included in the scope of the present invention.

[0177] In addition, as another embodiment (use scene) of the present invention, for example, various tasks performed by nurses in medical sites or caregivers in care sites can be considered. In these sites, when taking notes on vital data and various conditions of patients and care recipients, the nurses or caregivers can use the writing device described in this embodiment without choosing a recording medium such as paper or a tablet. Specifically, the nurses or caregivers can repeatedly write down desired text or images in a physically limited space (area) by performing a writing action with the writing device on their own palms or arms, wristband-type notepads, etc. This can be expected to improve the efficiency of the work of nurses, caregivers, etc., and to speed up responses according to the health conditions of patients and care recipients.

[0178] ■ Summary ■ For example, aspects of the present invention are as follows.

[0179] <First aspect> The writing device 3 (an example of a writing device; omitted below) as a first embodiment is capable of generating a trajectory of the writing device 3 accompanying a writing action as a stroke image, and has a memory unit 3000 (an example of a memory means; omitted below) that stores information related to the stroke image, a reception detection unit 32 (an example of a detection means; omitted below) that detects the start position on the writing surface of the writing device 3 accompanying the writing action, a calculation conversion unit 34 (an example of a calculation means; omitted below) that calculates coordinate information on the writing surface of the writing device 3 accompanying the writing action based on the detected start position, and a memory reading unit 39 (an example of a memory control means; omitted below) that stores the calculated coordinate information in the memory unit 3000 (an example of a memory means; omitted below) as information related to the stroke image when the information processing device 5 that communicates with the writing device 3 is not in a state where it can communicate with the writing device 3.

[0180] According to the first aspect, when a writing action is performed using a writing device, information relating to a stroke image generated in association with the writing action can be stored in the writing device itself.

[0181] <Second aspect> In the first aspect, the information relating to the stroke image in the second aspect includes a plurality of pieces of coordinate information and a data ID (an example of identification information; omitted below) for identifying each piece of coordinate information.

[0182] According to the second aspect, it is possible to obtain the same effect as the first aspect.

[0183] <Third aspect> The reception detection unit 32 of the writing device 3 in the third aspect detects the amount of movement of the writing device 3 on the writing surface associated with a writing action when the information processing device 5 is in a state capable of communicating with the writing device 3 in the first or second aspect.

[0184] According to the third aspect, it is possible to obtain the same effect as the first aspect.

[0185] <Fourth aspect> The reception detection unit 32 of the writing device 3 in the fourth aspect, in any of the first to third aspects, detects the starting position on the writing surface of the writing device 3 using an optical sensor 317 provided on the writing device 3, and detects the amount of movement on the writing surface of the writing device 3 using a pressure sensor 316 provided on the writing device 3.

[0186] According to the fourth aspect, in addition to the effect of the first aspect, it becomes possible to store information relating to a stroke image generated in association with a writing action in the writing device itself only while the writing action is being performed.

[0187] <Fifth aspect> The calculation conversion unit 34 of the writing device 3 in the fifth aspect calculates coordinate information based on the detected movement amount of the writing device 3 on the writing surface in the first to fourth aspects.

[0188] According to the fifth aspect, it is possible to obtain the same effects as those of the second and third aspects.

[0189] <Sixth aspect> The calculation and conversion unit 34 of the writing device 3 as a sixth aspect, in any of the first to fifth aspects, repeatedly calculates movement amount information when a part of the writing device 3 is in contact with the writing surface and calculates movement amount information when a part of the writing device 3 is away from the writing surface based on contact presence / absence information on the writing surface of the writing device 3 and movement amount information on the writing surface of the writing device 3.

[0190] According to the sixth aspect, it becomes possible to store an accurate stroke image linked to a writing action in the writing device.

[0191] <Seventh aspect> The writing device 3 as a seventh aspect, in any of the first to sixth aspects, further has a transmission / reception unit 31 (an example of a first transmission means; omitted below) that transmits at least one of movement amount information indicating the amount of movement and coordinate information to the information processing device 5 when the information processing device 5 is in a state capable of communicating with the writing device 3.

[0192] According to the seventh aspect, it becomes possible to share a stroke image obtained in association with a writing operation with other devices.

[0193] <Eighth aspect> In the seventh mode, the transmitting / receiving unit 31 of the writing device 3 in the eighth mode transmits to the information processing device 5 at least one of the movement amount information and the coordinate information, and writing-related information accompanying the writing action.

[0194] According to the eighth aspect, in addition to the effect of the sixth aspect, it becomes possible for the information processing device to process the stroke image obtained in association with the writing action.

[0195] <Ninth aspect> The writing-related information in the writing device 3 as the 9th aspect, in any of the 1st to 8th aspects, represents any of the following information: date and time information or time information when the writing action was performed, location information indicating the location where the writing device 3 is located, or person information related to a person present in the same space as the user using the writing device 3.

[0196] According to the ninth aspect, by associating various pieces of information with a writing action performed by a user, it becomes possible to share a stroke image obtained with the writing action performed by the user.

[0197] <Tenth aspect> The writing device 3 as a 10th aspect includes a pen shape, a ring shape, and a fingerstall shape in any of the 1st to 9th aspects, and writing operations are performed while the device is held in a hand or a finger, or while the device is worn on a hand or a finger.

[0198] According to the tenth aspect, in addition to the effect of the first aspect, it is possible to widen the range of choices of writing devices according to the user's situation.

[0199] <Eleventh aspect> A writing system 1 (an example of a writing system; omitted below) as an eleventh aspect has a writing device 3 (an example of a writing device; omitted below) capable of generating a trajectory of the writing action of the writing device as a stroke image, and an information processing device 5 (an example of an information processing device; omitted below) capable of communicating with the writing device 3, The writing device 3 has a memory unit 3000 (an example of a memory means; omitted below) that stores information related to a stroke image, a reception detection unit 32 (an example of a detection means; omitted below) that detects a start position on the writing surface of the writing device 3 associated with a writing action when the information processing device 5 is not in a state capable of communicating with the writing device 3, a calculation conversion unit 34 (an example of a calculation means; omitted below) that calculates coordinate information on the writing surface of the writing device 3 associated with the writing action based on the detected start position, a memory readout unit 39 (an example of a memory control means; omitted below) that stores the calculated coordinate information in the memory unit 3000 (an example of a memory means; omitted below) as information related to the stroke image, and a transmission / reception unit 31 (an example of a first transmission means; omitted below) that transmits the coordinate information and writing-related information associated with the writing action to the information processing device 5, The information processing device 5 has a transmission / reception unit 51 (an example of a receiving means; omitted below) that receives the coordinate information and writing-related information transmitted by the writing device 3, and a display control unit 54 (an example of a display control means; omitted below) that displays a stroke image represented by information related to the stroke image on a display 507 (an example of a display means; omitted below) based on the received coordinate information and writing-related information.

[0200] According to the eleventh aspect, as in the first aspect, when a writing action is performed using a writing device, information relating to a stroke image generated in association with the writing action can be stored in the writing device itself.

[0201] <Twelfth aspect> The writing system 1 in the 12th aspect, in the 11th aspect, further has a processing unit 58 (an example of a processing means; omitted below) that divides and processes coordinate information representing the stroke image transmitted by the writing device into predetermined chunks based on writing-related information.

[0202] According to the twelfth aspect, in addition to the effect of the tenth aspect, by dividing the stroke image data for each piece of related information regarding the writing action performed on the writing device, it becomes possible to display the information separately from other information.

[0203] <Thirteenth aspect> In the writing system 1 of the 13th aspect, in the 11th or 12th aspect, the information processing device 5 further has a transmission / reception unit 51 (an example of a second transmission means; omitted below) that transmits setting information related to the writing operation in the writing device 3 to the writing device 3, and the writing device 3 further has a transmission / reception unit 31 (an example of a receiving means; omitted below) that receives the setting information transmitted by the information processing device 5, and a processing unit 38 (an example of a second processing means; omitted below) that executes processing on a switch 318 (an example of an operation means; omitted below) provided on the writing device 3, and the processing unit 38 reflects the received setting information in the writing device 3 depending on at least one of the length of operation time and the operation pattern of the switch 318.

[0204] According to the thirteenth aspect, various settings relating to the writing operation of the writing device can be set to desired contents via the information processing device.

[0205] <14th aspect> The writing processing method performed by a writing device 3 (an example of a writing device; omitted below) capable of generating a trajectory of its own device accompanying a writing action as a stroke image as a 14th aspect is a writing device having a memory unit 3000 (an example of a memory means; omitted below) that stores information related to the stroke image, and executes processing including a detection step of detecting a start position on the writing surface of the writing device 3 accompanying the writing action, a calculation step of calculating coordinate information on the writing surface of the writing device 3 accompanying the writing action based on the detected start position, and a storage control step of storing the calculated coordinate information in the storage unit 3000 as information related to the stroke image when an information processing device 5 that communicates with the writing device 3 is not in a state capable of communicating with the writing device 3.

[0206] According to the fourteenth aspect, as in the first aspect, when a writing action is performed using a writing device, information relating to a stroke image generated in association with the writing action can be stored in the writing device itself.

[0207] <Fifteenth aspect> The program as a fifteenth aspect is a program for generating a stroke image based on a trajectory of a writing action of the writing device itself, the program including: The process includes a detection step of detecting a start position on the writing surface of the writing device 3 associated with a writing action, a calculation step of calculating coordinate information on the writing surface of the writing device 3 associated with the writing action based on the detected start position, and a storage control step of storing the calculated coordinate information in the memory unit 3000 as information related to a stroke image when the information processing device 5 that communicates with the writing device 3 is not in a state in which it can communicate with the writing device 3.

[0208] According to the fifteenth aspect, as in the first aspect, when a writing action is performed using a writing device, information relating to a stroke image generated in association with the writing action can be stored in the writing device itself. [Explanation of symbols]

[0209] 1 Writing system 3 Writing devices 5. Information processing equipment 31 Transmitting / receiving unit (an example of a first transmitting means, an example of a receiving means) 32 Reception detection unit (an example of reception means, an example of detection means) 33 Acquisition unit (an example of an acquisition means) 34 Calculation and conversion unit (an example of a calculation means, an example of a conversion means) 35 Judgment unit (an example of a judgment means) 37 Generation unit (an example of a generation means) 38 Processing unit (an example of a second processing means) 39 Memory readout unit (an example of a memory control means) 3000 Memory unit (an example of a memory means) 51 Transmitting / receiving unit (an example of a second transmitting means, an example of a receiving means) 54 Display control unit (an example of a display control means) 56 Registration unit (an example of a registration means) 58 Processing unit (an example of a first processing means) 507 Display (an example of a display means) 5001 Coordinate information management DB (an example of coordinate information management means, an example of metadata management means) [Prior art documents] [Patent documents]

[0210] [Patent Document 1] Japanese Patent Application Publication No. 9-114586 [Patent Document 2] JP 2018-106594 A

Claims

1. A writing device capable of generating a stroke image representing a trajectory of the device itself accompanying a writing action, a storage means for storing information relating to the stroke image; a detection means for detecting a start position of the writing device on the writing surface associated with the writing action; a calculation means for calculating coordinate information of the writing device on the writing surface associated with the writing action based on the detected start position; a storage control means for causing the storage means to store the calculated coordinate information as information relating to the stroke image when an information processing device that communicates with the writing device is not in a state in which communication with the writing device is possible; having A writing device characterized by:

2. The writing device according to claim 1 , wherein the information relating to the stroke image includes a plurality of pieces of the coordinate information and identification information for identifying each of the pieces of the coordinate information.

3. The detection means further comprises: When the information processing device is in a state capable of communicating with the writing device, the information processing device detects an amount of movement of the writing device on a writing surface associated with the writing action.

3. A writing device according to claim 1 or 2.

4. The detection means includes: The start position is detected by an optical sensor provided in the writing device, and the amount of movement is detected by a pressure sensor provided in the writing device.

3. A writing device according to claim 1 or 2.

5. The calculation means is Calculating the coordinate information based on the detected amount of movement.

3. A writing device according to claim 1 or 2.

6. The calculation means is based on contact information of the writing device on the writing surface and movement amount information of the writing device on the writing surface, repeatedly calculating the movement amount information when a part of the writing device is in contact with the writing surface and calculating the movement amount information when the part of the writing device is separated from the writing surface; 6. A writing device according to any one of claims 1 to 5.

7. 3. A writing device according to claim 1 or 2, further comprising: a first transmission means for transmitting at least one of movement amount information indicating the movement amount and the coordinate information to the information processing device when the information processing device is in a state capable of communicating with the writing device; having A writing device characterized by:

8. The first transmitting means includes: Transmitting at least one of the movement amount information and the coordinate information, and handwriting-related information associated with the writing action, to the information processing device. A writing device according to claim 7 .

9. The writing-related information represents any one of date and time information or time information when the writing action was performed, location information representing the location where the writing device is present, and person information relating to a person present in the same space as the user using the writing device.

3. A writing device according to claim 1 or 2.

10. The writing device may have a pen shape, a ring shape, or a finger cot shape, and the writing action is performed while the writing device is held by a hand or a finger, or while the writing device is attached to the hand or the finger.

3. A writing device according to claim 1 or 2.

11. A writing system having a writing device capable of generating a stroke image representing a trajectory of the writing device in association with a writing action, and an information processing device capable of communicating with the writing device, The writing device comprises: a storage means for storing information relating to the stroke image; a detection means for detecting a start position of the writing device on the writing surface associated with the writing action; a calculation means for calculating coordinate information of the writing device on the writing surface associated with the writing action based on the detected start position; a storage control means for causing the calculated coordinate information to be stored in the storage means as information relating to the stroke image when the information processing device is not in a state capable of communicating with the writing device; a first transmitting means for transmitting the coordinate information and handwriting-related information accompanying the writing action to the information processing device; having The information processing device includes: a receiving means for receiving the coordinate information and the handwriting-related information transmitted by the handwriting device; a display control means for controlling a display means to display a stroke image represented by the information relating to the stroke image based on the received coordinate information and the handwriting-related information; having A writing system characterized by:

12. 12. The writing system according to claim 11, wherein the information processing device further comprises: a first processing means for dividing and processing the coordinate information representing the stroke image transmitted by the writing device into a predetermined group based on the writing-related information; having A writing system characterized by:

13. The information processing device further comprises: a second transmission means for transmitting setting information relating to the writing action in the writing device to the writing device; having The writing device further comprises: a receiving means for receiving the setting information transmitted by the information processing device; A second processing means for executing a process for an operation means provided in the writing device; having The second processing means is reflecting the received setting information on the writing device in accordance with at least one of a length of operation time and an operation pattern of the operation means; 13. Writing system according to claim 11 or 12.

14. A writing processing method executed by a writing device capable of generating a stroke image representing a trajectory of the writing device itself accompanying a writing action, comprising: The writing device comprises: A storage means for storing information relating to the stroke image, a detection step of detecting a start position of the writing device on the writing surface associated with the writing action; a calculation step of calculating coordinate information on the writing surface of the writing device associated with the writing action based on the start position detected by the detection step; a storage control step of storing the calculated coordinate information in the storage means as information related to the stroke image when an information processing device that communicates with the writing device is not in a state in which communication with the writing device is possible; Perform a process that includes A writing processing method comprising:

15. A program to be executed by a writing device capable of generating a stroke image representing a trajectory of the writing device itself accompanying a writing action, the writing device having a storage means for storing information relating to the stroke image, the program comprising: a detection step of detecting a start position of the writing device on the writing surface associated with the writing action; a calculation step of calculating coordinate information on the writing surface of the writing device associated with the writing action based on the start position detected by the detection step; a storage control step of storing the calculated coordinate information in the storage means as information related to the stroke image when an information processing device that communicates with the writing device is not in a state in which communication with the writing device is possible; A program that executes a process including:

Images