Ink data generating device, method and program product
By comparing the inclusion relationship of stroke elements, different forms of metadata are generated, which solves the problem of low metadata generation and processing efficiency in the prior art, and realizes more efficient ink data processing and attribute input.
Patent Information
- Application Number
- CN201980061745.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-11-02
- Filing Date
- 2019-10-07
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2039-10-07
AI Technical Summary
In the prior art, when generating and processing ink data, the generation efficiency and processing of metadata are insufficient, making it difficult to effectively utilize the inclusion relationship between strokes.
By obtaining data of the stroke set that has not been assigned meta information and the stroke set that has been assigned meta information, comparing stroke elements to determine the inclusion relationship, and generating metadata of different forms based on the judgment results, improving the efficiency of metadata generation and processing.
It improves the processing ability of metadata during and after generation, and can automatically select appropriate attributes, reduce user input burden, and improves the accuracy of semantic attributes and metadata processing.
Smart Images

Figure CN112740201B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an ink data generating device, method and program for generating ink data. Background Art
[0002] Patent Document 1 discloses a method of creating a metadata block by associating N types of metadata that specifies an input device with M stroke data, and writing the block together with the stroke data into an ink file.
[0003] Prior art literature
[0004] Patent Literature
[0005] Patent Document 1: International Patent Application Publication No. 2016 / 157400 Summary of the invention
[0006] Problems to be solved by the invention
[0007] An object of the present invention is to provide an ink data generating device, method, and program for improving the processability of ink data during or after the generation of metadata.
[0008] Means for solving problems
[0009] The ink data generating device in the first present invention is a device for generating ink data including metadata, wherein the metadata describes meta-information of each set of strokes, wherein the ink data generating device comprises: a first acquisition unit, which acquires first set data representing stroke elements belonging to a first set to which meta-information has not yet been assigned; a second acquisition unit, which acquires second set data representing stroke elements belonging to a second set to which meta-information has been assigned for each set of strokes; a set determination unit, which uses the first set data acquired by the first acquisition unit and the second set data acquired by the second acquisition unit to determine the inclusion relationship between the first set and the second set by comparing the stroke elements of both sides; and a data generating unit, which generates first metadata of the first set by describing in a different form according to the determination result of the set determination unit.
[0010] The ink data generating device in the second present invention is a device that generates ink data including metadata, wherein the metadata records the meta-information of each set of strokes, wherein the ink data generating device comprises: a data acquisition unit that acquires set data of stroke elements representing a set of strokes for each set; and a data output unit that outputs metadata representing the inclusion relationship between multiple sets based on the set data of each set acquired by the data acquisition unit.
[0011] The ink data generating method in the third present invention is a method for generating ink data including metadata, wherein the metadata describes the meta-information of each set of strokes, wherein one or more computers execute: a first acquisition step of acquiring first set data representing stroke elements belonging to a first set to which meta-information has not yet been assigned; a second acquisition step of acquiring second set data representing stroke elements belonging to a second set to which meta-information has been assigned for each set of strokes; a determination step of using the first set data and the second set data respectively acquired to determine the inclusion relationship between the first set and the second set by comparing the stroke elements of both sides; and a generation step of generating first metadata of the first set by describing in a different form according to the determination result in the determination step.
[0012] The ink data generation program in the fourth present invention is a program for generating ink data including metadata, wherein the metadata describes the meta-information of each set of strokes, wherein one or more computers are caused to execute: a first acquisition step of acquiring first set data representing stroke elements belonging to a first set to which meta-information has not yet been assigned; a second acquisition step of acquiring second set data representing stroke elements belonging to a second set to which meta-information has been assigned for each set of strokes; a determination step of using the first set data and the second set data respectively acquired to determine the inclusion relationship between the first set and the second set by comparing the stroke elements of both sides; and a generation step of generating first metadata of the first set by describing in a different form according to the determination result in the determination step.
[0013] Effects of the Invention
[0014] According to the present invention, it is possible to improve the handleability of ink data during or after the generation of metadata. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 This is an overall structural diagram of an ink data generating device in one embodiment of the present invention.
[0016] Figure 2 It is shown Figure 1 A diagram showing an example of ink data.
[0017] Figure 3 It is shown based on Figure 2 A diagram of the content visualized by the ink data.
[0018] Figure 4 is used for Figure 1 The first flowchart illustrating the operation of the ink data generating device shown.
[0019] Figure 52 is a diagram showing a method of specifying a new set based on a lasso operation.
[0020] Fig. 6A It is a diagram showing a method of selecting a sub-group.
[0021] Figure 6B It is a diagram showing a method of extracting feature amounts in a sub-group.
[0022] Fig. 7A It is a diagram showing a configuration example of an identifier.
[0023] Figure 7B It is a diagram showing a method of deriving a histogram.
[0024] Figure 8 It is shown Figure 4 FIG. 4 is a diagram showing an example of a screen display in step S5.
[0025] Fig. 9 It is a diagram showing the positional relationship between the content and two lassoes.
[0026] Fig.10 is shown with Fig. 9 A diagram showing an example of description of corresponding ink data.
[0027] Fig.11 is used for Figure 1 A second flow chart illustrating the operation of the ink data generating device shown.
[0028] Fig.12 It is a diagram showing the positional relationship between the content and the three lassoes.
[0029] Fig.13 is shown with Fig.12 A diagram showing an example of description of corresponding ink data.
[0030] Fig.14 It is a diagram showing the positional relationship between the content and the three lassoes.
[0031] Fig.15 is shown with Fig.14 A diagram showing a first description example of corresponding ink data.
[0032] Fig.16 is shown with Fig.14 A diagram of a second description example of corresponding ink data.
[0033] Fig.17 is shown with Fig.14 A diagram showing a third description example of corresponding ink data.
[0034] Fig.18 It is a diagram showing the positional relationship between the content and the three lassoes.
[0035] Fig.19 is shown with Fig.18 A diagram showing an example of description of corresponding ink data.
[0036] Fig. 20 This is a diagram showing the result of assigning semantic attributes. DETAILED DESCRIPTION
[0037] [Structure of ink data generating device 10]
[0038] <Overall Structure>
[0039] Figure 1 1 is an overall structural diagram of an ink data generating device 10 in one embodiment of the present invention. The ink data generating device 10 is an electronic device having a touch panel display 12, such as a tablet terminal, a smart phone, or a personal computer. Specifically, the ink data generating device 10 is configured to include a touch panel display 12, a display driver IC 14, a touch sensor IC 16, a host processor 18, and a memory 20.
[0040] The touch panel display 12 is configured to include a display panel 22 capable of visually displaying content and a touch sensor 24 disposed on the display panel 22. The display panel 22 can display black and white images or color images, and may be, for example, a liquid crystal panel or an organic EL (Electro-Luminescence) panel. The touch sensor 24 includes a plurality of X-line electrodes for detecting the position of the X-axis of the sensor coordinate system and a plurality of Y-line electrodes for detecting the position of the Y-axis.
[0041] The display driver IC 14 is an integrated circuit that performs drive control of the display panel 22. The display driver IC 14 drives the display panel 22 based on a display signal supplied from the host processor 18. Thus, the content represented by the ink data 50 is displayed on the display panel 22.
[0042] The touch sensor IC 16 is an integrated circuit that performs drive control of the touch sensor 24. The touch sensor IC 16 drives the touch sensor 24 based on a control signal supplied from the host processor 18. Thus, the touch sensor IC 16 performs a "pen detection function" for detecting the state of the stylus pen 26 and a "touch detection function" for detecting a touch of a user's finger or the like.
[0043] The pen detection function includes, for example, a scanning function of the touch sensor 24, a receiving and analyzing function of a downlink signal, a function of estimating the state (e.g., position, posture, pen pressure) of the stylus pen 26, and a function of generating and transmitting an uplink signal including a command to the stylus pen 26. In addition, the touch detection function includes, for example, a two-dimensional scanning function of the touch sensor 24, a function of acquiring a detection map on the touch sensor 24, and a function of classifying regions on the detection map (e.g., classification of fingers, palms, etc.).
[0044] In this way, the input function of the touch sensor 24 and the stylus pen 26 and the output function of the display panel 22 are combined to construct a user interface (hereinafter referred to as UI unit 28 ).
[0045] The host processor 18 is composed of a processing and calculation device including a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit). The host processor 18 reads and executes a program stored in the memory 20 to function as a first acquisition unit 30, a second acquisition unit 32, an attribute estimation unit 34, a collection determination unit 36, a data generation unit 38 (data output unit), and a rendering processing unit 40.
[0046] The memory 20 is composed of a non-temporary computer-readable storage medium. Here, the computer-readable storage medium is a storage device including a hard disk drive (HDD), a solid state drive (SSD) or a portable medium such as an optical disk, a ROM, a CD-ROM, and a flash memory. In the example of this figure, the memory 20 stores new set data 42 (first set data), existing set data 44 (second set data), a learning parameter group 46, and ink data 50.
[0047] <An Example of Ink Data 50>
[0048] Figure 2 It is shown Figure 1FIG. 1 is a diagram showing an example of ink data 50. The following is an example of InkML (Ink Markup Language) that describes digital data in XML (eXtensible Markup Language) format. It should be noted that the data format of the ink data 50, the so-called "ink description language", is not limited to InkML, and may be, for example, WILL (Wacom Ink Layer Language) or ISF (Ink Serialized Format). In addition, by describing the ink data 50 in the data structure format of JSON (JavaScript (registered trademark) Object Notation), data exchange between various software and programming languages becomes easy.
[0049] In the example of this figure, the ink data 50 is composed of stroke data 51 that describes at least one (24 in the example of this figure) stroke. As can be understood from this figure, one stroke is composed of <trace>The tag contains multiple point data arranged in sequence. Each point data is composed of at least an indicated position (X coordinate, Y coordinate), separated by separators such as commas. For the convenience of illustration, only the point data representing the starting point and end point of the stroke are recorded, and the point data representing multiple passing points are omitted. It should be noted that in addition to the indicated position mentioned above, the point data may also include the order of writing, the pen pressure of the stylus 26, the posture, etc.
[0050] Figure 3 It is shown based on Figure 2 The ink data 50 is visualized in the figure. Figure 1 ) defines a two-dimensional coordinate system (hereinafter referred to as sensor coordinate system: XY) for determining the detection position of the touch sensor 24. The content is composed of a string consisting of "This is a pencil." and a pencil image. It should be noted that the numbers in brackets represent the writing order of the strokes (stroke ID = 01-24) and do not constitute part of the content. Please note this.
[0051] [First Operation of Ink Data Generating Device 10]
[0052] The ink data generating device 10 in this embodiment is configured as described above. First, regarding the first operation of the ink data generating device 10, while referring to Figure 4 Flowchart and Figure 5-Figure 10 The "first action" means an action of supporting the provision of user's meta-information (for example, data attributes, creator, creation date and time, used device, etc.).
[0053] Before the first operation, the drawing processing unit 40 analyzes the ink data 50 read from the memory 20, performs a desired rasterization process on the stroke data 51, and generates a display signal representing the content. The display driver IC 14 drives the display panel 22 based on the display signal supplied from the host processor 18. As a result, the visualized content is displayed on the display panel 22.
[0054] <Description of the First Action>
[0055] exist Figure 4 In step S1, the host processor 18 confirms whether a designation operation of a new set of strokes (first set) to which meta-information is to be assigned has been received. The "designation operation" is an operation for selecting one or more strokes from a plurality of strokes, such as a "lasso operation" for enclosing a selected object with a lasso.
[0056] Figure 5 2 is a diagram showing a method of specifying a new set by lasso operation. The user moves the stylus pen 26 on the touch surface of the display panel 22 while performing a writing operation to surround the entire stroke to which meta-information is to be given. Through this handwriting input, the lasso Ltmp is displayed superimposed on the visualized content.
[0057] If the user's designation operation has not been accepted (step S1: No), the process stays at step S1 until the designation operation is accepted. On the other hand, if the user's instruction operation has been accepted (step S1: Yes), the process proceeds to the next step S2.
[0058] In step S2, the first acquisition unit 30 acquires data representing stroke elements belonging to the new set (hereinafter referred to as new set data 42) based on the content of the instruction operation received in step S1. The new set data 42 is data that can identify the stroke elements, for example, it can be identification information of the stroke, that is, the stroke ID. Figure 2 and Figure 5 In the example of , the first acquisition unit 30 acquires stroke ID=01-18 as the new set data 42.
[0059] In step S3, the host processor 18 checks whether the mode for supporting the input of the meta-information (hereinafter referred to as "input support mode") is turned on or off. If the input support mode is "off" (step S3: OFF), the execution of step S4 is omitted and the process proceeds to step S5. On the other hand, if the input support mode is "on" (step S3: ON), the process proceeds to the next step S4.
[0060] In step S4, the attribute estimation unit 34 estimates the attribute to be assigned to the new set based on the relationship between the positions or shapes of the multiple strokes constituting the new set specified in step S1. Here, the attribute estimation unit 34 uses the identifier 70 constructed by machine learning (see Fig. 7A ) to infer the semantic attributes of the new set. Figures 6A-7B Explain in detail.
[0061] Fig. 6A and Figure 6B 70 is a diagram showing an example of a method for generating input data for the identifier 70. More specifically, Fig. 6A is a diagram showing a method of selecting a sub-group 74, Figure 6B 2 is a diagram showing a method of extracting feature amounts in the sub-group 74 .
[0062] like Fig. 6A As shown, the attribute estimation unit 34 selects a subgroup 74 consisting of a predetermined number of strokes from a group 72 consisting of 18 stroke elements (stroke ID = 01-18). Here, it is assumed that five strokes having five consecutive IDs = 01-05 starting from ID = 01 are extracted.
[0063] like Figure 6B As shown, the attribute estimation unit 34 sets a square area that includes all five strokes belonging to the sub-group 74, and defines a two-dimensional coordinate system (hereinafter, the normalized coordinate system; X'-Y') corresponding to the area. The origin O' of the normalized coordinate system is equivalent to the vertex of the square area that is closest to the origin O of the sensor coordinate system. The X' axis of the normalized coordinate system is parallel to the X axis of the sensor coordinate system, and the Y' axis of the normalized coordinate system is parallel to the Y axis of the sensor coordinate system. In addition, the scales of the X' axis and the Y' axis are normalized in such a way that the coordinates of the four vertices forming the square area are (0, 0), (1, 0), (0, 1), and (1, 1).
[0064] The attribute estimation unit 34 refers to the stroke data 51 ( Figure 2 ), and obtains the coordinate values (X, Y) of the starting point Ps and the coordinate values (X, Y) of the end point Pe of the stroke in the sensor coordinate system. In addition, the attribute estimation unit 34 derives the coordinate values (X', Y') of the starting point Ps and the coordinate values (X', Y') of the end point Pe of the stroke in the normalized coordinate system by performing a linear transformation of the coordinate system.
[0065] Fig. 7A and Figure 7B is a diagram showing an example of a method for estimating semantic attributes. Fig. 7A is a diagram showing a configuration example of the identifier 70, Figure 7B It is a diagram showing a method of deriving a histogram.
[0066] like Fig. 7A As shown, the recognizer 70 is composed of, for example, a hierarchical neural network, which is composed of an input layer 76, an intermediate layer 78, and an output layer 80. The operation rule of the recognizer 70 is determined by the value of the learning parameter group 46, which is a collection of learning parameters. The learning parameter group 46 may include, for example, coefficients describing the activation function of the unit equivalent to the neuron, weighting coefficients equivalent to the strength of synaptic binding, the number of units constituting each layer, and the number of intermediate layers 78. The learning parameter group 46 is stored in the memory 20 ( Figure 1 ) and is read out in a timely manner as needed.
[0067] The input layer 76 is a layer for inputting feature quantities related to the endpoints of the strokes, and is composed of 20 units in the example of this figure. The feature quantity is an input vector composed of 20 components, [1] the X' coordinate of the starting point Ps, [2] the Y' coordinate of the starting point Ps, [3] the X' coordinate of the end point Pe, and [4] the Y' coordinate of the end point Pe, arranged in the order of the stroke ID.
[0068] The output layer 80 is a layer that outputs a collection of label values representing semantic attributes (hereinafter referred to as a label group), and is composed of 6 units in the example of this figure. The label group is an output vector composed of 6 components representing the likelihood of [1] text (English), [2] text (Japanese), [3] image (graph), [4] image (illustration), [5] mathematical formula, and [6] chemical formula.
[0069] The identifier 70 inputs the feature quantity generated from the subgroup 74 from the input layer 76, and outputs the label group corresponding to the subgroup 74 to the output layer 80 via the intermediate layer 78. For example, when each label is defined by the range of [0, 1], the value of the selected label becomes the largest semantic attribute, such as "text (English)".
[0070] like Figure 7B As shown in FIG. 1 , after the attribute estimation unit 34 recognizes the semantic attributes of the initial sub-group 74, it shifts the stroke ID by one at a time, selects the next sub-group 74 consisting of 5 strokes (ID=02-06), and performs the above-mentioned recognition process. Then, the attribute estimation unit 34 sums up the recognition results of multiple sub-groups 74 and creates a histogram with the number of semantic attributes as the "frequency". In the example of this figure, it is estimated that "text (English)" with the largest frequency in the histogram should be added to the new set ( Fig. 6A The semantic attributes assigned by group 72).
[0071] exist Figure 4 In step S5, the host processor 18 displays an input window 82 for inputting meta-information (for example, comments) on the display panel 22. The input window 82 is displayed, for example, at a position that does not overlap with the closed area surrounded by the lasso Ltmp.
[0072] like Figure 8 As shown, two user controls 84 and 86 and buttons 88 labeled [OK] and [Cancel] are arranged in the input window 82. The first user control 84 is, for example, a pull-down menu configured to allow input of semantic attributes. The second user control 86 is, for example, a text box configured to allow input of value attributes.
[0073] When input support mode is "on", that is, after Figure 4 In the case of "step S3: ON", the semantic attribute estimated in step S4 is displayed as an initial value (that is, a recommended value) in the input field of the user control 84. On the other hand, in the case of "input support mode is OFF", that is, after Figure 4 In the case of "Step S3: OFF", a display indicating that no attribute is selected (for example, a blank display) is performed in the input field of the user control 84.
[0074] When a semantic attribute that matches the user's intention is presented, the user can omit the operation of the user control 84. On the other hand, when a semantic attribute that does not match the user's intention is presented, the user can operate the user control 84 to select a desired attribute.
[0075] In step S6, the host processor 18 confirms whether the confirmation operation of the data input (more specifically, the [OK] button 88 ( Figure 8 ) touch operation). If the confirmation operation has not been accepted (step S6: No), the process stays at step S6 until the confirmation operation is accepted. On the other hand, if the confirmation operation is accepted (step S6: Yes), the process proceeds to step S6. Fig.11 Step S7 described later.
[0076] Fig. 9 : is a diagram showing the positional relationship between the content and the two lassoes Lprv and Ltmp. Assume that the user performs the first annotation operation on the stroke set surrounded by the lasso Lprv, and then performs the second annotation operation on the stroke set surrounded by the lasso Ltmp. Through the first annotation operation, the set with ID=11-17, 19-23 as stroke elements is given meta-information that the semantic attribute (Sem) is "none" and the value attribute (Value) is "pencil". Through the second annotation operation, the set with ID=01-18 as stroke elements is given meta-information that the semantic attribute is "text (English)" and the value attribute is "none".
[0077] Fig.10 is shown with Fig. 9 FIG. 5 is a diagram showing an example of a description of corresponding ink data 50. The ink data 50 is composed of stroke data 51 and metadata 52. The metadata 52 is composed of two metadata chunks 53 and 54. The first metadata chunk 53 describes meta information assigned to a stroke set represented by the lasso Lprv. The second metadata chunk 54 describes meta information assigned to a stroke set represented by the lasso Ltmp.
[0078] <Effect of the First Action>
[0079] As described above, the ink data generating device 10 is a device for generating ink data 50 including metadata 52, wherein the metadata 52 records the meta-information of each set of strokes, wherein the ink data generating device 10 comprises: a first acquisition unit 30, which acquires new set data 42 representing stroke elements belonging to a new set to which meta-information has not yet been assigned; an attribute inference unit 34, which infers the attributes that should be assigned to the new set based on the mutual relationship between the positions or shapes of the multiple strokes constituting the new set; and a data generating unit 38, which generates metadata blocks 53, 54 (first metadata) for assigning the inferred attributes to the new set.
[0080] Alternatively, according to the ink data generating method and program, one or more processors (for example, the host processor 18) include: a first acquisition step ( Figure 4 S2), obtaining new set data 42 representing stroke elements belonging to a new set to which meta information has not yet been assigned; inferring step ( Figure 4 S4), inferring the attributes to be assigned to the new set based on the relationship between the positions or shapes of the multiple strokes constituting the new set; and generating step (described later Fig.11 S10, S11), generate metadata chunks 53, 54 (first metadata) for assigning the inferred attributes to the new set.
[0081] With this configuration, it is possible to automatically select an attribute with high validity in consideration of the mutual relationship between the positions and shapes of a plurality of strokes, thereby reducing the attribute input work of the user.
[0082] The attribute estimation unit 34 may include a recognizer 70 that inputs a feature value related to the end points of a stroke and outputs a label group of semantic attributes. By focusing on the correlation between the end points of a stroke and the semantic attributes, the estimation accuracy of the semantic attributes is improved.
[0083] Alternatively, the ink data generating device 10 may further include a UI unit 28 configured to simultaneously display a plurality of strokes in a display area and to select stroke elements belonging to a new set from the plurality of strokes through handwriting input by a user, the UI unit 28 further displays a user control 84 for inputting semantic attributes to be assigned to the new set, and the data generating unit 38 generates metadata chunks 53 and 54 representing the semantic attributes input using the user control 84. Thus, smooth input of semantic attributes using a user interface is performed.
[0084] Alternatively, the UI unit 28 may display the semantic attribute estimated by the attribute estimation unit 34 as the initial value of the user control 84. This increases the possibility that a change operation from the initial state of the user control 84 is not required, and the user can be provided with an opportunity to make a change.
[0085] [Second Operation of Ink Data Generating Device 10]
[0086] Next, regarding the second operation of the ink data generating device 10, while referring to Fig.11 Flowchart and Figure 12-Figure 17 The "second operation" means an operation of generating ink data 50 based on the meta-information input by the first operation. Here, the data generating unit 38 performs a "flow-type" data processing of sequentially updating the ink data 50 each time the meta-information is given.
[0087] <Description of the Second Action>
[0088] exist Fig.11 In step S7, the second acquisition unit 32 acquires data (hereinafter referred to as existing set data 44) representing stroke elements belonging to a set of strokes to which meta-information has been assigned (hereinafter referred to as an existing set). The existing set data 44, like the new set data 42, is data capable of identifying stroke elements, and may be, for example, stroke identification information, i.e., a stroke ID.
[0089] In step S8, the set determination unit 36 uses Figure 4 The new set data 42 obtained in step S2 and the existing set data 44 obtained in step S7 are compared to determine the inclusion relationship between the new set and the existing set. Specifically, the set determination unit 36 determines whether the existing set is (1) a superordinate set, (2) a subordinate set, or (3) not in an inclusion relationship with the new set. It should be noted that when there are multiple existing sets, the set determination unit 36 determines the inclusion relationship of each existing set. Please note this.
[0090] In step S9 , the cluster determination unit 36 checks whether there is an existing cluster that includes the new cluster.
[0091] Fig.12 This is a diagram showing the positional relationship between the content and the three lassoes Lpr1, Lpr2, and Ltmp. Here, it is assumed that after the user has performed the 1st to 8th annotation operations, the 9th annotation operation is performed on the stroke set surrounded by the lasso Ltmp. Through the 9th annotation operation, the set of stroke elements with ID = 19-24 is given meta-information that the semantic attribute (Sem) is "picture" and the value attribute (Value) is "none". For the convenience of illustration, the records of the 3rd to 8th lassoes are omitted.
[0092] As can be understood from this figure, the existing set specified by lasso Lpr1 (first annotation operation) includes stroke elements of ID = 19-23, but does not include stroke elements of ID = 24. In addition, the existing set specified by lasso Lpr2 (second annotation operation) does not include stroke elements of ID = 19-24. In this case, it is considered that there is no inclusion relationship between the new set and the existing set (step S9: No), and the process proceeds to step S10.
[0093] In step S10, the data generation unit 38 generates a data item described in Figure 4 The metadata chunk 55 of the meta-information determined in step S6 is input and arranged at a position according to the order of providing the meta-information determined in advance.
[0094] Fig.13 is shown with Fig.12 A diagram showing an example of a description of corresponding ink data 50. The ink data 50 is composed of stroke data 51 and metadata 52A. The metadata 52A is composed of 9 metadata chunks including metadata chunks 53, 54, and 55. The first metadata chunk 53 records the meta-information assigned to the existing set specified by the lasso Lpr1. The second metadata chunk 54 records the meta-information assigned to the existing set specified by the lasso Lpr2. The ninth metadata chunk 55 records the meta-information assigned to the new set specified by the lasso Ltmp. For the convenience of illustration, the description of the 3rd to 8th metadata chunks is omitted.
[0095] In the example of this figure, since a rule is set to sequentially add newly added meta-information to a lower position, the ninth metadata chunk 55 is added to the last position of the metadata 52. In contrast to this figure, when a rule is set to sequentially add newly added meta-information to an upper position, the metadata chunk 55 is added to the beginning position of the metadata 52.
[0096] on the other hand, Fig.14 2 is a diagram showing the positional relationship between the content and the three lassoes Lpr1, Lpr2, and Ltmp. Fig.14 The only difference is the specified range of the lasso Ltmp. Fig.12 Through the ninth annotation operation, meta information whose semantic attribute is "picture" and whose value attribute is "none" is given to the set of stroke elements with ID=19-23.
[0097] As can be understood from this figure, the existing set specified by lasso Lpr1 includes all stroke elements (ID=19-23), so it meets the "superordinate set" of the new set. In other words, it is considered that there is an inclusion relationship between the new set and the existing set (step S9: Yes), and the process proceeds to step S11.
[0098] In step S11, the data generation unit 38 generates metadata chunks 56-58 (first metadata) in a manner that is dependent on the metadata chunk 54 (second metadata) of the existing set that is the upper set or lower set of the new set. Figure 15-17 While explaining.
[0099] Fig.15 is shown with Fig.14 1 is a diagram showing a first description example of the corresponding ink data 50. The ink data 50 is composed of stroke data 51 and metadata 52B. The metadata 52B is composed of nine metadata chunks including metadata chunks 53, 54, and 56. The metadata chunks 53 and 54 are related to Fig.13 The data is the same as in the case of , so the description is omitted.
[0100] The ninth metadata chunk 56 describes the meta information added to the new set specified by the lasso Ltmp. In the example of this figure, the ninth metadata chunk 56 is added to the metadata chunk 53 in a manner that is incorporated therein, although there is a rule that newly added meta information is added sequentially downward.
[0101] In this way, when the data generation unit 38 determines that there is a containment relationship between the new set and the existing set, it can also generate the metadata block 56 of the new set at a position within the metadata block 53 of the corresponding existing set. Through this description example, when the ink data 50 is generated, the metadata blocks 53 and 56 having a containment relationship can also be associated with each other.
[0102] Fig.16 is shown with Fig.14 52C. The metadata 52C is composed of nine metadata chunks including metadata chunks 53, 54, and 57. The metadata chunks 53 and 54 are related to Fig.13 The data is the same as in the case of , so the description is omitted.
[0103] The ninth metadata chunk 57 describes the meta-information added to the new set specified by the lasso Ltmp. In the example of this figure, although there is a rule that the newly added meta-information is added to the lower position one by one, the ninth metadata chunk 56 is added to the position just below the metadata chunk 53 (just above the metadata chunk 54).
[0104] In this way, when the data generation unit 38 determines that there is a containment relationship between the new set and the existing set, it can also generate the metadata block 57 of the new set at a position just above or just below the metadata block 53 of the corresponding existing set. Through this description example, when the ink data 50 is generated, the metadata blocks 53 and 57 that have a containment relationship can also be associated with each other.
[0105] Fig.17 is shown with Fig.14 1 is a diagram of a third description example of the corresponding ink data 50. The ink data 50 is composed of stroke data 51 and metadata 52D. The metadata 52D is composed of 9 metadata chunks including metadata chunks 53, 54, and 58. The metadata chunks 53 and 54 are related to Fig.13 The data is the same as in the case of , so the description is omitted.
[0106] The ninth metadata chunk 58 describes the meta-information added to the new set specified by the lasso Ltmp. In the example of this figure, the ninth metadata chunk 58 is added to the last position of the metadata 52D according to the rule of adding the newly added meta-information to the lower position one by one. <traceview>The tag is described in a form that refers to an identification code (here, group ID=g1) of a stroke group included in the metadata block 53.
[0107] That is, when the data generation unit 38 determines that there is an inclusion relationship between the new set and the existing set, it can also generate the metadata block 58 of the new set in the form of referring to the identification code of the existing set included in the metadata block 53. Through this description example, when the ink data 50 is generated, the metadata blocks 53 and 58 having an inclusion relationship can also be associated with each other.
[0108] In this way, after the data generating unit 38 generates the metadata chunks 55, 56-58 by describing in different forms according to the determination result of step S8 (steps S10, S11), the process proceeds to the next step S12.
[0109] exist Fig.11 In step S12, the data generating unit 38 outputs the ink data 50 created and updated in step S10 or S11. Specifically, the “output” includes storage in the memory 20, writing to a data file, and sending to an external device. Thus, the operation of the ink data generating device 10 ends.
[0110] <Effect of the Second Action>
[0111] As described above, the ink data generating device 10 is a device for generating ink data 50 including metadata 52, wherein the metadata 52 records the meta-information of each set of strokes, wherein the ink data generating device 10 comprises: a first acquisition unit 30, which acquires new set data 42 (first set data) representing stroke elements belonging to a new set (first set) to which meta-information has not yet been assigned; a second acquisition unit 32, which acquires existing set data 44 (second set data) representing stroke elements belonging to an existing set (second set) to which meta-information has been assigned for each set of strokes; a set determination unit 36, which uses the new set data 42 and the existing set data 44 acquired respectively to determine the inclusion relationship between the new set and the existing set by comparing the stroke elements of both sides; and a data generating unit 38, which generates the first metadata (metadata blocks 55, 56-58) of the new set by describing it in a different form according to the determination result of the set determination unit 36.
[0112] Alternatively, according to the ink data generating method and program, one or more processors (for example, the host processor 18) execute: a first acquisition step ( Figure 4 S2), obtaining new set data 42 representing stroke elements belonging to a new set that has not yet been assigned metadata; a second acquisition step (S7), obtaining existing set data 44 representing stroke elements belonging to an existing set that has already been assigned metadata for each set of strokes; a determination step (S8), using the newly acquired set data 42 and the existing set data 44 to determine the inclusion relationship between the new set and the existing set by comparing the stroke elements of both sides; and a generation step (S10, S11), generating metadata chunks 55, 56-58 by describing in different forms according to the determination result in the determination step.
[0113] Due to such a configuration, the inclusion relationship between sets of strokes can be understood by analyzing the description form of the metadata 52, and various processing of the metadata taking the inclusion relationship into consideration can be performed (specifically, editing support (e.g., mutual reference of metadata), automatic editing (e.g., combination, replacement, inheritance of metadata), etc.). As a result, the processability of the ink data 50 during or after the generation of the metadata 52 can be improved.
[0114] Furthermore, when the data generation unit 38 determines that there is an inclusion relationship between the existing set and the new set, the metadata chunks 56-58 may be generated so as to be dependent on the metadata chunk 53 of the existing set as the upper or lower set. By providing a dependency relationship between the metadata chunks 53 and 56-58, it is easier to understand the inclusion relationship between the two.
[0115] [Modifications]
[0116] Hereinafter, an example will be described in which the ink data 60 whose ink description language is WILL is generated according to the description method of JSON.
[0117] <Example of Generating Ink Data 60>
[0118] Fig.18 : is a diagram showing the positional relationship between the content and the three lassoes Lpr1, Lpr2, and Lpr3. Assume that the user performs the 11th annotation operation on the stroke set surrounded by the lasso Lpr1, and performs the 12th annotation operation on the stroke set surrounded by the lasso Lpr2. Through the 11th annotation operation, the set with ID=01-06 as the stroke element is given the meta-information that the type (Type) is "word" and the value (Value) is "This". Through the 12th annotation operation, the set with ID=07-09 as the stroke element is given the meta-information that the type is "word" and the value is "is".
[0119] Assume that after the user has performed the 11th to 18th annotation operations, the user has performed the 19th annotation operation on the stroke set surrounded by the lasso Lpr3. Through the 19th annotation operation, the set of stroke elements with ID = 01-09 is given meta information with the type of "text line" and the value of "none". For the convenience of illustration, the records of the 13th to 18th lassos are omitted.
[0120] Here, the data generation unit 38 performs a "storage type" data processing in which the ink data 60 is updated at a predetermined timing after accumulating the meta-information assigned by the user. In this case, the data generation unit 38 obtains the determination result of the set determination unit 36, and generates the ink data 60 in a manner including a metadata block 65 that hierarchically describes the corresponding meta-information when the above-mentioned inclusion relationship is satisfied. In addition, the data generation unit 38 may also determine the meta-information that has not been assigned based on the relationship with the meta-information that has been assigned to the upper set or the lower set, and automatically add the code text 68 for assigning the meta-information.
[0121] Fig.19 is shown with Fig.18 FIG. 6 is a diagram showing an example of description of corresponding ink data 60. The ink data 60 is composed of stroke data 61 that describes strokes and stroke sets respectively and metadata 62 that describes semantic attributes. The metadata 62 is composed of 9 metadata chunks including metadata chunks 63, 64, and 65. The 11th metadata chunk 63 describes the meta-information assigned to the set (tg11) specified by the lasso Lpr1. The 12th metadata chunk 64 describes the meta-information assigned to the set (tg12) specified by the lasso Lpr2.
[0122] The 19th metadata chunk 65 describes the meta-information assigned to the set (tg19) specified by the lasso Lpr3. In the example of this figure, the 19th metadata chunk 65 is added to the last position of the metadata 62 according to the rule of sequentially adding the newly assigned meta-information to the lower position. However, in the metadata chunk 65, hierarchical data 66 defining two sets (tg11, tg12) as lower sets is added using "has_subgroup".
[0123] In addition, a code text 68 indicating that the value of the upper set (tg19) is "This is" is added to the metadata block 65. This "This is" is a string obtained by combining "This", which is the value of the lower set (tg11), and "is", which is the value of the lower set (tg12).
[0124] The results, such as Fig. 20 As shown, the set (tg19) with ID=01-09 as the stroke element is assigned "Type: Text line" as meta-information, and "Value: This is" and "Sub-group: tg11, tg12" are automatically assigned as meta-information. In the example of this figure, when the value is "None", the meta-information "This is" is automatically assigned, but the meta-information can also be assigned regardless of the user's intention.
[0125] In this way, when the data generation unit 38 determines that there is an inclusion relationship between the first set (tg19) and the second set (tg11, tg12), it can also generate the metadata block 65 of the first set in the form of referring to the identification code (tg11, tg12) of the second set. Through this description example, when the ink data 60 is generated, the metadata blocks 63-65 having an inclusion relationship can also be associated with each other.
[0126] <Effects of Modification Example>
[0127] As described above, the ink data generating device 10 is a device for generating ink data 60 including metadata 62, wherein the metadata 62 records the meta-information of each set of strokes, wherein the ink data generating device 10 comprises: a data acquisition unit (a first acquisition unit 30, a second acquisition unit 32), which acquires set data (new set data 42, existing set data 44) of stroke elements belonging to a set of strokes for each set; and a data output unit (data generating unit 38), which outputs the ink data 60 including metadata 62 representing the inclusion relationship between multiple sets based on the acquired set data of each set.
[0128] Alternatively, according to the ink data generation method and program, one or more processors (for example, the host processor 18) execute: obtaining step ( Figure 4 S2 Fig.11 S7), obtaining the set data of the stroke elements belonging to the set of strokes for each set; and output step ( Fig.11 S12), based on the obtained set data of each set, output ink data 60 including metadata 62 representing the inclusion relationship between multiple sets.
[0129] Due to such a configuration, the inclusion relationship between the stroke sets can be understood by analyzing the content of the output metadata 62, and various processing (specifically, editing support, automatic editing, etc.) of the meta-information taking the inclusion relationship into consideration can be performed. Thus, the processability of the ink data 60 during or after the output of the metadata 62 can be improved.
[0130] Furthermore, the data output unit may output metadata including the metadata chunk 65 in which the semantic attribute is described using a data structure indicating the inclusion relationship of a set.
[0131] Description of symbols
[0132] 10 ink data generating device, 12 touch panel display, 18 host processor, 20 memory, 28 UI unit, 30 first acquisition unit, 32 second acquisition unit, 34 attribute inference unit, 36 set determination unit, 38 data generating unit (data output unit), 40 drawing processing unit, 42 new set data (first set data), 44 existing set data (second set data), 50, 60 ink data, 51, 61 stroke data, 52 (A / B / C / D), 62 metadata, 53, 54, 65 metadata block (second metadata), 55-58, 63, 64 metadata block (first metadata).< / traceview> < / trace>
Claims
1. An ink data generating device, generating ink data including metadata, wherein the metadata describes meta information of each set of strokes, wherein the ink data generating device is characterized by: A first acquisition unit acquires first set data representing stroke elements belonging to a first set to which meta information has not yet been assigned; a second acquisition unit that acquires second set data representing stroke elements belonging to a second set to which meta-information has been assigned for each set of strokes; a set determination unit that uses the first set data obtained by the first acquisition unit and the second set data obtained by the second acquisition unit to determine an inclusion relationship between the first set and the second set by comparing stroke elements of both sets; and The data generating unit generates first metadata of the first set by describing the first set in a different format according to a determination result of the set determining unit.
2. The ink data generating device according to claim 1, It is characterized in that The data generation unit generates the first metadata so as to be dependent on the second metadata of the second set which is a superordinate set or a subordinate set, when it is determined that there is an inclusion relationship between the first set and the second set.
3. The ink data generating device according to claim 2, It is characterized in that The data generation unit generates the first metadata at a position within the second metadata when it is determined that there is an inclusion relationship between the first set and the second set.
4. The ink data generating device according to claim 2, It is characterized in that The data generation unit generates the first metadata at a position directly above or directly below the second metadata when it is determined that there is an inclusion relationship between the first group and the second group.
5. The ink data generating device according to claim 2, It is characterized in that The data generation unit generates the first metadata in a form that refers to an identification code of the second set, when it is determined that there is an inclusion relationship between the first set and the second set.
6. The ink data generating device according to claim 1, It is characterized in that The data generation unit generates the first metadata at a position in accordance with a predetermined order of providing meta information when it is determined that there is no inclusion relationship between the first group and the second group.
7. The ink data generating device according to claim 1, It is characterized in that The invention further includes an attribute estimating unit for estimating an attribute to be given to the first set based on a mutual relationship between positions or shapes of a plurality of strokes constituting the first set.
8. The ink data generating device according to claim 7, It is characterized in that The attribute estimation unit is configured to include a recognizer that inputs a feature value related to an endpoint of a stroke and outputs a label value of a semantic attribute.
9. The ink data generating device according to claim 8, It is characterized in that The device further comprises a UI unit configured to simultaneously display a plurality of strokes in a display area and to select stroke elements belonging to the first set from the plurality of strokes through handwriting input by a user, The UI unit further displays a user control capable of inputting a semantic attribute to be assigned to the first set, The data generation unit generates the first metadata indicating a semantic attribute input using the user control.
10. The ink data generating device according to claim 9, It is characterized in that The UI unit displays the semantic attribute estimated by the attribute estimation unit as an initial value of the user control.
11. An ink data generating device for generating ink data including metadata, wherein the metadata describes meta information of each set of strokes, wherein the ink data generating device is characterized by comprising: a data acquisition unit that acquires set data representing stroke elements belonging to a set of strokes for each set; and a data output unit that outputs metadata indicating an inclusion relationship between a plurality of sets based on the set data of each set acquired by the data acquisition unit, When the ink data is generated, metadata chunks having a containment relationship are associated with each other, and the metadata chunks describe semantic attributes using a data structure representing a containment relationship of a set.
12. The ink data generating device according to claim 11, It is characterized in that The data output section outputs the metadata including metadata chunks.
13. A method for generating ink data, generating ink data including metadata, wherein the metadata describes meta information of each set of strokes, wherein one or more computers execute: A first acquisition step of acquiring first set data representing stroke elements belonging to a first set to which meta information has not yet been assigned; a second acquisition step of acquiring second set data representing stroke elements belonging to a second set to which meta information has been assigned for each set of strokes; a determination step, using the first set data and the second set data obtained respectively, to determine the inclusion relationship between the first set and the second set by comparing the stroke elements of both sets; and The generating step generates first metadata of the first set by describing the metadata in a different form according to the determination result in the determining step.
14. An ink data generation program product for generating ink data including metadata, wherein the metadata describes meta information of each set of strokes, wherein the ink data generation program is characterized in that one or more computers are caused to execute: A first acquisition step of acquiring first set data representing stroke elements belonging to a first set to which meta information has not yet been assigned; a second acquisition step of acquiring second set data representing stroke elements belonging to a second set to which meta information has been assigned for each set of strokes; a determination step, using the first set data and the second set data obtained respectively, to determine the inclusion relationship between the first set and the second set by comparing the stroke elements of both sets; and The generating step generates first metadata of the first set by describing the metadata in a different form according to the determination result in the determining step.
