Data creation apparatus and data creation method

Abstract

To provide a data creation device and method for generating more accurate metadata from a non-structured document by using a large-scale language model, and for reducing the manual correction of the metadata afterwards.SOLUTION: The data creation server 1300 includes a token verification unit that uses metadata generated by inputting a document to a large scale language model and a token obtained from a node on a trace path of a dependency parse tree based on the document to verify whether the metadata is appropriate, a token match score calculation unit that acquires a token match score based on a verification result, a grouping unit that uses the metadata as it is when the token match score satisfies a predetermined condition and groups the metadata according to user orientation data in which an orientation of a user is reflected, a reliability score calculation unit that calculates a reliability score for the grouped metadata, and a rank calculation unit that ranks the metadata according to the reliability score.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention is suitable for application to a data creation device and a data creation method relating to a technology for generating metadata from unstructured documents using a large-scale language model (hereinafter also referred to as "LLM"). [Background technology]

[0002] In recent years, advanced technologies have been developed to ensure the reliability of metadata generated from unstructured documents using Large Language Models (LLMs). To ensure the reliability of the generated metadata, it is important to accurately extract necessary attribute information from unstructured documents before generating the metadata. Such methods for ensuring the reliability of metadata involve multiple steps for validating the metadata. For example, in the first step, dependency parse tree (DPT) analysis is employed for preliminary validation. This step ensures that the generated metadata faithfully represents the context, taking into account the syntactic complexity of the text. In the second step, expert users known as data stewards conduct critical reviews based on their specialized knowledge. This adds an additional layer of metadata validation, ensuring thorough investigation of metadata with questionable integrity (see, for example, Patent Documents 1 and 2). The integration of such expert knowledge serves as an important and crucial checkpoint for maintaining high-quality metadata generation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 11,664,099 [Patent Document 2] U.S. Patent Publication No. 2014 / 0082091 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the technologies described in Patent Documents 1 and 2, if the reliability of the generated metadata is low, extensive verification by experts is required, especially when dealing with large amounts of unstructured documents, which places a heavy burden on expert supervision.

[0005] The present invention has been made in consideration of the above points, and aims to propose a data creation device and a data creation method that can generate more accurate metadata from unstructured documents by utilizing a large-scale language model, thereby reducing the effort required to manually correct the metadata afterwards. [Means for solving the problem]

[0006] In order to solve this problem, the present invention provides a node extraction unit that extracts nodes on a trace path of a dependency analysis tree based on an input document; a token verification unit that performs token verification to verify whether the metadata is appropriate using metadata generated by inputting the document into a large-scale language model and tokens obtained from the nodes; a token match score calculation unit that obtains a token match score based on the result of the token verification; a grouping unit that uses the metadata as is if the token match score satisfies a predetermined condition, and groups the metadata according to user preference data that reflects user preferences; a reliability score calculation unit that calculates reliability scores for the grouped metadata; a rank calculation unit that ranks the metadata according to the reliability scores; and a review unit that displays the ranked metadata together with its rank on a rank display screen.

[0007] Furthermore, in the present invention, the method includes a node extraction step in which a node extraction unit extracts nodes on a trace path of a dependency analytic tree based on an input document; a token verification step in which a token verification unit performs token verification to verify whether the metadata is appropriate or not, using metadata generated by inputting the document into a large-scale language model and tokens obtained from the nodes; a token match score calculation step in which a token match score calculation unit obtains a token match score based on a result of the token verification; a grouping step in which, if the token match score satisfies a predetermined condition, the grouping unit uses the metadata as is and groups the metadata according to user preference data that reflects user preferences; a reliability score calculation step in which a reliability score calculation unit calculates reliability scores for the grouped metadata; a rank calculation step in which a rank calculation unit ranks the metadata according to the reliability scores; and a display step in which the ranked metadata is displayed on a rank display screen together with its rank. [Effects of the Invention]

[0008] According to the present invention, it is possible to generate more accurate metadata from unstructured documents using a large-scale language model, thereby reducing the effort required to manually correct the metadata after the fact. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a system configuration diagram showing an example of the configuration of a data creation system including a data creation server as an example of a data creation device according to a first embodiment. [Figure 2] FIG. 2 is a functional block diagram illustrating an example of the configuration of a data creation server. [Figure 3] FIG. 10 is a conceptual diagram showing an example of a conceptual procedure of a data creation method by a data creation server. [Figure 4] 10 is a flowchart illustrating an example of a procedure for a main process. [Figure 5]10 is a flowchart illustrating an example of a procedure for a metadata token verification process. [Figure 6] 10 is a flowchart illustrating an example of a procedure for metadata correction processing. [Figure 7] 10 is a flowchart illustrating an example of a procedure for a reliability score calculation process. [Figure 8] 10 is a flowchart illustrating an example of a procedure for calculating a reliability score for each field. [Figure 9] 10 is a flowchart illustrating an example of a procedure for a rank calculation process. [Figure 10] FIG. 10 is a diagram illustrating an example of a user prompt setting screen. [Figure 11] FIG. 10 is a diagram showing an example of a first user orientation setting screen. [Figure 12] FIG. 10 is a diagram showing an example of a second user orientation setting screen. [Figure 13] FIG. 10 is a diagram showing an example of a reliability score-based metadata review screen. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. (1) First embodiment 1 is a system configuration diagram showing an example of the configuration of a data creation system 1400 including a data creation server 1300 as an example of a data creation device according to the first embodiment. The data creation system 1400 includes the data creation server 1300, a client computer (hereinafter abbreviated as "client") 1330, and a network 1320. The data creation server 1300 and the client 1330 are connected via the network 1320.

[0011] The data creation server 1300 includes a CPU (Central Processing Unit) 1301, an input interface (hereinafter abbreviated as "I / F") 1302, an output I / F 1303, memory 1304, a communication unit 1305, and a storage device 1307. The memory 1304 has a volatile storage element such as RAM (Random Access Memory) and stores programs executed by the CPU 1301. The storage device 1307 is a non-volatile storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores programs, data, etc. in a non-volatile manner.

[0012] The storage device 1307 includes, for example, a metadata token verification program 1308, a metadata token correction program 1309, a reliability score calculation program 1310, and a database 1311. The storage device 1307 also includes programs having other functions, which will be described later, but the description thereof is omitted here.

[0013] The metadata token verification program 1308 has a function of verifying tokens related to the metadata 105. The metadata token verification program 1308 will be described in detail later.

[0014] The metadata token correction program 1309 has a function of correcting tokens related to the metadata 105. The metadata token correction program 1309 will be described in detail later.

[0015] The reliability score calculation program 1310 has a function of calculating the reliability score. Details of the reliability score calculation program 1310 will be described later.

[0016] The database 1311 includes metadata 105, scanned documents 101, and a DPT (Dependency Parse Tree) 104. The scanned documents 101 are prepared in advance in a text format. The documents are, for example, so-called unstructured documents.

[0017] The metadata 105, which will be described in detail later, is data generated from the scanned document 101 using a large-scale language model (hereinafter also referred to as "LLM"). The metadata 105 is information about predetermined attributes such as gender and age.

[0018] DPT 104 is a method for analyzing the structure of a sentence in natural language processing. Specifically, it represents the dependencies between words in a sentence as a tree structure. The tree structure indicates the relationships between words that make up a sentence, for example, contained in the scanned document 101. The tree structure indicates the dependencies between the subject, predicate, object, and other terms of the sentence. DPT 104 is used to understand the grammatical structure of the sentence. The grammatical structure includes a trace path that indicates the connections between nodes, which are, for example, words.

[0019] 2 is a functional block diagram showing an example of the configuration of the data creation server 1300. In the illustrated example, for simplicity of explanation, some components such as the storage device 1307 are omitted. The data creation server 1300 includes, as hardware, for example, a CPU (Central Processing Unit) 1301, memory 1304, a storage device 1307, and a communication unit 1305 including a network interface.

[0020] The CPU 1301 reads the program and data from the storage device 1307 into a storage area of ​​the memory 1304 and executes the program while using the data. The program has a function of performing calculations using the data and displaying the calculation results on a display device (not shown).

[0021] The memory 1304 stores the above-mentioned data, such as a user prompt setting screen 205, a scanned document 101, a prompt 102, generated metadata 105, a DPT (Dependency Parse Tree) 104, a users interest terms setting screen 207, a token match score calculation unit 108, user interest data 206, an interest term cluster display screen, a PDC (Pentaho Data Catalogue) 121, and corrected metadata 119. The corrected metadata 119 may also be referred to as corrected metadata 105.

[0022] In this embodiment, the metadata 105 includes, for example, a combination of a key representing an attribute of each field and a value of the attribute. Examples of the attributes of each field include patient information, prescription drug information, and prescription conditions. The PDC 121 is a data catalog composed of a combination of the attributes and values ​​of each field included in a large number of metadata 105, and the attributes and values ​​of each field of each metadata 105 can be searched using a predetermined API (Application Programming Interface) not shown. The PDC 121 can reflect the attributes and values ​​of each field of new metadata 105. Therefore, the reliability of the content of the PDC 121 depends on whether the metadata 105 is appropriate.

[0023] Furthermore, the memory 1304 has, as the above-mentioned programs, a node extraction unit 106, a metadata token verification unit 107, a token match score calculation unit 108, a reliability score verification unit, a rank calculation unit 208, and a review unit 117, and preferably further has a metadata correction unit 110. The reliability score verification unit includes a metadata reliability score calculation unit 114 and a field-specific metadata reliability score calculation unit 115.

[0024] The node extraction unit 106 extracts nodes on the trace path of the DPT 104 as a dependency analysis tree based on the input document.

[0025] The token verification unit 107 performs token verification to verify whether the metadata 105 is correct, using the metadata 105 generated by inputting the document into a large-scale language model and the tokens obtained from the nodes. The metadata 105 is attribute information such as gender.

[0026] The token match score calculation unit 108 obtains a token match score based on the result of the token verification by the token verification unit 107. The token match score indicates, for example, how valid the content of the metadata 105 is when considering the nodes on the trace path of the dependency parse tree (DPT 104) based on the input document. Therefore, a higher token match score indicates more validity, and a lower token match score indicates less validity.

[0027] If the token match score satisfies an example of a predetermined condition, that is, exceeds a threshold value α, the grouping unit 113 uses the metadata 105 as is and groups the metadata 105 according to user preference data 206 that reflects the preferences of a user, such as a data steward 111.

[0028] The confidence score calculation unit calculates a confidence score for the grouped metadata 105. The metadata 105 includes fields corresponding to each attribute. The confidence score calculation unit has a metadata confidence score calculation unit 114 and a field-specific metadata confidence score calculation unit 115. The metadata confidence score calculation unit 114 calculates a confidence score for the entire metadata 105. The field-specific metadata confidence score calculation unit 115 calculates a confidence score for each field of the metadata 105.

[0029] The rank calculation unit 208 ranks the metadata 105 according to the reliability score. The review unit 117 displays the ranked metadata 105 together with the rank on a rank display screen.

[0030] The data creation server 1300 includes a metadata correction unit 110 that corrects the metadata 105 if the token match score does not satisfy a predetermined condition (for example, exceeding a threshold α).

[0031] The metadata correction unit 110 corrects the metadata 105 based on the trace path of the DPT 104 and each node on the trace path, taking into consideration, for example, each word that is each node and the context between the preceding and following words.

[0032] The node extraction unit 106 extracts nodes on the trace path of the DPT 104 based on the matching result of the parts of speech (hereinafter also referred to as “PoS”) of the metadata 105.

[0033] In this embodiment, the rank calculation unit 208 ranks the metadata 105 based on the user preference data 206 and the reliability score.

[0034] 3 is a conceptual diagram showing an example of the conceptual procedure of a data creation method by the data creation server 1300. In the data creation server 1300, the data creation process is executed as follows.

[0035] First, the outline of the data creation method is a node extraction step in which a node extraction unit 106 extracts nodes on a trace path of a DPT 104 based on an input document; a token verification step in which a token verification unit 107 performs token verification to verify whether or not the metadata 105 is correct, using metadata 105 generated by inputting the document into a large-scale language model 103 and tokens obtained from the nodes; a token match score calculation step in which a token match score calculation unit 108 obtains a token match score based on the result of the token verification; If the score satisfies a predetermined condition (for example, exceeding a threshold value α), the metadata 105 is used as is, and the metadata 105 is grouped according to user-oriented data 206 that reflects the orientation of a user such as a data steward 111.The method includes a reliability score calculation step in which a reliability score calculation unit calculates reliability scores for the grouped metadata 105, a rank calculation step in which a rank calculation unit 208 ranks the metadata 105 according to the reliability scores, and a display step in which the ranked metadata 105 is displayed on a rank display screen together with the rank.

[0036] More specifically, the scanned document 101 is input to the LLM 103, and the node extraction unit 106 uses the DPT 104 to extract nodes of a trace path for the scanned document 101 in order to obtain tokens.

[0037] The prompt 102 contains an instruction to create metadata from the scanned document 101. When the prompt 102 and the scanned document 101 are input, the LLM 103 generates metadata 105. This metadata 105 includes a combination of a key indicating an attribute of each field extracted from the scanned document 101 and a value of the attribute.

[0038] The metadata token verification unit 107 performs token verification based on the nodes extracted by the node extraction unit 106 and the metadata 105 .

[0039] The token match score calculation unit 108 compares the token match score with a predetermined threshold α based on the result of token verification by the metadata token verification unit 107. If the token match score calculation unit 108 determines that the token match score is greater than the predetermined threshold α, it executes a grouping process, whereas if it does not determine that the token match score is greater than the predetermined threshold α, it executes a metadata correction process S110 and then executes a grouping process.

[0040] The grouping unit 113 performs a grouping process (to be described later) on each piece of metadata 105 in accordance with user-oriented data 206 that reflects the intention of a user, such as a data steward 111. In the grouping process, each piece of metadata 105 is classified by grouping.

[0041] The metadata reliability score calculation unit 114 calculates the reliability score for each metadata unit, for example. The field-specific metadata reliability score calculation unit 115 calculates the reliability score for each field of the metadata.

[0042] The rank calculation unit 208 calculates the rank based on the calculated reliability scores for each metadata unit and the reliability scores for each field. The rank calculation unit 208 may calculate the rank for each group, for example.

[0043] The review unit 117 uses, for example, a GUI (Graphical User Interface) to display a reliability score-based metadata review screen (described later) to perform a review and rank the metadata 105 based on the calculated reliability score. In this embodiment, the review refers to, for example, determining whether the metadata 105 generated from the document 101 is appropriate.

[0044] The review unit 117 determines whether the metadata 105 is appropriate and needs to be corrected, based on the ranked metadata 105. If it is determined that the metadata 105 does not need to be corrected, the review unit 117 reflects the metadata 105 in the PDC 121. On the other hand, if it is determined that the metadata needs to be corrected, the metadata correction unit 110 corrects the metadata 105 to produce corrected metadata 119, and reflects the corrected metadata 119 in the PDC 121.

[0045] 4 is a flowchart showing an example of the procedure of the main processing, which is an example of a data creation method according to this embodiment.

[0046] In step S301, a metadata generation process is executed to generate metadata 105 from the scanned document 101 according to the input prompt 102. In step S302, a token of the metadata 105 is obtained using the DPT 104, and a metadata token verification process is executed.

[0047] In step S303, a metadata correction process is executed to correct the metadata 105 based on the nodes on the trace path of the DPT 104 of the document 101. In step S304, a reliability score calculation process is executed to calculate the reliability score of the metadata, and a ranking calculation process is executed to perform ranking for review.

[0048] 5 is a flowchart showing an example of the procedure of the metadata token verification process, which is executed by the metadata token verification unit 107.

[0049] In step S401, the metadata token validator 107 generates a DTP 104 from the scanned document 101. This DTP 104 helps identify grammatical dependencies between words and relational analysis of the text. In step S402, the metadata token validator 107 calculates tokens consisting of key-value pairs of the metadata 105 generated from the LLM 103.

[0050] In step S403, the metadata token verification unit 107 calculates nodes on the trace path of the DPT 104 to evaluate the token. Specifically, the metadata token verification unit 107 first calculates which node (e.g., word) on the trace path of the DPT 104 corresponds to the token in the metadata 105. For example, a token containing a value can also be referred to as a value token. Next, the metadata token verification unit 107 searches for a leaf node on the trace path of the DPT 104 that begins with the token. The metadata token verification unit 107 traces the DPT 104 to confirm the trace path from the token to the topmost parent node of the DPT 104, thereby verifying whether the metadata 105 is correct, i.e., whether the correct usage and terminology are used in the context of the document 101.

[0051] In step S404, the token match score calculation unit 108 calculates the token match score. By calculating the token match score, the token match score calculation unit 108 determines to what extent the tokens in the metadata 105 match the nodes on the trace path of the DPT 104.

[0052] In step S405, the metadata token verification unit 107 determines whether the token match score is greater than a predetermined threshold α, as a predetermined condition. If it is determined in step S405 that the token match score is greater than the predetermined threshold α, the metadata 105 is determined to be correct and the metadata token verification process is terminated. On the other hand, if it is determined that the token match score is not greater than the predetermined threshold α, step S406 is executed. In the former case, the accuracy of the metadata 105 extracted from the unstructured document described above can be ensured. In step S406, the metadata correction unit 110 corrects the metadata 105.

[0053] In step S407, the metadata token verification unit 107 determines whether any other tokens remain. If any other value tokens remain in step S407, the metadata token verification unit 107 returns to step S403 and executes the process from here on, whereas if no other value tokens remain, it executes the metadata token verification process. In the former case, comprehensive scrutiny of all generated metadata 105 can be guaranteed. This ensures high accuracy and reliability in the extraction of the metadata 105.

[0054] 6 is a flowchart showing an example of the procedure of metadata correction processing. The metadata correction processing is executed by the metadata correction unit 110. The illustrated example shows an example of metadata correction processing applied to metadata 105 that was not determined to be correct in the token verification described above. In the metadata correction processing, for example, metadata 105 whose token match score is equal to or less than a threshold value α may be removed.

[0055] In step S501, the metadata correction unit 110 searches for or calculates previous tokens of the metadata 105 obtained from the LLM 103. Previous tokens are tokens that correspond to parts of previously generated metadata 105. In this step S501, the metadata correction unit 110 identifies and extracts tokens that were calculated before, for example, the most recently calculated token, of the currently targeted metadata 105.

[0056] In step S502, the metadata correction unit 110 calculates DPT nodes on the trace path based on the matching PoS of the metadata. In step S502, the metadata correction unit 110 analyzes the grammatical structure of the metadata 105, for example, to ensure consistency between different word fragments based on the matching result of the PoS of the metadata 105. In this step S502, the metadata correction unit 110 identifies relevant nodes (important points or elements in the structured path) determined by the matching result of the PoS of the metadata 105. This identification process is crucial for maintaining the grammatical and semantic structure of the metadata 105.

[0057] In step S503, the metadata correction unit 110 checks the consistency between the previous token (e.g., a value) and a node on the trace path by determining whether the previous token matches a node (word) on the trace path of the DPT 104. If the previous token does not match the node, the metadata correction unit 110 returns to step S501 and executes the process from here on, whereas if the previous token matches the node, the metadata correction unit 110 executes step S504.

[0058] In step S504, the metadata corrector 110, for example, the data steward 111, corrects the mismatched tokens based on the trace path. This correction is based on the previously used trace path and involves replacing, modifying, or updating tokens in the metadata 105 to ensure that they match the established trace path. That is, if a previously extracted token does not match an expected token on the trace path, the mismatched token is corrected. This correction ensures the consistency and accuracy of the metadata 105.

[0059] 7 is a flowchart showing an example of the procedure for the reliability score calculation process, which is executed by the metadata reliability score calculation unit 114.

[0060] In step S601, the metadata reliability score calculation unit 114 calculates the probabilities of tokens in the metadata 105. The "probability" here includes, for example, evaluating how structurally appropriate, for example, grammatically, the value of an attribute in each field of the metadata 105 is, or how relevant it is to a certain document, based on previous data, patterns, or predictions from a trained model.

[0061] In step S602, the metadata confidence score calculation unit 114 calculates the entropy of the tokens of each piece of metadata 105, where entropy is, for example, a measure of the uncertainty or unpredictability of each piece of metadata 105. High entropy indicates a wide range of possible values ​​for the token, while low entropy indicates more certainty about the value of the token.

[0062] In step S603, the metadata reliability score calculation unit 114 calculates the total entropy of the metadata 105. The metadata reliability score calculation unit 114 aggregates the entropy values ​​based on all of the individual metadata tokens to calculate the total entropy for the entire set of metadata 105. The total entropy reflects the overall uncertainty of the set of metadata 105 and is an important factor in determining the reliability score.

[0063] In step S604, the metadata confidence score calculator 114 calculates a logarithmic transformation as a mathematical operation on the total entropy. This mathematical operation is a way to make the total entropy value more interpretable or manageable, especially when the total entropy value spans several orders of magnitude. The total entropy is used for scaling. The logarithmic transformed total entropy value is used to generate all confidence scores for the entire set of metadata 105.

[0064] In step S605, the metadata reliability score calculation unit 114 obtains reliability scores for all generated metadata 105. The reliability score represents, for example, a quantified level of accuracy and reliability of the metadata 105. This reliability score is extremely important for a user's decision-making process regarding the metadata 105. Here, lower entropy (low randomness) is included in higher entropy. The metadata reliability score calculation unit 114 calculates a total entropy of 0.079, for example, by logarithmic transformation. As a result, the reliability score is calculated as, for example, 1-0.079, reflecting lower entropy (low randomness) in higher entropy.

[0065] Fig. 8 is a flowchart showing an example of the procedure for each field reliability score calculation process. The flowchart shown in Fig. 8 is almost the same as the flowchart shown in Fig. 7, except for whether the reliability score is calculated for each metadata 105 or for each field thereof. Therefore, some details that have already been explained will not be explained again. In the each field reliability score calculation process, the field-specific metadata reliability score calculation unit 115 calculates a reliability score for each field of the metadata 105.

[0066] In step S701, the field-specific metadata reliability score calculation unit 115 calculates the likelihood of the tokens in the metadata 105.

[0067] In step S702, the field-specific metadata reliability score calculation unit 115 calculates the entropy of the tokens of each metadata 105.

[0068] In step S703, the field-specific metadata reliability score calculation unit 115 calculates the grouping entropy for each group based on the tokens in each field of the metadata 105. Examples of each group include groups for each attribute included in the metadata 105 (e.g., patient information, prescription drug information, prescription conditions). This allows the variability within a specific field of the metadata 105 to be reflected in the reliability score.

[0069] In step S704, the per-field metadata reliability score calculation unit 115 calculates the total entropy of all fields based on the tokens in each field, thereby obtaining an overall reliability score across each field of the metadata 105, rather than the entire metadata 105.

[0070] In step S705, the field-specific metadata reliability score calculation unit 115 calculates the logarithmic transformation of the total entropy. In step S706, the field-specific metadata reliability score calculation unit 115 calculates the reliability score of each field of the metadata 105.

[0071] 9 is a flowchart showing an example of the procedure of the rank calculation process, which is executed by the rank calculation unit 208. In the rank calculation process, a rank is calculated for a review using the reliability score and the user orientation.

[0072] In step S801, a user's preference is input when grouping the generated metadata 105 (see, for example, FIG. 10 described later). In step S802, the rank calculation unit 208 calculates a reliability score for the entire metadata 105 of the document according to the input user's preference.

[0073] In step S803, the rank calculation unit 208 calculates the reliability score for each field of the metadata 105.

[0074] In step S804, the rank calculation unit 208 calculates the rank by taking into account both the user's preference and the reliability score. By taking both into account in this way, the ranking reflects both the statistical reliability of the metadata 105 and the specific purpose according to the user's preference. In step S805, the rank calculation unit 208 displays the ranked metadata 105 for review. Such a visual display allows the user to easily understand the ranking result and determine whether the metadata 105 is appropriate based on the evaluated quality of the metadata 105.

[0075] In step S806, the rank calculation unit 208 determines whether the metadata 105 is appropriate. If the metadata 105 is appropriate in step S806, the rank calculation unit 208 has the user review the metadata 105 and executes step S807 to end the rank calculation process. On the other hand, if the metadata 105 is not appropriate in step S806, the rank calculation unit 208 ends the rank calculation process. In step S807, the metadata 105 is incorporated into the PDC 121.

[0076] 10 is a diagram showing an example of the user prompt setting screen 205. The user prompt setting screen 205 is a screen for writing a prompt for generating the metadata 105 using the LLM 103.

[0077] The user prompt setting screen 205 is, for example, a screen using a GUI (Graphical User Interface). The user prompt setting screen 205 has, for example, a prompt setting field 900 having a prompt description field 901, and an execute button 902. The prompt description field 901 is an item field for the user to input the prompt 102 desired.

[0078] The execute button 902 is a button for executing the prompt 102 entered in the prompt description field 901. When the execute button 902 is operated, the prompt 102 entered in the prompt description field 901 is input to the LLM 103.

[0079] 11 and 12 are diagrams showing examples of the user preference setting screen 207. Fig. 11 is an example of the first user preference setting screen 207, and Fig. 12 is an example of the second user preference setting screen 1130. The user preference setting screen 207 displays groupings of user preference settings for review groups, which are units for reviewing the metadata 105.

[0080] 11 has an input field 1001 for a user, such as the data steward, to input a term of interest, and an execute button 1002. When the user inputs a first term (e.g., the age of a patient) as a term of interest in the input field 1001 and operates the execute button 1002, the first term is reflected in the user orientation data 206 and is used when the rank calculation unit 208 executes the rank calculation process.

[0081] The second user preference setting screen 1130 shown in FIG. 12 includes a display field 1100 with a grouping mode display field 1101 at the top and a review group drop-down menu 1102 at the bottom in the vertical direction.

[0082] The grouping mode display field 1101 displays at least one cluster or group of the metadata 105 in different colors as a grouping result of the metadata 105 grouped by the review group set in the review group drop-down menu 1102. In the illustrated example, a first cluster 1101A to a fifth cluster 1101E are displayed simultaneously.

[0083] For example, each cluster represents a different group or category of metadata 105, organized and displayed based on user-defined "terms of interest." Within the visualization area are multicolored dots that represent data points. Each dot represents an individual metadata item with common characteristics, and the color coding helps distinguish between the various groupings.

[0084] For example, "Grouping based on output of 'interesting terms'" is displayed at the top of review group drop-down menu 1102. Review group drop-down menu 1102 allows the user to select one review group from among multiple review groups. Depending on the selected review group, grouping mode display field 1101 displays one of the above-mentioned first cluster 1101A to fifth cluster 1101E as a grouping result of metadata 105, or a combination of any clusters depending on the selected review group.

[0085] 13 is a diagram showing an example of a reliability score-based metadata review screen (hereinafter abbreviated as "metadata review screen") 117. The metadata review screen shows, for example, an example of a document review by the data steward 111. The metadata review screen outputs, for example, the reliability score of the third metadata.

[0086] The metadata review screen includes, for example, a document ID (IDentifier) ​​1201A, a reliability score of the first field 1201B, a reliability score of the second field 1201C, ..., a total metadata reliability score 1201Y, and whether or not corrections have been made 1201Z as display items in a display area 1201. Note that in the illustrated example, the values ​​of the reliability score of the first field 1201B and the reliability score of the second field 1201C are omitted for simplicity of explanation.

[0087] The document ID 1201A is an identifier of the document from which the third metadata described above is generated. The reliability score of the first field 1201B represents the reliability score of the first field among the metadata fields calculated as described above. The reliability score of the second field 1201C represents the reliability score of the second field among the metadata fields calculated as described above.

[0088] The total metadata reliability score 1201Y represents the reliability score of all metadata calculated as described above. The presence or absence of correction 1201Z indicates whether corrections have been made (corrected) or not (not corrected). For example, if corrections have been made, a check mark is displayed, and if corrections have not been made, an X mark is displayed.

[0089] Correction or non-correction 1201Z is entered by inputting a check mark or an X mark according to the reliability score of all metadata 105 based on the insight of a user such as the data steward 111. Note that correction or non-correction 1201Z may automatically input a check mark or an X mark according to the value of the all metadata reliability score.

[0090] The data creation server 1300 according to this embodiment includes a node extraction unit 106 that extracts nodes on a trace path of a DPT 104 based on an input document; a token verification unit 107 that performs token verification to verify whether the metadata 105 is appropriate using metadata 105 generated by inputting the document into a large-scale language model and tokens obtained from the nodes extracted based on the input document; a token match score calculation unit 108 that obtains a token match score based on the result of the token verification; a grouping unit 113 that uses the metadata 105 as is if the token match score satisfies a predetermined condition, and groups the metadata 105 according to user preference data 206 that reflects the preferences of a user (e.g., a data steward 111); various reliability score calculation units 114, 115 that calculate reliability scores for the grouped metadata 105; a rank calculation unit 208 that ranks the metadata 105 according to the reliability scores; and a review unit 117 that displays the ranked metadata 105 together with their ranks on a rank display screen.

[0091] With the above configuration, it is possible to generate more accurate metadata 105 from unstructured documents 101 using a large-scale language model, and reduce the effort required to manually correct the metadata 105 afterward.

[0092] In this embodiment, the metadata 105 includes each field, and the various reliability score calculation units 114, 115 include a metadata reliability score calculation unit 114 that calculates a reliability score for the entire metadata 105, and a field-specific metadata reliability score calculation unit 115 that calculates a reliability score for each field of the metadata 105. In this manner, it is possible to generate more accurate metadata 105 from the unstructured document 101 using a large-scale language model based on the overall reliability score of the metadata 105 that takes into account each field of the metadata 105, and it is possible to reduce the effort required to manually correct the metadata 105 afterwards.

[0093] The data creation server 1300 according to this embodiment includes a metadata correction unit 110 that corrects the metadata 105 when the token match score does not satisfy a predetermined condition. This makes it possible to create a PDC 121 that reflects accurate metadata 105.

[0094] In this embodiment, the metadata 105 includes a combination of a key representing an attribute of each field and a value of the attribute, which makes it possible to generate more accurate metadata 105 based on a reliability score of the metadata 105 that takes into account the key and its value of each field of the metadata 105, thereby reducing the effort required to manually correct the metadata 105 afterwards.

[0095] In this embodiment, the metadata correction unit 110 corrects the metadata 105 based on the trace path of the DPT 104 and each node on the trace path. This makes it possible to generate more accurate metadata 105 and reduce the effort required to manually correct the metadata 105 afterward.

[0096] In this embodiment, the node extraction unit 106 extracts nodes on the trace path based on the matching result of the PoS of the metadata 105. In this way, it is possible to generate more accurate metadata 105 while taking into consideration grammatical context including parts of speech, based on the nodes on the trace path extracted from the matching result of the PoS of the metadata, and it is possible to reduce the effort required to manually correct the metadata 105 afterwards.

[0097] In this embodiment, the rank calculation unit 208 ranks the metadata 105 based on the user-oriented data 206 and the reliability score. In this way, it is possible to generate more accurate metadata 105 from the ranking results, and it is possible to reduce the effort required to manually correct the metadata 105 afterwards.

[0098] The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, the elements described in parallel in the present embodiment may be configured such that at least one of the elements is connected in series to the other elements. [Industrial Applicability]

[0099] The present invention can be applied to, for example, a data creation device related to a technology for generating metadata from unstructured documents using a large-scale language model. [Explanation of symbols]

[0100] 106...Node extraction unit, 107...Metadata token verification unit, 108...Token match score calculation unit, 110...Metadata correction unit, 113...Grouping unit, 114...Metadata reliability score calculation unit, 115...Field-specific metadata reliability score calculation unit, 208...Rank calculation unit, 1300...Data creation server

Claims

1. a node extraction unit that extracts nodes on a trace path of a dependency parse tree based on an input document; a token verification unit that performs token verification to verify whether the metadata is correct by using metadata generated by inputting the document into a large-scale language model and tokens obtained from the node; a token match score calculation unit that obtains a token match score based on a result of the token verification; a grouping unit that uses the metadata as is when the token match score satisfies a predetermined condition, and groups the metadata according to user preference data that reflects user preferences; a reliability score calculation unit that calculates reliability scores for the grouped metadata; a rank calculation unit that ranks the metadata according to the reliability score; a review unit that displays the ranked metadata together with the rank on a rank display screen; A data creation device comprising:

2. The metadata includes fields: The reliability score calculation unit a metadata reliability score calculation unit that calculates the reliability score for the entire metadata; a field-specific metadata reliability score calculation unit that calculates the reliability score for each field of the metadata; 2. The data creation device according to claim 1, further comprising:

3. a metadata correction unit that corrects the metadata when the token match score does not satisfy a predetermined condition; 2. The data creation device according to claim 1.

4. The metadata includes a combination of a key representing an attribute of each field and a value of the attribute.

2. The data creation device according to claim 1.

5. The metadata correction unit correcting the metadata based on a trace path of the dependency parse tree and each node on the trace path; 4. The data creation device according to claim 3.

6. The node extraction unit Extracting nodes on a trace path of the dependency parse tree based on a matching result of the parts of speech of the metadata.

2. The data creation device according to claim 1.

7. The rank calculation unit Ranking the metadata based on the user-oriented data and the reliability score.

2. The data creation device according to claim 1.

8. a node extraction step in which a node extraction unit extracts nodes on a trace path of a dependency parse tree based on an input document; a token verification step in which a token verification unit performs token verification to verify whether the metadata is correct by using metadata generated by inputting the document into a large-scale language model and tokens obtained from the node; a token match score calculation step in which a token match score calculation unit obtains a token match score based on a result of the token verification; a grouping step in which a grouping unit uses the metadata as is when the token match score satisfies a predetermined condition, and groups the metadata according to user preference data that reflects user preferences; a reliability score calculation step in which a reliability score calculation unit calculates reliability scores for the grouped metadata; a rank calculation step in which a rank calculation unit ranks the metadata according to the reliability score; a display step of displaying the ranked metadata together with the rank on a rank display screen; A data creation method comprising:

Images