Multidimensional automated classification and quality assurance system for nuclear medicine clinical trial data

Abstract

This invention provides an information processing system and method for automatically classifying clinical trial information in the field of nuclear medicine in multiple dimensions and ensuring its quality. [Solution] The method accepts stochastic fluctuations as an essential characteristic and designs a cross-sectional testing mechanism that guarantees quality based on the premise of fluctuations. Clinical trial texts are divided into multiple blocks and classified using LLM. Then, cross-sectional testing is performed on two axes and four patterns of verification targets (contradiction detection and RI judgment) and comparison criteria (source matching type and internal consistency type). Row shifts are detected and corrected, and contradictions and hallucinations are presented in HITL. An RI judgment agreement rate of 98.9% (100.0% with confidence level 5) was achieved for 3,000 cases, with zero cases of fabrication direction reversal.

Description

Technical Field

[0001] The present invention relates to a data processing technology in the medical and life science fields using large language models, and in particular to an information processing system and method that automatically multidimensionally classifies clinical trial information in the nuclear medicine field and guarantees the quality thereof.

[0002] More specifically, the present invention relates to a system provided with a quality assurance mechanism that performs multidimensional classification on large-scale trial description texts acquired from public clinical trial databases such as ClinicalTrials.gov using natural language inference by a large language model, detects and corrects row misalignment occurring in data processing by the large language model through a cross-verification test, detects contradictions and hallucinations, and presents them to an operator or HITL (Human-in-the-Loop, a mechanism incorporating human confirmation and judgment). Background Art

[0003] 580,880 clinical trials are registered in ClinicalTrials.gov (operated by the National Institutes of Health of the United States) (as of April 2026). Regarding the proportion of trials in the nuclear medicine and radiopharmaceutical field, a preliminary estimation by the inventor of the present invention expects it to be about several percent of all registered trials, but the calculation of the exact ratio is currently under verification and is not a fixed value. In any case, there exists no standard method for systematically identifying and classifying trials in this field.

[0004] Conventionally, the following methods have been used for such classification of large-scale data. (1) Extraction by keyword search A method of individually searching a database for keywords such as 177Lu, 68Ga, PSMA, FAPI, and manually collecting corresponding trials. However, in the implementation process of the present invention, as a result of quantitatively verifying the limitations of this method, the following problems have been confirmed. (Search omissions) Even after comprehensively searching all notational variants of the above four keywords, 1,199 out of 2,808 tests (42.7%) that were determined to be nuclear medicine tests according to this invention were missed. The main reason for these omissions is that there are many tests that use radionuclides not included in the above four keywords, such as 18F series (391 cases), 89Zr series (123 cases), 123I series (81 cases), 64Cu series (46 cases), and 11C series (39 cases). The simple total number of hits for the four (duplicate) keywords was 2,325, while after removing duplicates it was 1,685, meaning 27.5% were duplicates. The duplication was particularly significant for 68Ga and PSMA (250 hits) and 177Lu and PSMA (161 hits), indicating that keyword-based count management does not capture the true extent of the overlap. (False positive) Sixty studies were identified that hit using the keyword PSMA but did not receive a radioisotope (RI) determination. These studies use PSMA as a molecular target, such as PSMA-targeted CAR-T cell therapy or antibody-drug conjugates (ADCs), but do not use radioisotopes. Keyword searches cannot distinguish between these two types of studies. (Variations in notation) Even for just one nuclide, 177Lu, there are multiple notational variants such as "177Lu," "Lu-177," "lutetium-177," "177Lutetium," and "Lu 177," resulting in a large number of missed results when searching using a single notation. Variations in notation were also observed for 68Ga, such as "68Ga," "Ga-68," "gallium-68," "68Gallium," and "Ga 68." Furthermore, there was a problem in being unable to distinguish between dual uses of radioactive isotopes (e.g., 177Lu is used for beta-ray therapy and SPECT imaging). (2) Use of commercial databases Commercial databases such as Citeline (Trialtrove), Cortellis, and GlobalData provide trial information in the fields of nuclear medicine and radiopharmaceuticals for a fee. However, the implementation status of quality assurance mechanisms such as mapping individual radionuclides to their intended use (therapeutic use / imaging use / dual use), multi-context semantic identification of nuclear medicine molecular targets, hallucination detection, and alignment correction cannot be confirmed from publicly available information. Furthermore, practical challenges have been pointed out, including the annual licensing fees ranging from several million to tens of millions of yen, the difficulty in easily grasping the overall classification system from the outside, and the potential time lag between trial registration and classification completion. (3) Manual expert classification This describes a method used by individuals with expertise in nuclear medicine to manually classify and organize thousands of test data points in Excel. However, processing large datasets exceeding 3,000 entries presented intractable challenges, including decreased concentration due to prolonged repetitive work, accumulation of human errors, and fundamental limitations in processing speed.

[0005] In recent years, advancements in natural language processing techniques using large language models (LLMs) have made it possible to automatically classify unstructured text. For example, Razuvayevskaya et al. (Open Targets, 2024) published a study classifying the reasons for early termination of 28,561 trials registered on ClinicalTrials.gov using a fine-tuned BERT model (Non-Patent Literature 1). This study extends previous research (Pak et al., 2015, classification of 3,125 termination reasons) and achieves systematic classification using natural language processing with BERT. However, this method is specialized for classifying a specific item, the reason for termination, and is not applicable to multidimensional clinical trial classifications (complex judgments of treatment area, disease, modality, radionuclide, etc.). Furthermore, it lacks quality assurance mechanisms for the classification results (hallucination detection, row misalignment correction, inconsistency detection, etc.).

[0006] Furthermore, the following technical challenges are known in data processing using large-scale language models. (Problem 1) Hallucination (the phenomenon in which AI generates information that does not exist in the original source) Large-scale language models can sometimes "create" information that is not present in the training data. Furthermore, large-scale language models have limitations on the length of input they can process at once (context window limitations), making it technically difficult to process thousands of text entries simultaneously. Therefore, the technique of dividing the input text into multiple blocks for processing (block splitting, chunking) is widely adopted in the LLM industry. While the primary objectives of block splitting are to address input length limitations and improve search accuracy in RAG (Retrieval-Augmented Generation), it has also been reported to contribute to reducing hallucination. However, this technique presents new challenges. (Problem 2) Line shifts that occur in data processing using large-scale language models When using a large-scale language model (LLM) in conjunction with spreadsheet software (such as Excel) to classify large amounts of row data, a phenomenon called "row shift" can occur during the process where the LLM reads multiple rows of data at once and writes the analysis results back to the cells of each row. For example, the analysis result for row N may be written to the cell at row N+1 or row N-1, or all subsequent rows may be shifted by one row after a specific row is skipped. This row shift significantly impairs the reliability of the final classification results.

[0007] Conventional technologies lacked a quality assurance mechanism to detect and correct line misalignment that occurred during data processing using this large-scale language model. Furthermore, in the field of nuclear medicine, complex domain-specific classification rules are required due to factors such as the use of radioactive isotopes like 177Lu for both therapeutic and imaging purposes, the performance of incidental imaging (dosimetry, etc.) with therapeutic radionuclides like 225Ac·90Y, and the mention of molecular targets like PSMA in multiple contexts including prostate cancer, glioblastoma, renal cell carcinoma, and some neuroendocrine tumors. However, no automated classification system existed that implemented these rules. (4) Data verification using spreadsheet software

[0008] Traditionally, database quality assurance has involved using spreadsheet software such as Microsoft Excel to verify the consistency of data across multiple columns using functions like VLOOKUP, MATCH, and IF. For example, one method is to verify whether the data in column A and column B match using a function like "=IF(A1=B1, "OK", "NG")". This type of verification using spreadsheet software is based on deterministic rules and always returns the same output for the same input. Therefore, "row shifts" and "stochastic fluctuations" do not occur.

[0009] However, verification using spreadsheet software cannot solve the following problems of the present invention. These problems will be described in detail in the next section, "Problems the Invention Aims to Solve." [Prior art documents] [Non-patent literature]

[0010] [Non-Patent Document 1] Razuvayevskaya, O. et al., "Genetic factors associated with reasons for clinical trial stoppage", Nature Genetics, 56, 1862-1867 (2024) [Non-Patent Document 2] Anthropic, "Glossary - Claude API Docs", https: / / platform.claude.com / docs / en / about-claude / glossary (states "Even with temperature set to 0, the results will not be fully deterministic") [Non-Patent Document 3] OpenAI, "Reproducible outputs with the seed parameter", OpenAI Cookbook, https: / / cookbook.openai.com / examples / reproducible_outputs_with_the_seed_parameter (The seed parameter is described as "mostly deterministic") [Non-Patent Document 4] Google, "Content generation parameters", Vertex AI official documentation, https: / / docs.cloud.google.com / vertex-ai / generative-ai / docs / multimodal / content-generation-parameters (states "Deterministic output isn't guaranteed," and also clearly states that even with the same seed, the output will vary depending on the model and parameter changes.) [Non-Patent Document 5] Unstract, "Why is deterministic output from LLMs nearly impossible?", https: / / unstract.com / blog / understanding-why-deterministic-output-from-llms-is-nearly-impossible / (Technical explanation of nondeterminism due to the non-coupling nature of GPU floating-point arithmetic) [Non-Patent Document 6] Microsoft, "Expanding model choice in Microsoft 365 Copilot", Microsoft 365 Blog, published September 24, 2025, https: / / www.microsoft.com / en-us / microsoft-365 / blog / 2025 / 09 / 24 / expanding-model-choice-in-microsoft-365-copilot / [Non-Patent Document 7] Microsoft, "Connected AI subprocessors for Microsoft 365 Copilot", Microsoft Learn, https: / / learn.microsoft.com / en-us / copilot / microsoft-365 / connect-to-ai-subprocessor [Overview of the project] [Problems that the invention aims to solve]

[0011] This invention was made in view of the problems of the prior art described above. First, conventional verification methods using spreadsheet software (such as Excel) had the following problems: (Issue A: Contextual inconsistencies cannot be detected)

[0012] For example, the statement "177Lu-PSMA is used for PET imaging" is contradictory based on expertise in nuclear medicine (177Lu is primarily used for beta-radiation therapy and is not a PET radionuclide). However, detecting this contradiction using a spreadsheet function would require pre-defining rules for all radionuclide and modality combinations. Even considering just the major radionuclides such as 177Lu, 225Ac, 90Y, 68Ga, 18F, and 11C, there are dozens of types, and the combinations with modalities (PET, SPECT, CT, MRI, etc.) amount to hundreds. Furthermore, every time a new radionuclide appears (e.g., 161Tb, 149Tb, 47Sc), additional rules would be needed, which is not practical. (Problem B: Undetection of row misalignment occurring in data processing by LLM)

[0013] In classification processing of large-scale data in which a large language model (LLM) is linked with spreadsheet software, it has been confirmed that when the LLM writes analysis results back to cells, a phenomenon (row misalignment) occurs in which the row number of the writing destination deviates from the row number of the original data. A plurality of factors can be considered as causes of row misalignment, such as the write offset between batches when performing batch processing of a plurality of rows, constraints of the context window of the LLM, and the implementation method of the linkage environment between the spreadsheet software and the LLM. This row misalignment is difficult to detect with deterministic functions (VLOOKUP, MATCH, IF, etc.) of spreadsheet software. Even if a method is adopted in which mechanical identifiers such as NCT numbers are assigned to each block for matching, it does not function as a practical row misalignment detection mechanism due to the following implementation drawbacks. (Drawback 1) Explosive increase in the number of columns When input text is divided into blocks each having a fixed number of rows and an identifier is assigned to each block, N columns of identifiers are required to be managed for N blocks. In an embodiment of the present invention, when 6,000 rows of data are divided into 30 blocks of 200 rows each, 30 identifier columns are required, which complicates management by the spreadsheet software and increases processing load. (Drawback 2) Difficulty of detection in similar tests When similar nuclear medicine tests are arranged consecutively, row misalignment cannot be detected by simple identifier-based matching. For example, if "Prostate cancer therapy test using 225Ac-PSMA" is arranged in Row N and "Neuroendocrine tumor therapy test using 225Ac-DOTATATE" is arranged in Row N+1, both rows contain the common keywords "225Ac" and "alpha-ray therapy nuclide", therefore, even if the identifier matches the classification result, there is a possibility that consistency between the description content of the disease name, molecular target and drug name and the classification result is actually impaired. This type of "false positive matching due to keyword similarity" cannot be detected by mechanical matching of identifiers. Accordingly, detecting line misalignment requires a new mechanism that, instead of verifying identifier matching, verifies the semantic correspondence between classification results and original text through natural language inference. The cross-validation mechanism of the present invention is characterized in that it verifies this contextual correspondence through natural language inference by the large language model itself, and is designed not to rely on mechanical identifier matching. Furthermore, since the description content itself of data in the medical and life science fields has a medical semantic structure, verifying the consistency between this semantic structure and classification results enables detection of line misalignment, contradictions, and hallucinations without using mechanical identifiers. (Problem C: Stochastic fluctuations cannot be verified)

[0014] Classification results by LLMs fluctuate stochastically. For the same test description text, different results may be output: a "confidence score of 5" in the first run and a "confidence score of 3" in the second run. However, functions in spreadsheet software are deterministic and always return the same output for the same input, so such fluctuation cannot be verified. Although the magnitude of fluctuation is an important indicator for evaluating the reliability of classification results, spreadsheet software cannot handle this. (Problem D: Stochastic fluctuations cannot be eliminated in principle by prompt improvement)

[0015] It is known that systematic errors can be reduced through optimization of scripts (prompts). However, stochastic fluctuations still remain even after optimization. This is because large language models have an architecture based on stochastic token generation, and thus cannot be eliminated in principle by prompt design alone. Furthermore, when multiple quality verifications are performed on the same input, different contradictions are detected in each run. That is, missed detection occurs if only one quality verification is performed. To address this problem, a quality assurance mechanism independent of prompt optimization is indispensable. (Problem E: Limitations of fluctuation suppression via temperature parameters)

[0016] In conventional techniques, a method of setting a low temperature parameter is known as a means of suppressing fluctuations in the output of large-scale language models. By bringing the temperature parameter closer to 0, the probability distribution of token selection becomes sharper, and the variability of the output is reduced. However, all major providers of large-scale language models acknowledge in their official documentation that output nondeterminism remains even when the temperature parameter is set to 0. Anthropic (Claude) states in its official glossary that "even when the temperature is set to 0, the results are not entirely deterministic, and the same input can produce different outputs" (Non-Patent Literature 2). OpenAI (GPT) has introduced a seed parameter, but describes it as "mostly deterministic," stating that it cannot guarantee the same output due to system updates or load balancing across hardware (Non-Patent Document 3). Google (Gemini) officially states that it only supports the seed parameter as a "best-effort" setting and does not guarantee deterministic output (Non-Patent Literature 4).

[0017] These indeterminities stem from the discooperative nature of floating-point operations in GPUs ((a+b)+c ≠ a+(b+c)) and the uncertainty of thread execution order in parallel processing (Non-Patent Document 5), and are not problems specific to particular models or providers, but rather structural characteristics common to the architectures of current large-scale language models. In other words, stochastic fluctuations cannot be completely eliminated in the speed-first operating environment of commercial LLMs by adjusting temperature parameters, introducing seed parameters, optimizing prompts, switching models, or upgrading to higher-performance, more expensive models.

[0018] Furthermore, in commercial spreadsheet integration tools (such as Claude for Excel and ChatGPT for Excel), temperature and seed parameters may not be made available to the user. In other words, in many situations where large-scale language models are used for large-scale data processing in practical environments, there is no means for the user to control fluctuations. Furthermore, the "row misalignment" that this invention primarily addresses is a problem caused by a mismatch in the number of input and output rows in the block partitioning process, and is a structural issue that can occur regardless of the temperature parameter setting. (The relationship between the problem and the idea behind the invention)

[0019] Considering the above challenges A through E together, it is clear that data classification processing using large-scale language models inherently involves fundamental difficulties not found in conventional data processing. Traditional data processing transformation operations have been either irreversible or non-deterministic. For example, rounding (3.14159 → 3.14) is irreversible but deterministic, always returning the same output for the same input. Hash functions (such as SHA256) are similarly irreversible but deterministic. On the other hand, random number generation is non-deterministic, but it can be reproduced by fixing the seed value, and the original state can be managed reversibly. In contrast, data classification using large-scale language models results in both irreversibility and nondeterminism. For example, when classifying the clinical trial description text "A Phase II study evaluating 177Lu-PSMA-617 radioligand therapy in patients with metastatic castration-resistant prostate cancer with PSMA-positive lesions on 68Ga-PSMA-11 PET / CT" using a large-scale language model, one run might output "Metastatic castration-resistant prostate cancer (mCRPC)" as the disease name, while another run might output "PSMA-positive mCRPC". Both are medically correct, but they are different as text. Furthermore, it is impossible to reconstruct the original trial description text from the classification results. In other words, data classification using large-scale language models possesses properties not present in conventional data processing: "irreversible and non-deterministic transformations." Because it is irreversible, verifying the correctness of the classification results requires comparison with the original test description text, and because it is non-deterministic, it is necessary to compare the results of multiple executions for the same input. The 2x2 quality assurance schema of this invention is designed to address this "irreversible and non-deterministic transformation." Source matching types (test patterns 1 and 3) address irreversibility by verifying that the classification result is consistent with the original text. Internal consistency types (test patterns 2 and 4) address non-determinism by verifying that multiple classification results are consistent with each other. By combining both verifications, quality is guaranteed in irreversible and non-deterministic transformations. Conventional technologies have treated these fluctuations as "defects to be eliminated" and have attempted to suppress them by adjusting temperature parameters, adopting deterministic methods, and tuning models. However, as mentioned above, in the speed-prioritizing operating environment of commercial LLMs, it is impossible to completely eliminate fluctuations by any means, and conventional approaches have not led to a fundamental solution. Based on this understanding, this invention positions stochastic fluctuations not as "defects to be eliminated" but as "essential characteristics to be accepted." Furthermore, it designs an independent verification mechanism (cross-sectional verification mechanism) that guarantees quality under the premise that fluctuations exist. This shift in thinking forms the core of this invention.

[0020] Therefore, the first object of the present invention is to provide a quality assurance mechanism that detects and corrects row misalignment that occurs in data processing by a large-scale language model when automatically classifying clinical trial data in multiple dimensions using a large-scale language model.

[0021] Furthermore, the objective of this invention is to implement a set of judgment rules specific to the field of nuclear medicine (such as dual-use identification of radioisotopes and multi-context identification of molecular targets) and to provide a system that can be scalably applied to data of 16,244 cases extracted from a clinical trial database of over 580,000 cases. Currently, a contradiction rate of 3.97% has been achieved in the 3,000 cases for which quality verification has been completed, but the quality indicators for the entire 16,244 cases are still under verification.

[0022] Furthermore, an object of the present invention is to provide a novel quality assurance method for evaluating the reliability of classification results by detecting discrepancies between the inference process text output by a large-scale language model and the final classification result value. (The essential idea behind the invention)

[0023] The inventor began classifying clinical trial data using a Large-Scale Language Model (LLM) in March 2026, and after approximately four weeks of trial and error, obtained the following technical findings: Row shifts, hallucination, and stochastic fluctuations inevitably occur in data processing using a Large-Scale Language Model. This finding is consistent with the fact that Anthropic, OpenAI, and Google all acknowledge the nondeterminism of their output in their official documentation (see

[0016] to

[0017] above).

[0024] Based on this understanding, the inventor constructed the following three specific mechanisms as a system design that assumes the existence of fluctuations. (Mechanism 1: Detection and correction of row misalignment by cross-sectional verification)

[0025] Line misalignment that occurs in data processing by large-scale language models is due to the processing characteristics of LLMs and is difficult to completely prevent. This invention designs a cross-sectional verification mechanism that "detects and corrects line misalignment assuming it will occur" rather than "preventing its occurrence." Specifically, the contextual correspondence between the output of block partitioning processing and the source text is verified by the large-scale language model's own natural language inference, and if line misalignment is detected, the relevant line is reprocessed with pinpoint accuracy. (Mechanism 2: Use of stochastic fluctuations as a quality indicator)

[0026] The fact that the output of LLM fluctuates probabilistically is a fundamental difference from deterministic systems (such as Excel). As described in

[0016] to

[0018] above, this fluctuation cannot be completely eliminated in the speed-prioritizing operating environment of commercial LLM, even by adjusting the temperature parameter or introducing a seed parameter. This invention designs a mechanism that utilizes this fluctuation as a "quality indicator." Specifically, it processes the same input twice and measures the magnitude of the output fluctuation to evaluate the reliability of the classification result. If the fluctuation is small (e.g., score 5 both times), the reliability is high, and if the fluctuation is large (e.g., score 5 the first time, score 2 the second time), the reliability is low. (Mechanism 3: Contradiction detection utilizing LLM's contextual understanding capabilities)

[0027] By utilizing LLM's own contextual understanding capabilities as a tester, it achieves contradiction detection that is impossible with deterministic rules. For example, to detect inconsistencies in a statement like "177Lu-PSMA is used for PET imaging," an Excel function would require pre-defining a rule prohibiting the combination of "177Lu" and "PET." However, in this invention, inconsistencies can be detected by implicitly utilizing nuclear medicine knowledge by asking the LLM, "Is this statement inconsistent?"

[0028] The three mechanisms described above are integrated into a cross-sectional verification system with two axes and four patterns: verification target (contradiction detection / RI judgment verification) and comparison criteria (source matching type / internal consistency type). This integration creates a quality assurance mechanism with complementary detection capabilities for all of the following: discrepancies, contradictions, and hallucinations.

[0029] While the concept of quality assurance in this invention shares some common ground with the basic principles of quality control (QC), which involve recording and managing measured values ​​and their deviations, the technical feature of this invention lies in its application of these principles to the probabilistic output of a large-scale language model and its implementation as a concrete detection and correction mechanism. [Means for solving the problem]

[0030] The present invention provides a quality assurance system that integrates the three mechanisms described in

[0025] to

[0027] above, assuming the existence of stochastic fluctuations in large-scale language models (LLMs).

[0031] To solve the above problems, the present invention provides an information processing method for extracting and classifying nuclear medicine-related trials from a clinical trial database using a large-scale language model, (a) A division processing step in which the trial description text obtained from the clinical trial database is divided into multiple processing blocks and input into a large-scale language model, and classification processing by natural language inference is performed independently for each block, (b) A cross-sectional testing step in which the multiple classification results obtained in the division processing step are compared with the original test description text to detect line misalignment between blocks, inconsistencies between classification results, or hallucination by a large-scale language model, The aforementioned cross-sectional verification step is characterized by detecting line breaks, inconsistencies, and hallucinations by verifying the contextual correspondence between the classification result and the original test description text using natural language inference with a large-scale language model, and is characterized by not relying on mechanical identifier matching verification in this verification. (c) A correction step in which the row misalignment detected in the cross-sectional verification step is corrected by pinpointing and reprocessing the relevant row, (d) A presentation step to provide the inconsistencies and hallucinations detected in the cross-sectional verification step for final judgment by the operator or Human-in-the-Loop (HITL), It is characterized by including. Furthermore, if a systematic error is detected in a specific column or pattern, the prompt (script) itself can be modified. The present invention provides a detection mechanism that forms the basis of such a three-stage response strategy (pinpoint reprocessing, script modification, and HITL final decision).

[0032] Furthermore, in a preferred embodiment of the present invention, in the partitioning step, a set of decision rules specific to the field of nuclear medicine is applied to the large-scale language model, the set of decision rules including dual-use identification rules that distinguish between therapeutic and imaging uses of radioisotopes, and multi-context identification rules that identify the meaning of molecular target proteins in multiple contexts.

[0033] Furthermore, in a preferred embodiment of the present invention, the division processing step includes a Human-in-the-Loop mechanism that calculates an accuracy score indicating the degree of confidence for each classification result and automatically assigns boundary cases where the accuracy score is below a predetermined threshold to a human confirmation queue. The aforementioned Human-in-the-Loop mechanism can incorporate expert verification beyond mere human confirmation. For example, experts in the field of nuclear medicine (physicians, researchers, RI manufacturing experts, etc.) can verify borderline cases and confirm the validity of the judgment results, thereby enhancing the medical and scientific reliability of the database. This allows the present invention to achieve both the efficiency of automated processing and quality assurance by experts.

[0034] Furthermore, in a preferred embodiment of the present invention, in the cross-sectional testing step, the possibility that the classification result value is incorrect is determined by detecting a discrepancy between the text representing the inference process output by the large-scale language model and the final classification result value.

[0035] Furthermore, in a preferred embodiment of the present invention, the verification loop step further includes linking the detection results of the cross-sectional testing step to an external analysis module and generating prompt correction candidates by performing the following steps: (f-1) linking the judgment basis text and confidence score to an external analysis module; (f-2) aggregation step extracting records with confidence scores below a threshold and aggregating frequently occurring missing information from the judgment basis text; (f-3) generation step generating prompt correction candidates that complement the aggregated missing information using a large-scale language model; (f-4) re-verification step re-executing the cross-sectional testing step with a version in which the generated correction candidates are applied to the current prompt and evaluating changes in quality indicators; and (f-5) presentation step presenting the correction candidates for which quality improvement has been confirmed to the operator. This embodiment makes it possible to continuously improve the quality of the classification process.

[0036] Furthermore, in a preferred embodiment of the present invention, an information processing system is provided that comprises a division processing unit for executing the division processing step, a cross-sectional verification unit for executing the cross-sectional verification step, a correction processing unit for executing the correction step, a HITL distribution unit for executing the Human-in-the-Loop distribution step, and an external analysis module for executing the verification loop step, wherein each of these components works in cooperation with the detection result of the cross-sectional verification step as a common input, thereby providing an integrated workflow that consistently executes everything from classification processing to quality improvement. [Effects of the Invention]

[0037] The present invention provides the following effects. (Effect 1) Realization of large-scale data processing By applying the present invention, it is scalable to a clinical trial database of over 580,000 entries. In the example, of the 16,244 candidate nuclear medicine-related data extracted by a nuclear medicine-related term search, 3,000 were tested, and 2,809 were identified as nuclear medicine-related trials. This scale of processing is practically impossible with manual human work, and maintaining consistent quality is extremely difficult.

[0038] (Effect 2) High-precision quality assurance The cross-sectional verification mechanism of the present invention can detect and correct row misalignments that occur in data processing using large-scale language models. In the example, quality verification was performed on 3,000 data points, and the result was that the inconsistency rate for verification pattern 1 (inconsistency detection and source matching type) was 3.97% (119 cases), and the inconsistency rate for verification pattern 2 (inconsistency detection and internal consistency type) was 9.10%. The majority of the inconsistencies detected were interdisciplinary misclassifications of TA (Therapeutic Areas), indicating that the cross-sectional verification appropriately detects these boundary cases.

[0039] (Effect 3) Implementation of rules specific to nuclear medicine It is possible to accurately identify the dual use of radioisotopes such as 177Lu·225Ac and distinguish the multi-contextual meaning of molecular targets such as PSMA. In the examples, the agreement rate for RI (radioisotope use) / Non-RI determination was achieved at 98.9% (3,000 reproducibility tests). For data with clear nuclear medicine information (RI-I confidence level 5, 1,824 cases), the recall rate was 100.0%, and even with confidence level 3 or higher (2,298 cases), a recall rate of 100.0% was achieved. Furthermore, analysis of the direction of 34 cases of RI determination discrepancies revealed that 11 cases were YES→NO (oversight direction), 23 cases were not sure→NO, and there were 0 cases of reversal from NO→YES (direction of fabricating non-RI use into RI use).

[0040] (Effect 5) Demonstration of a three-layer stable structure This invention demonstrates that the output of a large-scale language model exhibits different levels of stability across its three-layer structure. The first layer (conclusion: YES / NO judgment) supports the reliability of the database with a recall rate of 87.5-98.9%, achieving a recall rate of 100.0% for data with clear nuclear medicine information (RI-I confidence level 5). The second layer (confidence score) shows a perfect agreement rate of 76.0-94.7% and an agreement rate within ±1 of 94.7-99.2%, providing a basis for difficulty-adaptive routing. The third layer (inference process text) follows different inference paths in each trial, but maintains a high level of agreement in conclusions. This three-layer structure provides design guidelines for "what should be verified" in cross-sectional testing, enabling efficient quality assurance by prioritizing verification of the first layer and performing reprocessing only when the variation in the second layer is large.

[0041] (Effect 6) Addressing residual fluctuations that cannot be resolved by prompt optimization. (Effect 7) LLM provider-independent quality assurance

[0042] The cross-sectional validation mechanism of the present invention is a general-purpose quality assurance method that is not dependent on a specific LLM provider. In 200 comparative tests of four LLM providers (Claude, Gemini paid / free, Copilot, ChatGPT) under five conditions, it was confirmed that all companies agreed for clear data (75 out of 75 perfect agreements), while for ambiguous data, the five companies were dispersed in five directions. This result demonstrates that the cross-sectional validation mechanism functions as a quality assurance mechanism regardless of the LLM provider used. We confirmed that systematic errors can be significantly reduced by optimizing the script (prompt) (Comparative example: 49.1% → Example: 90.9%). However, 9-13% of stochastic fluctuations still remain after optimization, and since this is due to the stochastic token generation of the large-scale language model, it cannot be eliminated in principle by prompt design alone. The cross-sectional verification mechanism of the present invention addresses these residual fluctuations and is a quality assurance means independent of prompt optimization.

[0043] (Effect 4) Reliability evaluation by inference and output deviation detection By detecting the discrepancy between the reasoning process and the final output of a large-scale language model, we have realized a novel quality assurance method that is not available in conventional technologies. [Brief explanation of the drawing]

[0044] [Figure 1] This is a block diagram showing the overall configuration of the information processing system of the present invention. [Figure 2] This flowchart shows the processing flow of the present invention. [Figure 3] This diagram shows the specific judgment rule structure of nuclear medicine. [Figure 4] This diagram shows the structure of the four quality assurance schema patterns. [Modes for carrying out the invention]

[0045] Embodiments of the present invention will be described in detail below with reference to the drawings. [Examples]

[0046] System Configuration Figure 1 is a block diagram showing the overall configuration of the information processing system of the present invention. This system comprises the following components. (1) Clinical trial database input section Obtain the test description text from publicly available databases such as ClinicalTrials.gov. (2) Block division processing unit The acquired test description text is divided into multiple blocks suitable for processing by a large-scale language model. The division is performed considering the amount of data to be processed and the characteristics of the classification items. (3) Natural language inference engine For each block, classification processing is performed using natural language inference with a large-scale language model. (4) Nuclear Medicine Judgment Rules Database This database stores a set of judgment rules describing the characteristics of radioactive isotopes such as 177Lu, 225Ac, 90Y, and 161Tb, as well as the characteristics of molecular targets such as PSMA and FAPI. (5) Cross-sectional testing organization Multiple classification results are compared with the original test description text to detect line shifts, inconsistencies, and hallucinations. Depending on the detection results, processing is branched as follows: Classification results for which no problems were detected proceed to the final output. If line shifts are detected, the results are sent to (6) the correction processing unit; if inconsistencies, hallucinations, or low confidence levels are detected, the results are sent to (7) the Human-in-the-Loop processing unit. (6) Correction Processing Unit The detected row misalignment is corrected mechanically by precisely reprocessing the affected row. After the correction, (5) the cross-sectional verification mechanism performs a re-verification to confirm that the problem has been resolved before proceeding to the final output. (7) Human-in-the-Loop Processing Unit Classification results that detect contradictions or hallucinations, as well as borderline cases with confidence scores below a predetermined threshold, are assigned to a human review queue for final judgment by the operator. [Examples]

[0047] Implementation of nuclear medicine-specific judgment rules Figure 3 shows the judgment rule structure specific to nuclear medicine. The present invention implements the following judgment rules. (Rule 1) Dual-use identification of 177Lu 177Lu has both beta-ray therapeutic applications and SPECT imaging applications. Based on the context of the test description text, determine whether the primary purpose is therapeutic or imaging. Example: "Treatment of metastatic prostate cancer with 177Lu-PSMA-617" → Therapeutic use Example: "SPECT diagnosis (dosimetry) using 177Lu-labeled PSMA ligand" → Imaging applications (Rule 2) Alpha-based therapy with 225Ac 225Ac is an alpha-emitting radionuclide (approximately 5.8 MeV) that emits a total of four alpha rays in its decay chain. Its primary clinical use is targeted alpha therapy. While 225Ac itself does not emit gamma rays, its daughter radionuclides (221Fr, 213Bi, etc.) do, making it possible to perform research-based SPECT imaging for dosimetry after treatment. Therefore, studies mentioning 225Ac should be classified as therapeutic applications, and if the context includes mention of dosimetry or post-treatment SPECT, it should be considered imaging incidental to treatment. Classifying it as an independent imaging radionuclide for diagnostic purposes is inappropriate. (Rule 3) Beta-ray therapy at 90Y 90Y is a pure beta-emitting radionuclide (maximum 2.28 MeV) and does not emit significant gamma rays. Its primary clinical use is therapeutic (e.g., radioembolization). However, SPECT imaging utilizing bremsstrahlung (Blemsstrahlung) generated by the interaction of beta rays with tissue, and 90Y-PET / CT using extremely low-probability internal pair production, are clinically performed to confirm post-treatment distribution and dose evaluation. Therefore, studies mentioning 90Y should be classified as therapeutic applications, and if Bremsstrahlung SPECT and 90Y-PET are mentioned, they should be accepted as imaging incidental to treatment. Classifying 90Y as an imaging radionuclide solely for diagnostic purposes is inappropriate. (Rule 4) Multi-context identification of PSMA PSMA is a prostate-specific membrane antigen, but it is mentioned in various contexts other than prostate cancer, such as glioblastoma, renal cell carcinoma, and some neuroendocrine tumors. Accurately extract the target disease from the examination description text. [Examples]

[0048] Block splitting and row misalignment detection Figure 2 is a flowchart showing the processing flow of the present invention. (Step S1) Obtain the trial description text from the clinical trial database. (Step S2) Divide the exam description text into multiple blocks. (Step S3) Perform classification processing using a large-scale language model for each block. (Step S4) Integrate the classification results of each block. (Step S5) Compare the integrated classification results with the original exam description text (cross-sectional comparison) (Step S6) Detect misalignment or inconsistency (Step S7) For line misalignment, reprocess the relevant line precisely to correct it. Report any inconsistencies or hallucinations to the operator or HITL. (Step S8) Output the corrected classification results.

[0049] This section explains how to detect line misalignment. For example, if 1,000 test data points are divided into 5 blocks of 200 each for processing, the processing results for each block will be obtained as follows. Block 1: Classification results for tests 1-200 (200 rows) Block 2: Classification results for tests 201-400 (200 rows) Block 3: Classification results for tests 401-600 (200 lines) Block 4: Classification results for tests 601-800 (200 lines) Block 5: Classification results for tests 801-1000 (200 rows) In this case, if, for example, the processing result of block 3 only outputs 199 lines during the processing of a large-scale language model, a line shift will occur. The cross-sectional verification mechanism verifies the contextual correspondence between the classification results for each line (disease name, treatment modality, radionuclide used, molecular target, confidence score, etc.) and the original examination description text (Study Title, Brief Summary, Interventions, etc.) using natural language inference by a large-scale language model. Specifically, it determines whether the combination of disease name, radionuclide, and modality output as a classification result is semantically consistent with the description in the original text of that line. If a line misalignment occurs, the classification result will not be contextually consistent with the original text of the line it is misaligned in, and this misalignment will be detected. A key feature of this verification method is that it does not rely on mechanical identifier matching. As mentioned above (

[0013] ), identifier-based matching methods have drawbacks such as "explosive increase in the number of columns" and "difficulty in detection in similar tests," but the cross-sectional verification mechanism of the present invention avoids these drawbacks by verifying semantic correspondences using natural language inference.

[0050] The detected row misalignment will be corrected using the following procedure. (Correction Procedure 1) Identify the row where the row misalignment occurred. (Correction Procedure 2) Reprocess the relevant row precisely. (Correction Procedure 3) Re-verify the reprocessing results against the original test description text. (Correction Procedure 4) Repeat correction procedures 2-3 until the line misalignment is resolved. Line misalignment is a technical error in the processing of large-scale language models and can be mechanically corrected by pinpoint reprocessing. On the other hand, inconsistencies may be inherent in the source text itself, and hallucination is a discrepancy between the source and the AI ​​output; therefore, neither should be mechanically corrected, but rather the operator's judgment or a final judgment by Human-in-the-Loop (HITL) is appropriate. This invention is based on a design philosophy that distinguishes and detects these three types of defects, with a division of labor: automatic correction of line misalignment and presentation of inconsistencies and hallucination to the operator or HITL. [Examples]

[0051] A three-stage response strategy to detected defects For defects detected by the cross-sectional inspection of this invention, the following three-stage response strategies can be adopted depending on the nature of the defect. (Response 1) Pinpoint reprocessing Line misalignment is a technical error in the processing of large-scale language models and occurs sporadically and randomly. In this case, the affected line is identified and reprocessed precisely. This solution has already been implemented and can be corrected mechanically. (Solution 2) Script modification If systematic errors occur in a specific column or pattern, consider modifying the prompt (script) itself. For example, if reproducibility is significantly reduced for a particular nuclide, add a judgment rule for that nuclide to the prompt. This action should be taken at the operator's discretion, and the impact on quality should be evaluated by validation before and after the modification (see Example 19). (Response 3) Human-in-the-Loop (HITL) Final Decision If the original text itself contains inconsistencies (such as ambiguity inherent in the original text or boundary determination in interdisciplinary examinations), or if the boundary between hallucination and the original expression is unclear, the final judgment will be based on HITL. This invention provides a detection mechanism that forms the basis of these three-stage response strategies, with the decision to implement each response being left to the operator. The design philosophy of dividing roles, where line misalignment can be corrected mechanically, while contradictions and hallucination require judgment by the operator or HITL, functions as a practical quality assurance system that takes into account the probabilistic characteristics of large-scale language models. [Examples]

[0052] Prompt correction candidate generation mechanism using an external verification loop This embodiment demonstrates a verification loop type prompt correction candidate generation mechanism that links the detection results of a cross-sectional verification mechanism to an external module. This embodiment is described as a design concept, and its implementability has been confirmed by the inventors. [architecture] The main processing module (a cross-sectional testing mechanism operating on spreadsheet software) outputs the judgment justification text and confidence score to an external format such as CSV. The external analysis module (implemented using a programming language script, etc.) takes the output data as input and performs the following processing. [Processing flow] Step 1 (Data Integration) The main processing module retrieves data including the judgment justification text and confidence score for each record. Step 2 (Data Filtering) Records with a confidence score of 3 or less are extracted. In the embodiment of the present invention, records with a confidence score of 2 have a recall rate of 99.1% (3,000 records x 2 times), and these records are identified as targets for quality improvement. Step 3 (Keyword Analysis) The system aggregates frequently occurring keywords related to "missing information that made the decision difficult" from the judgment basis text of the extracted records. In the script of this invention, if the confidence level is 2 or less, the judgment basis text explicitly includes "[Missing] Missing information that made the decision difficult," and this content is included in the aggregation. Step 4 (Generating correction candidates) Based on the aggregated keywords, a large-scale language model is used to generate rules that should be added to the current prompt. For example, if there is a frequent lack of information regarding the distinction between therapeutic and imaging applications for a particular nuclide (e.g., 177Lu), the addition of a judgment rule for that nuclide will be generated as a candidate for modification. Step 5 (Re-verification) The cross-sectional testing mechanism is rerun with the version in which the generated correction candidates are applied to the current prompt. Changes in quality indicators (inconsistency rate, RI decision agreement rate, confidence distribution, etc.) are evaluated to quantitatively determine the effect of the correction candidates. Step 6 (Operator's Decision) After re-verification, the proposed corrections, whose quality improvements have been confirmed, are presented to the operator for a decision on whether or not to adopt them. In this embodiment, the adoption of the proposed corrections is left to the operator's judgment, and full automation is not intended. However, in the future, it may be possible to automate the adoption decision by learning from the operator's decision history. [Advantages of this embodiment] The external verification loop configuration in this embodiment offers the following advantages: (Benefit 1) Advanced analytical processing can be performed without increasing the load on the main processing module. It can be implemented with loose coupling without incorporating heavy processing such as aggregation and natural language processing into spreadsheet software. (Benefit 2) Changes and improvements to the analysis method can be carried out independently of the main processing. Different analysis algorithms can be tried by simply replacing the external analysis module. (Benefit 3) It is easy to integrate with future HITL systems (Example: HITL distribution mechanism of claim 5). The external analysis module can use the same data format as the distribution to the HITL queue. (Benefit 4) It enables the realization of a continuous quality improvement cycle (verification loop). This embodiment does not generate correction candidates on a one-off basis, but rather forms a cyclical structure of generation → verification → adoption → re-detection. [Differentiation from prior art] Existing metaprompting technologies aim to improve the accuracy of general natural language tasks and modify prompts based on general metrics (accuracy, consistency, etc.). In contrast, the prompt modification candidate generation mechanism of this embodiment is novel in that it takes the detection results of a unique quality assurance mechanism called the cross-sectional verification mechanism as input. In other words, by utilizing information unique to this invention, namely "the basis for determining detected defects," it achieves the generation of sophisticated modification candidates that cannot be reached with conventional metaprompting. [Examples]

[0053] 2x2 Quality Assurance Schema (4 patterns of cross-sectional testing) Figure 4 shows four patterns of quality assurance schemas that constitute the cross-sectional verification mechanism of the present invention. In the present invention, quality is verified in the following four patterns by combining the two axes of "object to be verified" and "comparison criteria". The first axis of verification is "inconsistency detection" and "RI judgment verification." Inconsistency detection verifies whether there are any medical or logical inconsistencies in the overall AI classification results, while RI judgment verification verifies the most important judgment in the database: whether or not radioactive isotopes are used. The second axis of the comparison criteria is "source-matching type" and "internal consistency type." The source-matching type verifies the AI's classification results by comparing them with the original clinical trial description text (source), while the internal consistency type verifies multiple classification results output by the AI ​​by comparing them with each other.

[0054] (Test Pattern 1) Inconsistency Detection and Source Matching Type The 10 classification items generated by the AI ​​(treatment area, disease name, category, treatment modality, imaging tracer, treatment tracer, imaging nuclide, treatment nuclide, imaging target, and treatment target) are compared with the original clinical trial description text (trial title, trial summary, target disease, and intervention) to verify that the AI ​​is not outputting information that is not present in the original source. This testing pattern is particularly effective in detecting hallucination and misalignment.

[0055] (Test Pattern 2) Inconsistency Detection, Internal Alignment Type This study verifies whether there are any combinations of the 10 classification items generated by AI that are medically or nuclearly invalid. The original text is not referenced; the verification is completed solely based on the AI ​​output. This testing pattern is effective in detecting logical inconsistencies between AI outputs. For example, the internal consistency of the AI ​​output is verified in cases such as when the disease name is Alzheimer's disease despite the treatment area being Oncology, or when the therapeutic nuclide 177Lu is included in the imaging nuclide column.

[0056] (Test Pattern 3) RI Judgment / Original Source Verification Type The AI ​​verifies whether the six radioisotope-related items (imaging / therapy tracers, nuclides, and targets) it has identified are actually described in the original clinical trial description text (trial title, trial summary, intervention details, primary endpoint, secondary endpoint, and other endpoints). It also detects whether the AI ​​has made RI determinations that are not based on the original source.

[0057] (Test Pattern 4) RI Judgment, Internally Consistent Type The AI ​​determines six radioisotope-related items and compares them with a nuclide correspondence table (list of PET nuclides, SPECT nuclides, alpha-ray therapy nuclides, and beta-ray therapy nuclides) to determine whether or not radioisotopes (RI) were used (YES / NO). The original text is not referenced; the determination is made solely based on the AI ​​output and the nuclide correspondence table. The consistency of the RI usage determination will be verified.

[0058] The four patterns described above differ in the nature of the problems they detect: patterns 1 and 3 (source matching type) and patterns 2 and 4 (internal consistency type). The source matching type detects hallucination and line misalignment, while the internal consistency type detects logical inconsistencies and radionuclide contamination between AI outputs. Failure to use only one of these methods may result in detection errors. In fact, in 3,000 quality checks, 272 cases were identified where detection was performed using only one of the patterns. In the examples, the reproducibility of inconsistency detection was confirmed in 3,000 reproducibility tests. Furthermore, in a cross-validation of contamination-AI(Hard) and contamination-AI-easy(Easy), 272 inconsistencies were detected using only one of the scripts, quantitatively demonstrating that the combined use of multiple patterns is essential.

[0059] Furthermore, applying the above four quality assurance schema patterns to 3,000 data points yielded the following results. • Contradiction rate using test pattern 1 (contradiction detection and source comparison): 3.97% (119 out of 3,000 cases) • Contradiction rate using test pattern 2 (contradiction detection, internal consistency type): 9.10% • Agreement rate for test patterns 3 and 4 (RI judgment validation): 98.9% (100.0% for 3,000 reproducibility tests with confidence level 5 data) [Examples]

[0060] Human-in-the-Loop mechanism In embodiments of the present invention, a Human-in-the-Loop (HitL) mechanism can be provided that automatically assigns boundary cases where the confidence score is below a predetermined threshold (e.g., score 3) to a human confirmation queue. This mechanism allows us to compensate for the accuracy limitations of current large-scale language models with human expertise.

[0061] Furthermore, with future improvements in the performance of large-scale language models, it is expected that fully automated processing without the Human-in-the-Loop mechanism will become possible. This invention has a flexible design that can accommodate such technological advancements. [Examples]

[0062] Inference and output discrepancy detection A distinctive quality assurance method of the present invention is a mechanism for detecting discrepancies between the reasoning text output by a large-scale language model and the final classification result value (output). For example, if the inference process states "This test uses 177Lu," but the final output is "RI used: NO," this will be detected as a discrepancy between the inference and the output. If such a discrepancy is detected, the relevant test will be reprocessed or sent to a human review queue. [Examples]

[0063] Practical application to large-scale databases The following results were obtained by applying the present invention to the ClinicalTrials.gov database of over 580,000 clinical trials. • Database size: Over 580,000 entries (all fields) • Results extracted by nuclear medicine-related term search: 16,244 (candidates related to nuclear medicine) • Classification processing by generative AI: 3,000 out of 16,244 items have been tested. • Identified as nuclear medicine tests: 2,809 cases • Quality verified: 3,000 cases (all cases classified) + 3,000 cases x 2 (reproducibility testing) • Test pattern 1 (inconsistency detection, source comparison type): Inconsistency rate: 3.97% (119 out of 3,000 cases) • Test pattern 2 (inconsistency detection, internal consistency type): Inconsistency rate: 9.10% • Test patterns 3 and 4 (RI judgment verification): Agreement rate: 98.9% (3,000 reproducibility tests, 100.0% confidence level 5) • Reproducibility test: Agreement rate within ±1 of score: 94.7-99.2% (3,000 cases x 2 runs) • Row misalignment detection: Not detected in all data (3,000 entries x 2 trials). • RI judgment reversal patterns: There were 0 cases of reversal from NO to YES (fabrication direction). • Processing speed: Significantly faster than manual human work (quantitative measurement is a future challenge). This demonstrates that the present invention is scalable and applicable to databases of up to 580,000 records, achieving practical accuracy and speed. [Examples]

[0064] Reproducibility testing (3,000 cases x 2 runs, fixed script version) To verify the reproducibility of the cross-sectional verification mechanism of the present invention, the following tests were conducted. (Test conditions) Data: 3,000 nuclear medicine-related clinical trials extracted from ClinicalTrials.gov. Scripts: Four different test scripts are used with fixed versions. Number of executions: The same data is executed twice (Run-1, Run-2). Each Run completes processing all data within one day. The chat history of large language models was cleared between runs to prevent the results of the previous processing from affecting the next processing.

[0065] (Results: Layer 1 - Stability of conclusions) The agreement rate for RI usage determination (YES / NO / not sure) was 98.9% (2,966 / 3,000 cases). Of the 34 cases of disagreement, 11 (32.4%) were YES→NO (oversight direction), 23 (67.6%) were not sure→NO, and there were 0 cases of reversal from NO→YES (fabricating non-RI usage to RI usage direction). The agreement rate for the presence or absence of contradiction detection (no contradiction / contradiction present) was 92.6% for source-matching type (Hard) and 87.5% for internal consistency type (Easy). The agreement rate for source-matching type RI determination (RI-nonRI-hard) was 95.8%.

[0066] (Results: Layer 2 - Stability of confidence score) The perfect agreement rate for the inconsistency detection score (1-5 scale) was 94.7% for the source-matching type and 89.0% for the internal-alignment type. The agreement rate within ±1 scale was 98.7% for the source-matching type and 99.2% for the internal-alignment type. The perfect agreement rate for the RI accuracy score was 88.2% for the internal-alignment type and 76.0% for the source-matching type. The agreement rate within ±1 scale was 96.1% and 94.7%, respectively. Furthermore, the relatively low perfect match rate (76.0%) for the source-referenced verification of the RI accuracy score is due to the fact that the confidence level of LLM determination fluctuates depending on the amount of information in the source text in source-referenced verification, indicating that the density of nuclear medicine information directly affects the stability of the score. Fluctuations in confidence scores were concentrated around the boundary between score 4 (inferable from context) and score 5 (no inference required), and between score 3 (inference involving multiple sentences) and score 4. In clear cases (score 1 or score 5), fluctuations were almost zero.

[0067] (Result: Layer 3 - Stability of the inference process) While the string matching rate of the inference text varied significantly across columns, the agreement rate of the conclusions (YES / NO decisions) remained high. This quantitatively demonstrated that large-scale language models have the characteristic of deriving the same conclusions through different inference paths. In an analysis of 3,000 cases, the recall rate for data with clear nuclear medicine information (confidence level 5) was 100.0%, compared to 99.1% for confidence level 2 and 91.8% for confidence level 1, confirming that the confidence score accurately predicts the recallability of the data.

[0068] (Result: Verification of row alignment) After verifying the presence or absence of row misalignment across all 3,000 data points (2 sets), no row misalignment was detected. It was confirmed that the drug names and radionuclide names mentioned in the inference text matched the source data for the corresponding row, and that the inference content did not correspond to data in adjacent rows. [Examples]

[0069] Actual row misalignment in initial data (comparative example) To illustrate the background for the need for the cross-sectional verification mechanism of the present invention, the actual row misalignment in the initial data before script improvements and the introduction of cross-sectional verification is shown below. (Summary of initial data) We performed classification processing using a large-scale language model on 7,036 candidate nuclear medicine-related datasets extracted from ClinicalTrials.gov. At this point, cross-sectional validation mechanisms were not implemented, and script version control was not in place. (Results of line misalignment detection) Upon verifying the consistency of the nine AI classification columns with the original clinical trial description text (source), the following row discrepancies were detected. RI-T-AI column (therapeutic radionuclides): 100 out of 606 cases (16.5%) showed suspected row misalignment. Disease-AI column (disease name): 24 out of 2,735 entries (0.9%) show suspected row misalignment. target-AI column (molecular target): 11 out of 1,673 cases (0.7%) show suspected row misalignment. Therapeutic Modalities - AI column (Therapeutic Modalities): 2 out of 2,735 cases (0.1%) showed suspected row misalignment. TA-AI column (treatment area): 4 out of 2,736 cases (0.15%) showed suspected row misalignment. Furthermore, no row misalignment was detected in the tracer-I-AI, RI-I-AI, tracer-T-AI, and category-AI columns.

[0070] (Specific examples of misaligned lines) The following are examples of how misaligned lines have led to medically fatal errors. Example 1: Prostate cancer treatment radionuclides were mixed into an Alzheimer's disease trial. The source data for the row in question (Conditions column) was "Alzheimer's Disease," and the intervention was solely drug therapy. However, the RI-T-AI column contained "177Lu." 177Lu is a therapeutic radionuclide used in beta-ray radioligand therapy and is not used for Alzheimer's disease. A 177Lu-PSMA prostate cancer treatment trial exists in an adjacent row, and it is thought that data contamination occurred from there. Example 2: Multiple columns are misaligned simultaneously (the most serious case) The source data for the row in question (Conditions column) was "Alzheimer Disease." The TA-AI column was correctly classified as "Neurology," but the Disease-AI column listed "Prostate Cancer," and the RI-T-AI column listed "177Lu." In other words, although the treatment area was correct, the disease name and treatment radionuclide were mixed in from another row (an RLT trial for prostate cancer). This case demonstrates that detection is difficult with single-column matching, and that cross-sectional matching of multiple columns, known as "cross-sectional testing," is essential. Example 3: Alpha-emitting radionuclides were mixed into a Parkinson's disease trial. The source data for the row in question is from an exenatide treatment trial for "Parkinson's Disease," and the imaging column correctly listed "123I-DaTscan." However, the RI-T-AI column listed "225Ac." 225Ac is an alpha-emitting radionuclide and is not indicated for Parkinson's disease. The characteristic feature is that the imaging column is accurate, while only the treatment column is misaligned. Example 4: Therapeutic radionuclides were mixed into a study involving healthy subjects. The row in question was a study on drug distribution in the brain of healthy individuals, but "177Lu" was listed in the RI-T-AI column. It is clinically unacceptable to use therapeutic radionuclides in healthy individuals. A study on 177Lu treatment for neuroendocrine tumors was located in an adjacent row, and it was confirmed that the data had been mixed in with a shift of one or two rows.

[0071] (Consideration of patterns of misalignment) From the above verification, the following patterns of line misalignment were revealed. Pattern 1: Focus on treatment-related columns Row misalignment was mainly concentrated in the RI-T-AI column (16.5%), and hardly occurred in imaging-related columns (tracer-I-AI, RI-I-AI). This bias may be due to differences in the maturity of the initial scripts. Specifically, the script for the RI-T-AI column may have been processed with a version that did not adequately prevent row misalignment, and subsequent script improvements and the introduction of cross-sectional testing reduced the row misalignment rate to 0%. Pattern 2: Intrusion from adjacent rows Most of the observed row misalignments were due to contamination from test data one or two rows away. This is consistent with the phenomenon of row correspondence shifting near block boundaries. Pattern 3: Simultaneous row shifting of multiple columns In some rows, the Disease-AI and RI-T-AI columns were misaligned simultaneously. In this case, a single-column comparison would not detect the misalignment, making a cross-sectional comparison across multiple columns essential.

[0072] (Resolving discrepancies and introducing cross-sectional testing) After discovering the aforementioned misalignment, the inventor took the following steps to address the issue. Phase 1: Script improvement (reducing line misalignment) By optimizing the batch size for block splitting, verifying row numbers before writing, and checking for column misalignment within the batch, script improvements have significantly reduced the occurrence of row misalignment. Phase 2: Design of a cross-sectional verification mechanism (detection of row misalignment) To address the stochastic fluctuations that still persist even after script improvements, we designed a cross-sectional testing mechanism using a 2x2 quality assurance schema (test patterns 1-4). Phase 3: Verification through reproducibility testing (confirmation of quality assurance) We performed the full analysis, including cross-sectional testing, twice on 3,000 data points, confirming a RI agreement rate of 98.9% (100.0% for data with a confidence level of 5) and zero row misalignment. In other words, it was demonstrated that the initial data's row misalignment rate of 16.5% (RI-T-AI columns) was reduced to 0% through script improvements and the introduction of cross-sectional testing. [Examples]

[0073] Effectiveness of script version control (comparative example) We compared the quality of the script when the version was not fixed (comparison example: original) and when it was fixed (example example: Run-1 / Run-2). In internally consistent contradiction detection, the contradiction-free rate was as follows: • Comparative example (script not fixed): 49.1% • Example Run-1 (fixed script): 90.9% • Example Run-2 (fixed script): 94.1% Furthermore, the recall rate (the agreement rate between the two runs) was as follows: ·Comparative example vs Run-1:47.4% ·Comparative example vs Run-2:47.2% Run-1 vs Run-2 (both with fixed scripts): 87.5% These results demonstrate that version control (fixing) of scripts is a necessary condition for reproducibility. However, even with fixed scripts, 6-21% of stochastic fluctuations remain. This is due to the stochastic token generation of large-scale language models and cannot be eliminated in principle by prompt design alone. The cross-sectional verification mechanism of the present invention addresses these residual fluctuations. [Examples]

[0074] Demonstrating the necessity of using all four patterns together In the reproducibility test, we analyzed the degree of agreement between the findings for 74 cases where inconsistencies were detected in both runs. • 15 instances of different wording being used to point out issues with the same column (AI output item): • 13 instances of completely different columns (AI output items) being identified. • Number of identical comments: 0 In other words, in 72 out of 74 cases (97.3%) where a contradiction was identified, Run-1 and Run-2 detected different contradictions. This means that performing a cross-sectional test only once can lead to missed detections, demonstrating that multiple runs or the combined use of multiple patterns are essential for quality assurance. [Examples]

[0075] Incorporating an expert verification process The Human-in-the-Loop mechanism of the present invention can incorporate a verification process by experts. (Implementation method) Borderline cases or classification results with low accuracy scores can be moved to a confirmation queue by experts in the field of nuclear medicine (physicians, researchers, radioisotope manufacturing experts, clinical trial design experts, etc.). Experts can verify the validity of the classification results from the following perspectives: • Medical validity: Clinical appropriateness of diagnosis and treatment. • Scientific validity: Physicochemical properties of radioactive isotopes • Contextual validity: Is the identification of multiple meanings of the same term correct (e.g., is PSMA a target for prostate cancer, a target for CAR-T cells, or an abbreviation for a dietary study)? • Nuclear medicine validity: Is the classification of the radionuclide's use (therapeutic use / imaging use / dual use) consistent with the context of the study? • Consistency with the clinical trial design: Does the phase, endpoints, intervention content, and classification results constitute a valid structure for a clinical trial? (Expected effects) By incorporating expert verification, the following effects can be expected. 1. Improving database reliability By combining the efficiency of automated processing with the knowledge of experts, a high-quality database can be built. 2. Continuous improvement of the judging rules By incorporating expert feedback into the judgment rules, the system's accuracy can be continuously improved. 3. Think tank function By accumulating the knowledge of experts, a simple database can be developed into an intelligent information infrastructure. 4. Contribution to industry standardization A classification system based on expert consensus may be adopted as an industry standard. [Examples]

[0076] Fluctuation comparison test among multiple LLM providers (Assumptions regarding test conditions and evaluation expressions in this specification) The comparative tests of LLM providers described in this embodiment and in several subsequent comparative tests were conducted by the inventor using the standard chat interface of LLM services (paid and free versions) provided to general users, and were not measurements under a strictly controlled environment via APIs provided by each company. The test results and descriptions and evaluations of the responses of each LLM service in this specification are reports of individual cases based on the inventor's impressions as a general user, and do not generalize or evaluate the performance of each LLM service, nor are they official evaluations by each LLM service provider. Since each LLM service is continuously updated, these test results record the response to specific data at the time the test was conducted. Under these premises, in order to demonstrate the necessity of the cross-sectional verification mechanism of the present invention, repeated tests were conducted with the same script and data for LLM providers (3 paid conditions and 2 free conditions) under 5 conditions from 4 companies. (Experimental design) The AI ​​used was one of five types: Claude Pro (Anthropic), Gemini Advanced (Google, paid), Copilot Pro (Microsoft), ChatGPT Free (OpenAI, free), and Gemini Free (Google, free). For each condition, four rows of sample data (Row 4: Rheumatology / 89Zr, Row 186: Oncology / 18F, Row 1077: Neurology / 11C+123I, Row 1506: Oncology / unknown-PET) were used, and Script A (RI determination, internally consistent) and Script B (inconsistency detection, internally consistent) were run five times each, for a total of 200 tests. Each run was conducted in a new conversation session to ensure that the history from the previous run did not have any influence.

[0077] (Result 1: Consistency in clear data) For Row 4 (89Zr), Row 186 (18F), and Row 1077 (11C / 123I), where the radionuclides were specified, all 75 tests (5 tests x 5 companies) resulted in a "YES" RI determination and a confidence score of 5. In other words, it was confirmed that when specific radionuclide names are included in the input data, the conclusions are completely consistent regardless of the LLM provider.

[0078] (Result 2: Fluctuations in ambiguous data - Dispersion of judgments across 5 companies and 5 directions) For Row 1506, where the nuclide name is not specified (only the modality name "PET" is listed in the RI-I-AI column), the RI determination results from each company were as follows: • Claude Pro: "not sure" x 5 times (confidence level 2 x 5) • Gemini Advanced: "not sure" x 4 times + "YES" x 1 time (confidence level 1-2) • Copilot Pro: "not sure" x 4 times + "NO" x 1 time (confidence level 1-3) • ChatGPT Free: "not sure" x 4 times + "NO" x 1 time (confidence level 2 x 5) • Gemini Free: "YES" x 5 times (Confidence level 5 x 5) In other words, for the same ambiguous data, Gemini Free asserted "YES (confidence level 5)" five times, Copilot and ChatGPT Free fluctuated towards "NO," Claude remained stable at "not sure," and Gemini Advanced fluctuated once towards "YES." This result, with the five companies diverging in five directions, means that without a cross-sectional verification mechanism, the RI judgment results become dependent on the LLM provider, demonstrating the necessity of the quality assurance mechanism of the present invention.

[0079] (Result 3: Differences between AIs in contradiction detection sensitivity) In Row 4 (NCT04659031: PET study for inclusion body myositis), an inconsistency was found where KLRG1 was listed as the target-T-AI (therapeutic target), while both the tracer-T-AI (therapeutic tracer) and RI-T-AI (therapeutic radionuclide) were listed as N / A. The detection results for this inconsistency by each company were as follows: • Claude Pro: Detected in 3 out of 5 trials (inconsistency score 1→3→3→4→1) • ChatGPT Free: Detected 2 out of 5 times (inconsistency score 2→3→1→1→3) • GeminiAdvanced: 0 detections out of 5 trials (total contradiction score: 1) • Copilot Pro: 0 detections out of 5 trials (total contradiction score: 1) • Gemini Free: 0 detections out of 5 trials (total contradiction score: 1) In other words, only Claude and ChatGPT detected the inconsistency, and even then, their scores differed each time. Gemini and Copilot did not detect the same inconsistency at all. This result indicates that inconsistency detection can also be missed if verification relies on a single LLM, and supports the need for complementary quality assurance through cross-sectional testing of multiple patterns. [Examples]

[0080] Differences in response characteristics between paid and free versions from the same vendor. A comparison of Google's paid and free versions under identical conditions revealed a difference in response characteristics: the paid version recognized uncertainty and withheld its decision, while the free version misjudged the same data with the highest level of confidence. In the RI determination for Row 1506, Gemini Advanced determined "not sure" (confidence level 1-2) in 4 out of 5 attempts, withholding the determination because there was no specific nuclide name in the nuclide column. On the other hand, Gemini Free determined "YES" (confidence level 5) in all 5 attempts for the same data, determining that a nuclide was used based solely on the inclusion of the modality name "PET". This difference is structurally similar to the decision-making processes of nuclear medicine technologists with varying levels of clinical experience. A similar phenomenon is observed where uncertainty is perceived in context-aware judgments, while definitive judgments tend to be produced based on superficial characteristics. Similar differences in response patterns were observed among LLMs in this study as well. The important point is the observation that the difference between the "fluctuation" of the paid version (fluctuating to "YES" once out of five times) and the "stability" of the free version (same answer in all five times) indicates different response characteristics. In other words, output stability and judgment accuracy do not necessarily coincide, and a stable output does not necessarily mean agreement with the original source. This finding supports the reason why the cross-sectional verification mechanism of the present invention is designed to perform multifaceted verification of the response, rather than merely confirming the agreement rate.

[0081] Furthermore, in contradiction detection, the contradiction score for the Advanced version fluctuated from 1→2→1→4→1 with each inference, while the contradiction score for the Free version remained consistent at 4→4→4→4→4. The stability of the free version's score can be interpreted as a result of consistently detecting superficial inconsistencies, while the fluctuation of the paid version's score can be interpreted as a result of performing context-dependent evaluations. In either case, this demonstrates that a single validation cannot provide a multifaceted evaluation of the response, and that the reliability of the judgment can only be ensured through multiple cross-sectional tests. [Examples]

[0082] Guidelines for Selecting an LLM Provider in Practical Operations Based on the comparative test results of Examples 15 and 14, in actual operation, the LLM provider that showed the most stable judgment in the comparative test was adopted as the single engine, and an operational method was selected that ensures quality through a cross-sectional verification mechanism. The selection criteria are as follows: Firstly, it output "not sure" for ambiguous data (Row 1506) in all five attempts, demonstrating its characteristic of honestly reporting uncertain data as uncertain. Secondly, it achieved the highest detection rate among the five companies (3 out of 5 attempts) in contradiction detection (Row 4 KLRG1 inconsistency). Thirdly, its output of pending judgments and low confidence levels is compatible with cross-sectional testing mechanisms, allowing the testing mechanism to efficiently narrow down the "areas that need to be checked."

[0083] On the other hand, Gemini Free (Google's free version) determined "YES (confidence level 5)" for Row 1506 all five times, but source-matching verification revealed that the original text does not contain any mention of PET. In other words, output stability (perfect match) and judgment accuracy are independent indicators, and stable output does not necessarily mean a match in comparison with the original text. This finding supports the necessity of designing the cross-sectional validation mechanism of the present invention to provide multi-layered quality assurance, including not only "conclusion agreement rate" but also "multifaceted verification of responses." In practical operation, "honesty in outputting pending judgments for uncertain data" is more important than "accuracy rate" as a selection criterion for LLMs, and the cross-sectional validation mechanism plays a role in resolving these pending judgments. This operational design is not limited to the classification of nuclear medicine clinical trial data, but is a finding applicable to large-scale data processing in general using LLMs. [Examples]

[0084] Source-referenced hallucination detection and demonstration of four-pattern complementarity For Row 1506 (NCT05781217), which was used in the internal consistency test in Example 15, source-matching tests (Script C: RI determination and source-matching, Script D: inconsistency detection and source-matching) were performed 5 times each by 5 companies, for a total of 50 tests. In the source-matching tests, both the source text from ClinicalTrials.gov (Study Title, Brief Summary, Conditions, Interventions) and the AI ​​classification results were input, and the validity of the AI ​​classification results was verified based on the source text.

[0085] (Content of the original text) The trial in question (NCT05781217) was a phase III randomized trial comparing the optimal duration (6 months vs. 24 months) of salvage radiotherapy (SRT) combined with androgen deprivation therapy (ADT) for biochemical recurrence of prostate cancer. The intervention drugs were triptorelin, goserelin, and leuprorelin (all GnRH agonists, non-radioactive drugs), and the original text did not contain any nuclear medicine-related terms such as PET, SPECT, radioisotopes, or PSMA. In other words, the AI ​​classification results "category-AI: PET(n / a)", "tracer-I-AI: unknown-PET", "RI-I-AI: PET", and "target-I-AI: PSMA(PSMA PET)" are the result of the AI ​​supplementing information that is not present in the original text using medical knowledge, and thus correspond to hallucination.

[0086] (Script C result: Internal consistency type → Source verification type, RI determination corrected) The RI (Rating of Identifier) ​​determination results for internal consistency type (Script A) and source-reference type (Script C) were compared. • Claude Pro: "not sure" x 5 → "NO" x 5 (Nuclide listed in original source: Not listed, Consistency with AI classification: No basis for AI classification) Gemini Advanced: "not sure" x 4 + "YES" x 1 → "NO" x 4 + "Needs verification" x 1 Gemini Free: "YES" x 5 (Confidence level 5) → "Needs verification" x 5 (Confidence level 2) • Copilot Pro: "not sure" x 4 + "NO" x 1 → "NO" x 3 + "Needs verification" x 2 ·ChatGPT Free: "not sure" x 4 + "NO" x 1 → "Verification required" x 3 + "NO" x 2 All five companies correctly identified "Nuclides listed in the original source: Not listed." In particular, Gemini Free, which had confidently stated "YES (confidence level 5)" five times in the internal consistency test, reversed to "Needs verification (confidence level 2)" five times in the original source verification test. This indicates that the AI ​​was able to recognize the error in its own reasoning by referring to the original text.

[0087] (Script D result: Hallucination detected throughout the company) The results of the contradiction detection and source-matching type (Script D) showed a contradiction score of 5 (fatal contradiction) in all 25 trials (5 companies x 5 trials). In contrast to the internal consistency type (Script B), where the contradiction score fluctuated between 1 and 4, providing the source text as a "correct answer criterion" caused all companies' judgments to converge to the highest contradiction score. The status of hallucination label assignment was as follows: • Claude Pro: Explicitly labeled as "hallucination" in 4 out of 5 instances. • Copilot Pro: Explicitly labeled as "hallucination" in 4 out of 5 trials. • Gemini Free: Explicitly labeled as "hallucination" in 3 out of 5 tests. • Gemini Advanced: Explicitly labeled as "hallucination" in 1 out of 5 tests. • ChatGPT Free: 0 out of 5 trials (inconsistencies were detected in all trials, but the label "hallucination" was not used) In other words, while all companies agreed that a critical contradiction had been detected, each AI provider differed in whether or not they named the type of contradiction "hallucination." This, too, is a form of fluctuation in the reasoning process.

[0088] (Complete demonstration of 4-pattern complementarity) Based on the above results, the complementarity of the 2x2 quality assurance schema of the present invention (four patterns of verification target x comparison criteria) has been fully demonstrated. The internal consistency type (Script A / B) can detect logical inconsistencies between AI outputs, but it cannot detect whether the AI ​​output is based on the source text (hallucination). In fact, in Row 1506, the internal consistency type resulted in RI judgments being dispersed across 5 companies and 5 directions, and the inconsistency score fluctuated between 1 and 4. The source-matching type (script C / D) can detect discrepancies (hallucination) between the AI ​​output and the source text. In fact, in Row 1506, the source-matching type allowed all companies to correct their RI (Research Indication) judgments to NO / Requires Verification, and the inconsistency scores converged to 5 across the board. However, performing source-matching tests on all data (580,000 entries) is financially difficult. Therefore, this embodiment demonstrates that a two-stage operation—primarily screening large amounts of data using internal consistency tests, and then performing secondary verification using source-matching tests on data flagged as "not sure," "low confidence," or "inconsistent"—is the optimal design for the practical operation of a cross-sectional verification mechanism. This two-stage operational design guides AI inference in the first stage using domain-specific rules based on the inventor's expertise in nuclear medicine (such as dual-use determination of 177Lu and multi-context identification of PSMA), and in the second stage removes any remaining hallucinations by comparing them with the original text. It is a quality assurance architecture that complementarily combines human expertise and AI processing capabilities. [Examples]

[0089] Principles for Script Change Management and Quality Assurance The script (prompt) of the present invention is positioned as a quality control measure similar to a standard operating procedure (SOP) in pharmaceutical manufacturing. In pharmaceutical quality control (QC), the following operational procedures have been established based on the concept of GMP (Good Manufacturing Practice). (Principle 1) Validation obligation when changing SOPs When changing the Standard Operating Procedures (SOPs) for manufacturing processes or quality control methods, the data before and after the change is compared and validated, and the impact of the change on quality is evaluated before approving the change. In this invention, when changing the version of a script, the same data is processed with both the old and new versions, and the difference in results is quantitatively evaluated before implementing the change. (Principle 2) Management of retroactive reassessment Data judged as passing QC under the old SOP will, in principle, not be retroactively rejected under the new SOP's criteria. The old SOP was the approved procedure at the time, and the results obtained under it are valid under the quality standards of that time. In this invention as well, data classified and tested with the old version of the script will, in principle, not be retroactively re-evaluated under the new version of the script's criteria, and the new version will be applied to new data or when intentional re-verification is required. However, if the validity of the past criteria themselves changes due to domain-specific rule changes (e.g., changes in the clinical positioning of nuclides, clinical introduction of new nuclides / tracers, etc.) or updates in scientific knowledge, re-verification of past data may be carried out after recording the reason for the change and the scope of its impact. (Principle 3) Version control and record of change history All scripts are assigned a version number (e.g., category-AI v3.2, contamination-AI v2.0), and the data range processed by each version, the processing date and time, and the processing results are recorded. This allows for retrospective tracking of which version of the script processed any given data (traceability).

[0090] The difference between the Original (unfixed script: 49.1% consistency rate) and Run-1 / Run-2 (fixed script: 90.9% / 94.1% consistency rate) shown in Example 12 corresponds to the transition from a state where SOPs were not established (Original) to a state where SOPs were established (Run-1 / Run-2). The Original data is recorded as the result under the processing conditions at the time, but the Run-1 / Run-2 script-fixed version is considered the correct standard data for quality control.

[0091] These operational principles incorporate knowledge gained from the inventor's more than 30 years of experience in the field of nuclear medicine, particularly from working in radiopharmaceutical manufacturing sites (under GMP control) since 2008. They draw on the principles of quality assurance in GMP manufacturing, namely, procedure standardization through SOPs, validation during changes, and record-keeping and traceability, and apply them to data processing of large-scale language models. The idea of ​​treating prompts (scripts) as similar management targets to SOPs in large-scale data processing using LLM is an operational concept not seen in conventional prompt engineering, and is a component of the quality assurance system of this invention.

[0092] While it is technically possible to reprocess data already processed with the old script using the new script and update the results, this should be done cautiously from a GMP (Good Manufacturing Practice) perspective. If reprocessing is to be performed, it is necessary to (1) document the purpose and rationale for reprocessing, (2) compare and validate the results of processing the same data using both the old and new scripts, (3) analyze the causes of any discrepancies, and (4) approve the reprocessed data. Simply overwriting the results undermines the reliability and traceability of the data and should be avoided.

[0093] Furthermore, the large-scale language models targeted by this invention are continuously updated by their respective providers. For example, the large-scale language model used when implementing this invention may be replaced by a successor model in the future. Model updates essentially amount to a change in the very foundation of the SOP, and have a greater impact on quality than changes in the script. Therefore, when updating large-scale language models, it is essential to implement the following verification process. (1) Process the same script and data using both the old and new models, and quantitatively compare the agreement rate of the classification results. (2) The cross-sectional test (4 patterns) will be re-executed with the new model to confirm that quality indicators such as the inconsistency detection rate and RI judgment agreement rate are maintained. (3) If quality indicators change significantly, consider modifying the script (optimizing for the new model) and perform validation before and after the modification. The cross-sectional verification mechanism of the present invention also functions as a means of quality verification during such model updates. In other words, cross-sectional verification has a dual role not only as quality assurance during data processing but also as a verification tool in change management of the processing infrastructure (LLM model), and is designed as a quality assurance architecture that can accommodate the continuous development of large-scale language models.

[0094] The design concept of the cross-sectional testing mechanism of the present invention is based on the concept of boundary value determination in the quality control (QC) of radiopharmaceuticals. In the quality control of pharmaceuticals, samples that fall near the center of the specification range will pass or fail regardless of the examiner's technique (skill). The same applies to samples that clearly deviate from the specification range. The most difficult samples to judge in quality control are those that fall near the boundary between pass and fail, and it is in this region that the reliability of the testing method is put to the test. In Example 15 of the present invention, clear data (Row 4: 89Zr, Row 186: 18F, Row 1077: 11C / 123I) corresponded to the center of the specification value, and perfect agreement (YES, confidence level 5) was achieved in all 5 companies x all 5 tests = 75 tests. On the other hand, ambiguous data (Row 1506: unknown-PET) corresponded to the boundary value, and the judgments of the 5 companies were dispersed in 5 directions. In other words, it was demonstrated that the same boundary value problem as in quality testing occurs in data classification by LLM.

[0095] In quality testing of radiopharmaceuticals, retesting is performed on samples near the boundary value using the same conditions and standard, and a final judgment is made after confirming the variability of the data. The reproducibility test of the present invention (Example 10: 3,000 samples x 2 times) applies this retesting process to LLM data processing. Furthermore, just as reproducibility decreases when the examiner's technique is inappropriate, Example 12 confirmed that the classification accuracy of LLM decreases when the quality of the script (prompt) is insufficient (script not fixed: 49.1% → fixed: 90.9%). In other words, in this invention, the script corresponds to the examiner's technique, the cross-sectional test corresponds to the re-examination process, and the confidence score corresponds to an index of the variability of the measured values. This correspondence indicates that this invention draws inspiration from the concept of boundary value determination in quality control methodologies and applies it to LLM data processing.

[0096] The operational goal of the cross-sectional testing mechanism of the present invention is not to maintain the inconsistency rate below a specific threshold, but rather to enable the identification of "why the inconsistency occurred" in data where inconsistencies are detected. In the example, the inconsistency rate increased in data from row 2,000 onwards (4.4% in Run-2), but this is because the proportion of interdisciplinary trials (trials where nuclear medicine is not the primary objective but rather the evaluation tool, such as cognitive function assessment after cardiac surgery and SPECT evaluation in stroke rehabilitation) increased as the dataset expanded, and this means that the cross-sectional testing appropriately flagged this boundary data. In other words, a system that detects discrepancies in an explainable way has greater practical value in terms of quality assurance than a system that achieves a high agreement rate but leaves the cause of errors unknown. The cross-sectional verification mechanism of the present invention is designed not to "filter out" data, but to "visualize suspicious data and trace its causes," and has the function of quantitatively monitoring quality fluctuations that occur with the expansion of the dataset or changes in search conditions. This design philosophy is based on the GMP (Good Manufacturing Practice) approach to quality control of radiopharmaceuticals, which involves not simply rejecting products when they deviate from specifications (OOS), but instead conducting an OOS investigation to improve the manufacturing process. [Examples]

[0097] Correlation analysis of data characteristics and quality indicators (3,000 cases) The objective of this embodiment is to verify whether the confidence score of the present invention adaptively generates quality flags according to the characteristics of the input data (nuclear medicine information density, category specificity, therapeutic area, clinical trial phase). In other words, to quantitatively analyze whether the confidence score functions as a predictive indicator of data quality. A correlation analysis of quality indicators by data attribute was performed on 3,000 processed data points. (Analysis 1: Correlation between clarity of nuclide information and judgment stability)

[0098] A strong positive correlation was found between the confidence score of nuclide information (RI-I-AI) in AI classification and the reproducibility of RI determination (Run-1 / Run-2 agreement rate). • RI-I-AI confidence level 5 (nuclide name specified): 1,824 cases, RI determination recall rate 100.0% ·RI-I-AI confidence level 4:345 cases, RI judgment recall rate 100.0% ·RI-I-AI confidence level 3: 129 cases, RI judgment recall rate 100.0% ·RI-I-AI confidence level 2: 324 cases, RI judgment recall rate 99.1% • RI-I-AI confidence level 1 (no nuclide information): 378 cases, RI determination recall rate 91.8% The recall rate for confidence level 2 (324 cases) was 99.1%, and the recall rate for confidence level 1 (378 cases) was 91.8%. The YES rate for RI determination in data with confidence level 1 was 49.7%, indicating that LLM often makes a clear NO determination in cases where nuclear medicine information is limited. (Analysis 2: Correlation between the specificity of category classification and the stability of the judgment)

[0099] A clear correlation was also found between the specificity of the AI ​​classification category information (category-AI) and the RI judgment recall rate. ·Classification with specific nuclide (e.g. PET (18F), SPECT (99mTc), etc.): 4,174 items, RI determination recall rate 99.0% ·Unknown nuclide classification (e.g. PET (n / a), SPECT (n / a)): 1,236 cases, RI determination recall rate 66.7% ·Uncategorized (category-AI=None): 589 items, RI judgment recall rate 71.5% There was only a slight difference in recall rates between data where specific nuclide names were included in the category (2,125 cases, 99.7% recall) and data classified as having unknown nuclides (333 cases, 98.8% recall), reflecting the generally high density of nuclear medicine information within the 3,000-case range. (Analysis 3: Correlation between interdisciplinary nature of the treatment field and the rate of inconsistency detection)

[0100] Analysis of the inconsistency detection rate (percentage of inconsistency scores of 3 or higher) by treatment area (TA-AI) revealed a significant increase in the inconsistency detection rate in certain treatment areas. ·Oncology: 3,711 cases, discrepancy rate 1.8% • Neurology: 946 cases, contradiction rate 1.7% ·Cardiology: 499 cases, discrepancy rate 7.4% ·Psychiatry: 227 cases, discrepancy rate 1.3% ·Endocrinology: 134 cases, discrepancy rate 1.5% An analysis of the high inconsistency rate (7.4%) in the cardiovascular field revealed that, similar to the 3,000-case study, the majority of inconsistencies were attributed to classification boundaries in interdisciplinary trials. For example, in trials situated at the boundary between cardiology and neurology, such as cognitive function assessments after stroke and studies of cerebrovascular reactivity after cardiac surgery, the AI's classification of treatment areas fluctuated. This indicates that cross-sectional testing is appropriately flagging inconsistencies in such boundary data. (Analysis 4: Relationship between clinical trial phase and quality indicators)

[0101] The following trends were observed between the clinical trial phase (development stage) and the radioisotope recall rate. ·EARLY_PHASE1: 180 results, recall rate 100.0%, YES rate 100.0% ·PHASE1: 332 results, recall rate 99.7%, YES rate 99.4% ·PHASE1 / PHASE2 (Phase 1-2): 148 results, recall rate 99.3%, YES rate 99.3% ·PHASE2: 416 results, recall rate 99.8%, YES rate 98.6% ·PHASE2 / PHASE3 (Phase 2-3): 46 results, recall rate 100.0%, YES rate 100.0% ·PHASE3: 135 results, recall rate 100.0%, YES rate 90.4% ·PHASE4: 23 results, recall rate 100.0%, YES rate 95.7% In trials with phase information (1,280 trials), the recall rate for radioisotopes (RI) determination remained at an extremely high level of 99.3-100%. On the other hand, in trials without phase information (4,027 trials), the recall rate dropped to 84.8%. Trials without phase information were mostly included in the later data, which is consistent with the influence of nuclear medicine information density in the data shown in analyses 1 and 2. The trend of the YES rate decreasing to 90.4% in Phase 3 trials is consistent with the analysis at the 3,000-patient point, reflecting the frequent use of nuclear medicine as an adjunct evaluation tool in large-scale Phase 3 trials. (Analysis 5: The impact of changes in data composition on quality indicators - New findings)

[0102] Through the analysis of 3,000 cases, the following important insights were obtained. The 3,000 data points targeted for quality verification in this embodiment are a collection of data extracted using specific keywords such as nuclide names, PSMA, and FAPI, and therefore have a high density of nuclear medicine information. As the database expands, if data with a lower density of nuclear medicine information that can be found using general keywords such as PET and Nuclear Medicine is added, a decrease in recall is expected. These results demonstrate that the quality indicators of the present invention (confidence score, inconsistency score) function as adaptive quality flags that accurately reflect the nuclear medicine information density of the data. In other words, the confidence score not only measures the likelihood of classification results but also functions as an indicator that indirectly measures the information quality of the input data itself. This characteristic has practical value in automatically identifying areas of focus for quality control when data of different quality levels are mixed as the database expands. Furthermore, as the database expands, the proportion of data with low nuclear medicine information density may increase, potentially leading to an increase in data with confidence scores of 1-2 and data eligible for HITL (Highly Effective Tolerance) testing. This issue can be partially addressed by adding contextual inference rules based on disease name and nuclide combinations to the script. However, cases where nuclide information is not included in the source text itself represent an information quality issue with the input data and cannot be resolved solely by script improvements. The present invention's design, which appropriately assigns low confidence scores to such low-information-density data and automatically identifies them as HITL targets, functions rationally in quality control during database expansion. The adaptive behavior of the quality indicators shown in this embodiment, and how it functions as complementarity among the four test scripts, will be detailed in Example 23. The correlation analysis described above demonstrates that the cross-sectional testing mechanism of the present invention adaptively generates quality flags according to the characteristics of the data (clarity of nuclide information, interdisciplinary nature of the therapeutic area, clinical trial phase, information density of the input text), and is designed to allow for the identification of the cause when quality degradation occurs. This "cause-traceable quality assurance" is a fundamentally different approach from mere agreement rate thresholding and is an essential function for continuous quality control of a database of 570,000 records. [Examples]

[0103] The relationship between multi-model routing and quality assurance During the implementation of the present invention, when the script of the present invention was executed in Microsoft's Copilot for Excel environment, it was discovered through the AI's self-reporting that the backend LLM of the environment was Anthropic's Claude model, not OpenAI's GPT model. Microsoft has officially announced that Microsoft 365 Copilot employs a multi-model architecture that uses both OpenAI's GPT model and Anthropic's Claude model, and that it is designed to dynamically switch backend models depending on the type and situation of the task (Non-Patent Literature 6). Furthermore, the official Microsoft Learn documentation (Non-Patent Literature 7) also states that Anthropic operates as a subprocessor for Microsoft 365, and that the Claude model may be used in Excel Agent Mode and Researcher.

[0104] This fact further strengthens the need for the cross-sectional validation mechanism of the present invention. In other words, even if a user selects "Copilot" and performs data processing, the user cannot control whether the LLM actually running in the backend is GPT or Claude. Furthermore, even within the same session, the backend model may switch depending on the type of task. Therefore, quality control through the selection of an LLM provider has inherent limitations, and a cross-functional verification mechanism that verifies quality on the output side is the only quality assurance means that does not depend on the selection of an LLM. The present invention is designed to ensure traceability of processing results (tracking which model processed which row) by including a self-declaration of the "name of the AI ​​engine used" in the script. This finding demonstrates that the present invention is a general-purpose system that does not depend on a specific LLM provider and can provide quality assurance in any LLM execution environment, including multi-model environments. [Examples]

[0105] Comparative study of classification accuracy and row misalignment in Excel integration environments of three companies. To verify the necessity of the cross-sectional verification mechanism of the present invention in an Excel-linked environment, disease name classification (Disease-AI) was performed using the same data (300 rows) and the same script on three LLM-linked spreadsheet environments (Claude for Excel, Copilot for Excel, ChatGPT for Excel), and the results were compared with the original text (Study Title, Brief Summary, Conditions, Interventions). (Test conditions) • Claude for Excel: Backend Model Self-Reporting "Claude 4 Sonnet" • Copilot for Excel: Backend model self-reporting "Claude 3.5 Sonnet" (Anthropic models were used due to Microsoft's multi-model routing) • ChatGPT for Excel: Backend model self-reporting "OpenAI o3" (during Disease-AI processing) and "GPT-4o" (during RI-T-AI processing, when the model switched within the same session)

[0106] (Result 1: Classification of agreement rate and discrepancies among three companies) After comparing 300 lines of the original text, the discrepancies were classified into the following four categories. ·3 companies exact match: 100 lines (33.4%) • Granularity difference (all correct answers company-wide, but varying levels of detail): 125 lines (41.8%) • Partially correct (all three companies reflected parts of the original source, but the extracted ranges differed): 31 lines (10.4%) • Suspected misclassification (one or more companies do not match the original source): 41 lines (13.7%) • Suspected line misalignment: 2 lines (0.7%) While the string exact match rate was only 33.4%, including 125 lines with granularity differences and 31 lines with partial correctness, the company as a whole correctly reflected the content of the original text in 256 lines (85.6%).

[0107] (Result 2: Verification of row misalignment in Claude for Excel by comparing with the original source) Of the 41 lines suspected of being misclassified, several lines were found in Claude for Excel where disease names completely unrelated to the original text were entered. For example, for rows where the source condition column is "Alzheimer's Disease," Claude for Excel entered "Prostate Cancer" (Copilot and ChatGPT both correctly entered "Alzheimer's disease"). Similarly, for rows where the source is "Parkinson Disease," Claude for Excel entered "Advanced Solid Tumor." These erroneous entries matched information from other tests (prostate cancer test, solid tumor test) listed in adjacent lines, and were determined to be caused by line shifts (write offsets) in the data processing of the large-scale language model. No identical line shift patterns were observed in Copilot or ChatGPT. These results indicate that the pattern of row misalignment depends on the implementation method of the LLM provider and the Excel integration environment. This supports the necessity of cross-sectional verification of output, as row misalignment may not occur in one LLM but may occur in another.

[0108] (Result 3: Mechanism of particle size difference generation) By comparing source texts with a granularity difference of 125 lines, it was found that each LLM prioritizes referencing different columns of the source text. • Claude for Excel: Tendency to concisely list the first disease name in the Conditions column (e.g., "Alzheimer's Disease") • Copilot for Excel: Tendency to add subtypes to the primary disease in the Conditions column (e.g., "Alzheimer's disease / MCI") • ChatGPT for Excel: Tends to comprehensively list multiple diseases in the Conditions column and further infer and add disease subtypes from the Brief Summary column (e.g., "Castration-resistant prostate cancer" - the original Conditions column only lists "Prostate Cancer," but this is inferred from the Brief Summary column). In other words, even when applying the same script to the same data, differences in LLM reading strategies (which columns to read and to what extent) result in variations in the granularity of the output. While these differences are not errors, they indicate that cross-sectional testing is necessary for quality assurance to maintain database consistency.

[0109] (Result 4: Verification of actual data for multi-model routing) In ChatGPT for Excel, the model self-declaration during Disease-AI processing was "OpenAI o3," and the model self-declaration during RI-T-AI processing was "GPT-4o," confirming that the backend model dynamically switches depending on the column (task) being processed within the same workbook. Furthermore, it was consistently confirmed across all 300 rows that the backend for Copilot for Excel is "Claude 3.5 Sonnet" (a different version from "Claude 4 Sonnet" used in the main Claude for Excel application). These results demonstrate, with real-world data, that even if a user selects an LLM provider, they cannot control the version of the model that actually works or the switching between tasks. This supports the validity of the design of ensuring traceability by including model self-declaration in the script of the present invention. [Examples]

[0110] Verification of the complementarity of cross-sectional testing using 3,000 script cross-validation. The objective of this embodiment is to demonstrate, through script cross-validation of all 3,000 data points, that the 2x2 quality assurance schema (internal consistency type and source matching type) of the present invention functions complementaryly and can detect quality problems that cannot be detected by either one alone. While Example 20 analyzes the correlation between data characteristics and quality indicators, this embodiment verifies the complementarity of the cross-validation mechanism itself through cross-validation of four test scripts. Four validation scripts (RI-nonRI: internally consistent, RI-nonRI-hard: source matching, contamination-AI: source matching, contamination-AI-easy: internally consistent) were run on all 3,000 data points, and cross-validation was performed between the scripts. The reference columns for each script are as follows. RI-nonRI (internally harmonized): Only the six AI generation columns (tracer-I / T-AI, RI-I / T-AI, target-I / T-AI) are referenced, and the presence or absence of radioisotopes (RI) is determined based on the nuclide correspondence table. The original text is not referenced. RI-nonRI-hard (Source-matching type): In addition to the 6 columns generated by AI, the system refers to 6 columns from the source text (Study Title, Brief Summary, Interventions, Primary / Secondary / Other Outcome) to verify whether the AI ​​classification content is included in the context of the source text. contradiction-AI (source matching type): It matches four columns of source material (Study Title, Brief Summary, Conditions, Interventions) with ten columns generated by AI and makes a comprehensive judgment on whether the combination is medically and nuclear medicine-valid using five axes (disease x treatment method, tracer x disease, nuclide x use, target x disease, modality x overall context). Contradiction-AI-easy (internal consistency type): Refers only to the 10 columns generated by AI and verifies internal consistency using the same 5 axes. The original text is not referenced.

[0111] (Result 1: Complementarity between internally consistent and source-referencing types) A cross-tabulation of contradiction-AI and contradiction-AI-easy revealed that 59 contradictions were detected using only the source-matching method, 213 contradictions were detected using only the internal consistency method, and 60 contradictions were detected using both methods. The total number of contradictions detected using only one of the methods was 272, quantitatively demonstrating that detection failures can occur on a scale of 3,000 cases if only one of the scripts is used.

[0112] (Result 2: Cross-testing of RI-nonRI and RI-nonRI-hard) Cross-tabulation of RI-nonRI (referring only to the AI ​​column) and RI-nonRI-hard (referring to the original 6 columns) revealed the most noteworthy inconsistency pattern, with 183 cases where RI-nonRI=NO (AI determined no radioisotopes were used) and RI-nonRI-hard=YES (the original text indicates the use of radioisotopes). These cases represent situations where the upstream AI classification script did not include radioisotope-related information in the AI ​​column, but the original text contains radionuclide information. This is a quality issue that is, in principle, undetectable by internal consistency verification alone.

[0113] (Result 3: Correlation between the level of nuclear medicine information in the original text and the confidence level of radioisotopes) The presence or absence of nuclear medicine-related terms in the source text was classified into levels (specific nuclide names, drug names, tracer names, modality names only, no keywords), and the correlation with RI-nonRI confidence scores was analyzed. The results showed that the YES rate was 99.9% (2,678 out of 2,681 cases) when specific nuclide names were listed in the source text. On the other hand, even when nuclear medicine-related terms were completely absent from the source text, the YES rate was 74.2% (1,005 out of 1,355 cases). The latter cases were those where RI-nonRI gave a YES judgment because nuclide information was included in the AI ​​classification column, indicating that its validity needs to be verified through cross-validation with the source-matching type (RI-hard).

[0114] (Result 4: Correct NO determination in PSMA content test) A detailed examination of 59 studies containing the PSMA keyword that were determined to be RI-nonRI=NO revealed that all of them were non-radioactive therapies, including CAR-T cell therapy (26 studies), antibody-drug conjugates (ADCs), vaccine therapy, and CAR-NK therapy. It was confirmed that the RI-nonRI script correctly determined to be NO for studies that use PSMA as a molecular target but do not use radioisotopes. This result demonstrates that the nuclear medicine-specific determination rules of the present invention correctly perform multi-context identification of molecular targets (claim 4).

[0115] (Result 5: Variation patterns of YES rates by confidence level and causal relationship with data composition) An analysis of 3,000 RI-I-AI confidence levels revealed that the YES rates were 100.0% for confidence level 5, 100.0% for confidence level 4, 100.0% for confidence level 3, 99.7% for confidence level 2, and 49.7% for confidence level 1. To understand the reason for the low YES rate of 49.7% for confidence level 1, we analyzed the data composition. Here, "nuclear medicine-related terms" in this analysis refers to any of the following: (i) names of nuclear medicine imaging modalities (PET, SPECT, PET / CT, PET / MRI, PET-CT, etc.), (ii) names of specific radionuclides (18F, 11C, 68Ga, 177Lu, 225Ac, 90Y, 131I, 123I, 64Cu, 89Zr, 99mTc, and their hyphenated forms, etc.), or (iii) names of nuclear medicine drugs / molecular targets (FDG, FDOPA, PSMA, DOTATATE, 177Lu-PSMA-617, 177Lu-DOTATATE, etc.). | Composition | Number of responses | Ratio | YES rate | Ratio of a confidence level of 5 | |---|---|---|---|---| | Specific radionuclide names included | 1,163 entries | 38.8% | 99.2% | 99.1% | | Modality name only (PET / SPECT) | 797 results | 26.6% | 95.4% | 64.7% | | No nuclear medicine-related terms listed | 1,039 results | 34.6% | 72.5% | 68.7% | This analysis demonstrated the following important causal relationships. (Causal relationship 1) When the data includes specific nuclide names (e.g., 18F, 68Ga, 177Lu), the AI ​​makes a judgment with a confidence level of 5 (highest), and the YES judgment reaches 99.2%. (Causal relationship 2) When the data only contains the modality name (PET / SPECT) and no specific nuclide name, the AI's confidence level is distributed between 5 (64.7%) and 3 or less (32.6%), and the difficulty of determination is visualized as a decrease in confidence level. (Causal relationship 3) In tests that did not include nuclear medicine-related terminology, the AI ​​made a NO judgment with a confidence level of 5 (YES rate 67.6%), while also making a NO judgment with a confidence level of 1 (YES rate 0.0%). In other words, a confidence level of 1 indicates a case where "the AI ​​clearly determined NO." This correlation indicates that the confidence score of the cross-sectional testing mechanism functions as a quality indicator that appropriately reflects the diversity of data composition. In particular, the phenomenon of data containing only modality names being dispersed with a confidence score of 3 or less suggests that there is room for improvement in the area where the script (prompt) "determines a nuclear medicine test from modality names alone without nuclide information." This finding supports the value of the aforementioned Example 5 (prompt modification candidate generation mechanism using an external verification loop) as a diagnostic function with real data. Furthermore, this analysis revealed that data with a confidence level of 1 does not necessarily mean "cases where the AI ​​is unsure of its judgment." Data with a confidence level of 1 had a YES rate of 0.0% in both cases where only the modality name was given (8 cases) and cases where no nuclear medicine-related terminology was included (54 cases), meaning that a confidence level of 1 corresponds to "cases where the AI ​​clearly made a NO judgment." Therefore, it is suggested that the true diagnostic indicator in this invention is not the absolute value of the confidence level for a single data point, but rather the variance of the confidence level in a group of similar data points (particularly the bimodal distribution between confidence level 5 and confidence level 3 or lower). While a complete verification of this hypothesis is a future task, this perspective is important as a design guideline for the mechanism for discovering script improvement points in Example 5 (prompt correction candidate generation mechanism using an external verification loop). The inventors position the verification of this hypothesis and the quantification of the diagnostic indicator as ongoing improvement tasks for this invention.

[0116] (Result 6: Relationship between data difficulty gradient and reproducibility) Reproducibility tests involving 3,000 cases x 2 cycles revealed a clear gradient between the nuclear medicine information density of the data and reproducibility. The first half (Rows 1-2000) included many clear nuclear medicine tests such as 177Lu-PSMA and 68Ga-DOTATATE, and since the nuclide names were directly stated in the original texts, the RI-nonRI reproducibility rate was 99-100%. On the other hand, the second half (Rows 3001-5000) included many tests where the nuclide name was not stated, such as tests only described as PET scans or tests in which PSMA appeared as a target for CAR-T cell therapy, requiring LLM to infer from the context, and the reproducibility rate dropped to 70-83%. The recall rates by confidence score accurately reflected this gradient, with a recall rate of 100.0% for confidence level 5 (1,824 cases), 99.1% for confidence level 2 (324 cases), and 91.8% for confidence level 1 (378 cases). All 34 discrepancies were in the direction of YES → NO or not sure → NO, with no cases of reversal from NO → YES (fabrication direction). These results indicate that the cross-sectional validation mechanism is most valuable not for data with clear nuclear medicine information, but for data where nuclide information is ambiguous and LLM (Low-Level Malformation) determination is inconsistent. The confidence score has a function to automatically identify "which rows should be prioritized for verification," supporting the efficient operation of the cross-sectional validation mechanism. In other words, even with data that has a high confidence score for RI (radioisotope) determination, cross-disciplinary misclassification cannot be detected unless consistency with therapeutic area classification (TA-AI) and disease name (Disease-AI) is verified in a separate context. This is why the RI determination script and the contamination detection script were designed with independent reference columns.

[0117] (Result 7: Triple correlation between information density of the original text, AI confidence level, and reproducibility) The level of nuclear medicine information contained in the source texts was classified into four stages (with specific nuclide names, with tracer names, with modality names only, and without nuclear medicine-related terminology), and a triple correlation with AI confidence and reproducibility was analyzed. In the 2,629 cases where specific nuclide names (18F, 68Ga, 177Lu, etc.) were listed in the source text, the confidence level of 5 was 82.5%, and the RI-nonRI reproducibility was 100.0%. In the 734 cases where only tracer names (PSMA, FAPI, DOTATATE, etc.) were listed, the confidence level of 5 was 49.6%, and the reproducibility was 98.2%. In the 1,258 cases where only modality names were listed (PET or SPECT was mentioned, but nuclide names were not), the confidence level of 5 dropped to 39.1%, and the reproducibility to 74.1%. In the 1,378 cases where nuclear medicine-related terminology was completely absent, the confidence level of 5 was 26.9%, and the reproducibility was 79.2%. In particular, an analysis of the internal structure of 1,258 data points containing only modality names revealed that the recall rates were 100.0% for confidence level 5 (492 cases), 94.5% for confidence level 4 (200 cases), 60.3% for confidence level 3 (121 cases), and 29.1% for confidence level 2 (306 cases). This indicates that even with the same dataset containing only "PET," when the AI ​​was able to estimate the nuclide (confidence level 5), the results were perfectly reproduced, but when it was unable to estimate (confidence level 2), approximately 70% of the judgments fluctuated from trial to trial. In other words, a triple correlation was quantitatively demonstrated: the lower the information density of the original text, the lower the AI ​​confidence score and the lower the reproducibility. The cross-sectional verification mechanism provides quality assurance for such low-information-density data, automatically identifying areas for quality verification based on confidence scores. [Industrial applicability]

[0118] Although this invention is a technology demonstrated in the classification of nuclear medicine clinical trial databases, its fundamental principle, "a mechanism for detecting and correcting row misalignment that occurs in data processing using large-scale language models," has applicability to other fields that require high-precision data processing using large-scale language models.

[0119] For example, similar technical challenges (row misalignment due to block partitioning, or consistency verification across multiple data sources) may occur in the following fields, and the cross-sectional verification mechanism of the present invention may be effective. • Classification of large-scale transaction data in the financial sector • Multidimensional classification of case law data in the legal field Classification of large-scale data analysis in research fields • Multidimensional classification, sentiment analysis, topic extraction, and systematization of customer survey responses in the field of market research. • Quality assurance of supply chain data in manufacturing and logistics industries, particularly data integrity verification and system validation to meet regulatory requirements such as GMP (Good Manufacturing Practice). • Consistency verification in data transfer between different industries and organizations. This involves verifying that semantic consistency with the original data is maintained when converting and mapping data between systems with different terminology and data formats. • Other processing of large-scale data with multiple classification axes or multiple data sources.

[0120] However, specific demonstrations in these fields are outside the scope of this invention, and further research and development combined with expertise in each field are necessary. Furthermore, while the cross-sectional verification mechanism of the present invention verifies consistency in the row direction (consistency between multiple columns within a single row), quality problems of a different nature from cross-sectional verification were identified during the implementation of the present invention. Specifically, problems such as the inclusion of strings in the score column (a column where only numerical values ​​from 1 to 5 should be entered), the entry of descriptive text in a column where only nuclide names should be entered, and the output of a large number of values ​​that do not conform to the output format defined by the script were found in over 2,000 rows through visual inspection. These were found in specific columns, but considering the scale of detection by visual inspection, it is highly likely that similar problems are latent in other columns as well. This discovery revealed that the quality verification system of the present invention requires a new verification axis called "script conformance testing." Script conformance testing verifies, column by column, whether the actual output of a large-scale language model conforms to the output specifications (value range, data type, format, description rules) defined by the script (prompt) used in the classification process. This is equivalent to conformity testing of product manufacturing specifications (SOPs) in GMP manufacturing. Specifically, the system verifies that the value range, data type, and format defined by the script are consistently maintained across all rows within the same column. For example, it checks each column to ensure that the confidence score column does not contain values ​​or strings other than 1-5, and that the nuclide column does not contain any explanatory text. The verification of relationships between columns (e.g., if tracer-I-AI is "unknown-PET", then RI-I-AI should be "PET") is included in the detection scope of the existing cross-sectional test (2x2 quality assurance schema). In other words, a quality verification system in which script conformance testing (vertical: column-level output specification conformance testing) and cross-sectional testing (horizontal: contextual consistency verification within rows) function complementaryly is essential for quality assurance of data processing using large-scale language models. As a result, the quality verification system of the present invention develops into the following three-layer structure. Layer 1: Script conformance check (Vertical direction: Output specification conformance check on a column-by-column basis) Layer 2: Cross-sectional testing (horizontal direction: semantic consistency verification between multiple columns within a single row, 2x2 quality assurance schema) Layer 3: Reproducibility testing (verification of reproducibility by running the test twice under identical conditions) The first-layer script conformance test serves as a prerequisite for the second-layer cross-sectional test. This is because performing a cross-sectional test on data whose output format does not conform to the specifications does not guarantee the reliability of the test results themselves. Implementing this three-layer structure is an urgent task that should be addressed as soon as possible after filing the application. This study provides important insights gained from the discovery of numerous data processing errors in the final stages of patent application preparation, demonstrating that a systematic mechanism for verifying the suitability of output to script instructions is essential in data processing using large-scale language models. Furthermore, in the implementation of this invention, it was empirically confirmed that increasing the length and complexity of scripts (prompts) increases the frequency of hallucinations. When rules for quality verification (e.g., confirmation of compliance with output specifications equivalent to script conformance testing) are incorporated into the classification processing script, the entire script becomes longer, creating a paradox where hallucinations actually increase. From this finding, it became clear that script conformance testing should be implemented as a simple script dedicated solely to confirming compliance with output specifications, independent of the classification processing script and cross-sectional testing script. In other words, each layer of quality verification (script conformance testing, cross-sectional testing, and reproducibility testing) should be loosely coupled as independent, single-function scripts, which is a practical design principle that takes into account the probabilistic characteristics of large-scale language models. In addition, during the implementation of the present invention, it was empirically confirmed that there are two types of hallucination in large-scale language models. (Type 1: Data fabrication) Hallucination, which has been recognized before, outputs nuclide names, disease names, tracer names, etc. that do not exist in the original text. It can be detected by cross-sectional testing (especially the original source matching type). (Type 2: Rule Fabrication) Hallucination in which a large-scale language model independently generates judgment axes and criteria not defined in the script, applies them, and outputs a classification result. This has been observed almost daily in the implementation of the present invention. For example, even though the script defines four judgment axes, A, B, C, and D, there have been cases in which the large-scale language model independently generates and applies judgment axis E, and changes the classification based on the result. This type is difficult to detect by cross-sectional testing (contextual consistency verification in the row direction) because the output data itself may not be medically inconsistent. Furthermore, even with script conformance testing (value range and format check in the vertical direction), it may not be detected as long as the type and format of the output value conform to the specifications. If Type 2 hallucination affects the confidence score, the input data for the cross-sectional test itself becomes contaminated, posing a risk to the reliability of the entire quality assurance system. Fortunately, no impact on the score was observed during the implementation of this invention, but designing a mechanism to systematically detect this type of hallucination is one of the most important challenges after filing the patent application.

[0121] This invention conducted quality validation on 3,000 data points in the field of nuclear medicine, achieving an RI judgment agreement rate of 98.9% (100.0% for data with clear nuclear medicine information) and a confidence score agreement rate of 94.7-99.2% within ±1 in 3,000 x 2 reproducibility tests. In particular, it demonstrated that the confidence score accurately predicts reproducibility (confidence level 5: 100.0%, confidence level 2: 99.1%), and showed that the cross-sectional validation mechanism has the function of automatically identifying key targets for quality validation. Cross-validation of the four test scripts confirmed that 59 inconsistencies detected by source-matching alone and 213 inconsistencies detected by internal consistency alone, for a total of 272 inconsistencies, were detected by only one pattern, quantitatively demonstrating the complementarity of the 2x2 quality assurance schema. Furthermore, the correct NO (Not Accepted) classification of 62 trials containing the PSMA keyword as non-radioactive therapies such as CAR-T cell therapy and antibody-drug conjugates confirmed the effectiveness of the molecular target multi-context discrimination function of the nuclear medicine-specific judgment rule set. In the present example, the 3,000 cases subjected to quality verification are a data group with high nuclear medicine information density, and an RI determination recall of 98.9% has been achieved. Since the confidence score accurately predicts reproducibility (confidence level 5: 100.0%, confidence level 1: 91.8%), the system is designed such that even when the amount of data with low nuclear medicine information density increases with the expansion of the database, the confidence score appropriately reflects this difference in information density and automatically identifies key targets for quality verification. When the proportion of data with low nuclear medicine information density increases along with the expansion of the database, the amount of HITL target data may increase, and the system is designed such that the automatic identification mechanism based on the confidence score functions reasonably to address this issue. In addition, in 200 internal consistency comparison tests and 50 original text collation verification tests conducted on LLM providers from 4 companies under 5 conditions (3 paid conditions and 2 free conditions from Anthropic, Google, Microsoft, and OpenAI), stochastic fluctuation was confirmed across all companies, and all companies succeeded in detecting hallucinations that could not be detected by the internal consistency method through the original text collation method, thereby demonstrating the complementarity of the 2×2 quality assurance schema and the versatility of the cross-checking mechanism of the present invention. Furthermore, through a 300-row comparison test in LLM-linked spreadsheet environments from 3 companies (Claude for Excel, Copilot for Excel, ChatGPT for Excel), it was demonstrated that the occurrence pattern of row shifts differs for each LLM provider, that the output granularity depends on the reading comprehension strategy of the LLM even for the same script, and that the back-end model is switched without user control due to multi-model routing. These verification results clearly indicate the industrial applicability of the present invention. Furthermore, by incorporating an expert verification process into the Human-in-the-Loop mechanism of the present invention, it is possible to develop the system from a mere data processing system into an intelligent information infrastructure with think tank-like functions that accumulate specialized knowledge. Accordingly, wide industrial application can be expected as a high-reliability database that supports decision-making by pharmaceutical companies, CROs (contract research organizations for pharmaceutical development), radiopharmaceutical manufacturing companies, medical device manufacturers, research institutions, medical institutions, investment institutions, and other entities. (Definition of Terms) Key terms used in the present specification are defined below. | Term | Definition | |------|------| | <Core technology of the present invention> | | | Cross-Field Interpolation Validation (CFIV) | Cross-Field Integrity Validation. A quality assurance mechanism that verifies consistency across multiple data fields (columns). A core technical element of the present invention | | 2×2 Quality Assurance Schema | A framework for quality assurance consisting of four patterns based on the combination of verification target (RI determination / contradiction detection) and comparison reference (internal consistency type / original source matching type) | | Internal Consistency Type | A method that only verifies the logical consistency between columns of AI classification results, without reference to the original text. Script A (RI determination) and Script B (contradiction detection) fall into this category | | Original Source Matching Type | A method that inputs both the AI classification result and the original text (described in ClinicalTrials.gov) to verify whether the AI classification has a basis in the original text. Script C (RI determination) and Script D (contradiction detection) fall into this category | | Row Shift | A phenomenon where, in the process of a large language model reading, analyzing, and writing back data on spreadsheet software, the row number of the writing destination deviates from the row number of the original data | | Hallucination | A phenomenon where a large language model generates (fabricates) information that does not exist in the input data. Also referred to as AI fabrication | | Irreversibility | A characteristic of data classification by large language models that the original test description text cannot be restored from the classification result. If a hallucination is passed as "data" to subsequent processes, it cannot be detected without original source matching type verification | | Confidence Score | An index that quantifies the certainty of AI classification on a 5-level scale from 1 (low) to 5 (high). Equivalent to the quantification of threshold determination in quality verification | | RI Determination | Determination of whether a radioisotope is used. Determines whether any of PET nuclides, SPECT nuclides, alpha-ray therapeutic nuclides, or beta-ray therapeutic nuclides are used in a clinical trial, with the result indicated as YES (used) / NO (not used) / not sure (judgment pending) | | <Processing Mechanism> | | | Block Division Processing | A method that divides a large amount of row data into a plurality of blocks (batches) and causes a large language model to process the data sequentially. This method is adopted to address the constraint of the context window of LLM | | Script | An instruction (prompt) for a large language model. In the present invention, the script is handled as a change management target similar to GMP SOP (Standard Operating Procedure) | | Validation | Comparative verification that processes the same data before and after script modification, and quantitatively evaluates differences in results. This method applies the concept of validation for changes in GMP manufacturing | | Script Conformity Test | A quality inspection that verifies, on a column-by-column basis, whether the actual output of a large language model conforms to the output specifications (value range, data type, format, description rules) defined by the script (prompt). This inspection functions as a precondition for cross-checking | | Human-in-the-Loop (HITL) | A quality assurance mechanism that allows expert human judgment to intervene for data with a low confidence score (data near boundary values). This is abbreviated as HITL in the present specification | | Multi-Model Routing | A mechanism in which an LLM platform (e.g., Microsoft Copilot) dynamically switches the backend AI model according to the type and situation of a task | | <AI Classification Column> | | | TA-AI | AI classification results for Therapeutic Areas. Including Oncology, Neurology, Cardiology, etc. | | Disease-AI | AI classification results for disease names | | category-AI | AI classification results for nuclear medicine category. PET (18F), RLT (177Lu), SPECT (99mTc), etc. | | tracer-I-AI / tracer-T-AI | AI classification results for imaging tracers / therapeutic tracers | | RI-I-AI / RI-T-AI | AI classification results of nuclides for imaging / nuclides for treatment | | target-I-AI / target-T-AI | AI classification results for imaging target molecules / therapeutic target molecules | | <Nuclear Medicine> | | | Nuclear Medicine | A medical field that includes diagnosis (PET / SPECT) and treatment (RLT / TAT) using radioactive isotopes (RI). | PET | Positron Emission Tomography. Uses positron-emitting radionuclides such as 18F, 68Ga, 11C, and 89Zr. | | SPECT | Single Photon Emission Computed Tomography. Uses gamma-ray emitting radionuclides such as 99mTc, 123I, and 201Tl. | | RLT | Radioligand Therapy. Targeted therapy using beta-emitting radionuclides such as 177Lu and 131I. | TAT | Targeted Alpha Therapy. Targeted therapy using alpha-emitting radionuclides such as 225Ac and 211At. | Dosimetry | Dosimetry (dose measurement). Evaluating the distribution of radiopharmaceuticals within the body and the absorbed dose to organs and tumors after administration. Essential for treatment planning and safety assessment. | | PSMA | Prostate-Specific Membrane Antigen. A target molecule for the diagnosis (68Ga / 18F-labeled PET ligand) and treatment (177Lu / 225Ac-labeled RLT / TAT) of prostate cancer. | | FAPI | Fibroblast Activation Protein Inhibitor. A PET tracer that targets cancer-associated fibroblasts. | | 177Lu | Lutetium-177. Beta-ray emitting radionuclide. Used in RLT (PSMA-617, DOTATATE, etc.). Also emits gamma rays, making it suitable for SPECT imaging (dual-purpose). | | 225Ac | Actinium-225. Alpha-emitting nuclide (approx. 5.8 MeV). Emits a total of four alpha particles in its decay chain. Used in TAT (e.g., PSMA-617). Research SPECT imaging is possible using gamma rays from daughter nuclides (221Fr, 213Bi, etc.). | 90Y | Yttrium-90. Pure beta-emitting radionuclide (maximum 2.28 MeV). Used in treatments such as radioembolization. Bremsstrahlung SPECT imaging and PET / CT with extremely low-probability internal electron-pair production are available for post-treatment evaluation. | <Quality Control / Industry> | | | GMP | Good Manufacturing Practice (Standards for manufacturing and quality control of pharmaceuticals) | | SOP | Standard Operating Procedure | | OOS | Out of Specification. An investigation process conducted when a measurement value deviates from the specified value during quality testing. | CRO | Contract Research Organization (a contract research organization that handles the planning, implementation, and data management of clinical trials on behalf of pharmaceutical companies).

[0122] (Notes Regarding Test Results and Evaluation Expressions) The results of each example in the comparative test of multiple LLM providers and the descriptions and evaluations of the responses of each LLM service in this specification are observations made by the inventor using a chat-style LLM service provided to general users, and are reports of individual cases based on the inventor's impressions when using the service as a general user. These are not measurement results under a controlled evaluation environment via each company's API, nor are they official evaluations by each LLM service provider. Since each LLM service is continuously updated, these test results represent one example of the response to specific input data at the time the test was conducted, and do not evaluate the performance of the LLM service as a whole. Each expression in this specification is a frank description of the inventor's observations and considerations to the extent necessary to explain the feasibility and inventive step of the invention in the patent specification.

[0123] (Trademark Notice) GPT, GPT-4, GPT-4o, ChatGPT, Claude, Gemini, and Copilot as used herein are registered trademarks or trademarks of their respective owners.

Claims

1. An information processing system that classifies text data obtained from a database using a large-scale language model, A partitioning processing unit takes text data obtained from the aforementioned database, divides it into multiple processing blocks, inputs them into a large-scale language model, and independently performs classification processing using natural language inference for each processing block. A cross-sectional verification unit compares the multiple classification results obtained by the division processing unit with the original text data to detect row misalignment between blocks (a phenomenon that occurs when a large-scale language model writes classification results back to cells in spreadsheet software, where the destination row is shifted positionally from the row of the original text data; the same applies hereinafter), inconsistencies between classification results, or hallucination by the large-scale language model, The aforementioned cross-sectional verification unit corrects the row misalignment detected by identifying the row in which the misalignment occurred, and corrects it by re-inputting the text data of that row into the large-scale language model and re-executing the classification process. Equipped with, The aforementioned cross-sectional verification unit uses natural language inference based on a large-scale language model to verify the contextual correspondence between the classification results and the original text data, thereby detecting line misalignment between blocks, inconsistencies between classification results, or hallucination by the large-scale language model. This verification is performed not by assigning an identifier to each processing block and checking for matching identifiers, but by using natural language inference with a large-scale language model to verify the semantic correspondence between the classification results and the original text data. Information processing system.

2. The information processing system according to claim 1, The cross-sectional verification unit includes a source-matching verification that compares the classification result with the original text data and detects whether the large-scale language model has output information not contained in the original text data, and an internal consistency verification that verifies the mutual consistency between multiple classification items output by the large-scale language model without referring to the original text data. Furthermore, the information processing system evaluates the reliability of the classification result by detecting discrepancies between the text showing the inference process output by the large-scale language model and the final classification result value.

3. An information processing system according to claim 2, wherein the database is a clinical trial database, the text data is descriptive text of a clinical trial, and the classification processing is performed on multidimensional classification items including therapeutic area, disease name, therapeutic modality, and combinations thereof.

4. An information processing system according to claim 3, wherein the clinical trial database includes clinical trial information in the field of nuclear medicine, the classification process includes determining whether or not radioisotopes are used, identifying imaging nuclides and therapeutic nuclides, identifying molecular target proteins, and classifying imaging modalities, and the segmentation processing unit applies a set of determination rules specific to the field of nuclear medicine to the large-scale language model, which includes dual-use identification rules for distinguishing between therapeutic and imaging uses of radioisotopes, and multi-context identification rules for identifying the meaning of molecular target proteins in multiple contexts.

5. An information processing system according to claim 1, further comprising a Human-in-the-Loop processing unit that distributes classification results in which contradictions or hallucinations are detected, and boundary cases in which the confidence score is below a predetermined threshold value, to a human confirmation queue and provides them for final judgment by an operator.

6. An information processing system according to claim 1, further comprising an external analysis module that takes the detection results of the cross-sectional testing unit as input, acquires the judgment basis text and confidence score output by the cross-sectional testing unit, extracts records in which the confidence score is less than or equal to a predetermined standard value, aggregates frequently occurring missing information from the judgment basis text of the records, generates prompt correction candidates that supplement the aggregated missing information using a large-scale language model, applies the generated prompt correction candidates to the current prompt and re-executes the testing by the cross-sectional testing unit to evaluate changes in quality indicators, and presents prompt correction candidates in which quality improvement has been confirmed to the operator.

7. An information processing method for classifying text data obtained from a database using a large-scale language model, which is performed by an information processing system comprising a division processing unit, a cross-sectional verification unit, and a correction processing unit, The partitioning processing unit divides the text data obtained from the database into multiple processing blocks and inputs them into a large-scale language model, and performs a classification process using natural language inference independently for each processing block. The cross-sectional verification step involves the cross-sectional verification unit comparing the multiple classification results obtained in the division processing step with the original text data to detect row misalignment between blocks (a phenomenon that occurs when a large-scale language model writes classification results back to cells in spreadsheet software, where the destination row is positionally shifted from the row of the original text data; the same applies hereinafter), inconsistencies between classification results, or hallucination by the large-scale language model, using natural language inference by the large-scale language model, wherein this detection is performed by verifying the semantic correspondence between the classification results and the original text data using natural language inference by the large-scale language model, rather than by a verification method that assigns an identifier to each processing block and confirms the matching of said identifiers. The correction processing unit corrects the row misalignment detected in the cross-sectional verification step by identifying the row in which the misalignment occurred, re-inputting the text data of that row into the large-scale language model, and re-executing the classification process. Information processing methods including

Images