Knowledge graph and transmutation test-based spatiotemporal reasoning test method for large language model

Abstract

This invention discloses a spatiotemporal reasoning testing method for large language models based on knowledge graphs and metamorphic testing, belonging to the field of large language model testing technology. This invention uses structured spatiotemporal facts in knowledge graphs as data foundation to automatically construct temporal and spatial reasoning tasks, and generates variant samples through numerical, logical, geometrically symmetric, and structurally perturbative mutation operators. Subsequently, the original and variant samples are input into the large language model under test, and the model output is automatically evaluated through rule extraction, numerical soft constraints, and relation consistency judgment. This invention can evaluate the accuracy, consistency, and structural stability of the model in temporal and spatial reasoning tasks, achieving end-to-end automated testing and reducing the cost of manual annotation and evaluation.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence software testing and large language model evaluation technology, specifically a spatiotemporal reasoning testing method for large language models based on knowledge graphs and metamorphic testing. Background Technology

[0002] In recent years, large language models have been widely used in intelligent question answering, decision support, software agents, scheduling, geographic information question answering, and complex interactive systems. As large language models increasingly participate in real-world applications, the reliability, stability, and interpretability of their outputs have become critical issues in software quality assurance. Compared to general text generation tasks, reasoning tasks place greater emphasis on the model's ability to make stable, correct, and self-consistent judgments based on explicit premises. Temporal reasoning and spatial reasoning, corresponding to the fundamental cognitive needs of the time and space dimensions in the real world respectively, are important aspects for measuring the structured reasoning capabilities of large language models.

[0003] Existing evaluation methods for large language models typically rely on fixed benchmarks and static accuracy metrics, which measure whether the model's output matches the standard answer given a set of test questions. While these methods can reflect the model's overall performance on a specific dataset, they still have significant limitations. First, static accuracy is insufficient to determine whether the model truly grasps the underlying reasoning structure. The model may rely on high-frequency memories, entity co-occurrence patterns, or surface linguistic cues in the training corpus to provide correct answers, rather than reasoning based on explicit spatiotemporal constraints in the questions. Second, existing evaluation methods lack sufficient analysis of behavioral stability under varying input conditions. For example, the model may answer correctly on the original time question but become inconsistent after overall time shifting or relation inversion; it may also answer correctly on the original spatial question but err after direction inversion, distance symmetry, or scrambling of premise order. Third, many large language models output free text, resulting in insufficient test prediction, meaning it is difficult to manually construct complete, reliable, and automatically verifiable standard answers for every input. Summary of the Invention

[0004] The purpose of this invention is to provide a spatiotemporal reasoning testing method for large language models based on knowledge graphs and metamorphic testing, so as to solve the problems mentioned in the background art of insufficient factual support, excessive static accuracy, difficulty in distinguishing between knowledge memory and structural reasoning, and difficulty in automated adjudication in the spatiotemporal reasoning evaluation of large language models.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: A spatiotemporal reasoning testing method based on knowledge graphs and metamorphic testing using a large language model includes the following steps: Entities with explicit temporal or spatial attributes are extracted from the knowledge graph to form a temporal fact set and a spatial fact set. For temporal facts, attributes such as start time, end time, and occurrence time of events or entities are extracted and represented as follows: ; Indicates the name of the event or entity. Indicates the start time. Indicates the end time. For an event that contains only a single point in time, it can be represented as: t represents a point in time.

[0006] For spatial facts, the latitude and longitude coordinates of geographic entities are extracted and represented as: ;in, Indicates the location name. and These represent latitude and longitude, respectively. This invention utilizes temporal and spatial facts from a knowledge graph to construct test samples, ensuring that the test data has a clear source and verifiable standard answers, thus overcoming the problems of limited sample size and insufficient factual consistency in manually constructed samples.

[0007] Based on the set of temporal facts, we construct multiple types of temporal reasoning tasks and determine the temporal mutation operators corresponding to these tasks.

[0008] Based on the set of spatial facts, we construct multiple types of spatial reasoning tasks and determine the spatial mutation operators corresponding to these tasks.

[0009] Construct a natural language prompt template system.

[0010] Based on temporal and spatial mutation operators, mutation samples are obtained. The expected relationship between the original samples and mutation samples is constructed through the metamorphic test concept. Instead of requiring a complete standard answer to be manually designed for each free text output, the stability of the model behavior is judged by the constraints of the relationship between the outputs, thereby improving the automated evaluation capability.

[0011] Based on the established natural language prompt template system, the original and variant samples are input into the large language model under test for reasoning testing. Answers are extracted and judged for the reasoning tests. The model under test can be a closed-source commercial large language model or an open-source local large language model. For each model under test, its output text, extraction results, and judgment results on the original and variant samples are recorded.

[0012] The large language model under test outputs quantitative evaluation indicators, and a test report on the model's spatiotemporal reasoning ability is generated based on the quantitative evaluation indicators.

[0013] According to the above technical solution, the temporal fact set and the spatial fact set are filtered. The filtering includes filtering for time format validity, invalid date, coordinate validity, entity name ambiguity, and fuzzy relation boundary samples, so as to ensure that the test samples have clear standard answers.

[0014] According to the above technical solution, the temporal reasoning tasks include time calculation tasks, duration comparison tasks, event sequence tasks, and Allen interval relationship tasks; the multi-type spatial reasoning tasks include direction judgment tasks, distance judgment tasks, and multi-hop direction reasoning tasks.

[0015] This invention simultaneously covers tasks such as time point calculation, duration comparison, event sorting, Allen interval relationships, direction judgment, distance judgment, and multi-hop spatial reasoning, and can evaluate the spatiotemporal reasoning ability of large language models from multiple perspectives.

[0016] The time calculation task involves giving a specific point in time and a time offset, requiring the model to calculate the time after the offset. For example, given the time of an event and the condition of "advancing by several years / months" or "delaying by several years / months", the model is required to output the new time point.

[0017] Duration comparison task: Given two events with start and end times, the model is required to determine which event lasts longer, or whether the two events have the same duration.

[0018] Event sequence task: Given the start time or occurrence time of two events, the model is required to determine which event occurred earlier or later.

[0019] Allen Interval Relationship Task: Given two time intervals, the model is required to determine the interval relationship between them. Interval relationships include Before (one interval completely precedes the other), After (one interval completely follows the other), Meets (the end time of one interval equals the start time of the other), Met-by (the end time of one interval is exactly the same as the start time of the other), Overlaps (one interval partially overlaps with the other), Overlapped-by (one interval comes before the other), Starts (both intervals have the same start time, but one is shorter), Started-by (both intervals have the same start time, but one is longer), During (one interval is completely inside the other), Contains (the other interval contains this interval), Finishes (both intervals have the same end time, but one is shorter), Finished-by (both intervals have the same end time, but one is longer), and Equals (both intervals are exactly the same).

[0020] Direction determination task: Based on the latitude relationship between two locations, construct a north-south direction determination problem. If the locations... The latitude is greater than the location ,but lie in It is located to the north; conversely, it is located to the south.

[0021] Distance estimation task: Construct a distance estimation problem based on the latitude and longitude coordinates of two locations. Reference distances can be calculated using spherical distance formulas or geodesic distance functions, but the standard answer is not directly exposed to the model.

[0022] Multi-hop directional reasoning task: Construct multi-hop relationship chains based on directional relationships between multiple locations. For example, if... exist north, exist north, exist In the north, model reasoning is required. Compared to The direction. This task is used to test the model's ability to propagate spatial relationships under multiple explicit premises.

[0023] According to the above technical solution, the time-series mutation operator includes a time translation operator, a duration-preserving mutation operator, a sequence-preserving mutation operator, an entity abstraction operator, and an inverse relation transformation operator; The time translation operator is used to apply a fixed offset to all time points in a time calculation task. For example, if the original answer is... Then the mutated answer should change in the same direction and magnitude.

[0024] The duration-preserving mutation operator is used to synchronously increase the end times of two events by the same length, keeping the relationship between their durations unchanged. If the model has a stable ability to compare durations, the output relationship on the original sample and the mutated sample should be consistent.

[0025] The order-preserving mutation operator is used to synchronously shift the start times of two events by the same length, ensuring that their order remains unchanged. If the model has a stable event ordering capability, the outputs should remain consistent.

[0026] The entity abstraction operator replaces the real historical event names with abstract symbols such as Event_A (Event A) and Event_B (Event B) while keeping the underlying time constraints unchanged, thereby reducing the impact of entity memory on the reasoning results.

[0027] The inverse relation transformation operator is used to swap the order of two events in the Allen interval relation task and requires the model to output the corresponding inverse relation. For example, if the original relation is Before, the swapped relation should be After; if the original relation is Starts, the swapped relation should be Started-by.

[0028] According to the above technical solution, the spatial mutation operator includes a direction inversion operator, a distance symmetric operator, a spatial entity abstraction operator, a premise order permutation operator, and a composite structure perturbation operator; The direction inversion operator is used to swap the positions of two locations in the problem. For example, if in the original problem... lie in In the north, the problem of variation... Should be located south.

[0029] The distance-symmetric operator is used to exchange the starting and ending points in the distance problem. This is because, in physical space... The distance output by the model in the two queries should remain consistent within the tolerance range.

[0030] The spatial entity abstraction operator is used to replace real place names with abstract symbols such as Location_A (location A) and Location_B (location B), so that the model can only perform spatial reasoning based on explicit premises.

[0031] The premise order substitution operator is used to randomly shuffle the order of premise statements in multi-hop directional reasoning tasks, but keeps the underlying logical relationship unchanged, so that the model can only perform spatial reasoning based on explicit premises.

[0032] The composite structure perturbation operator is used to simultaneously execute the spatial entity abstraction operator and the premise order permutation operator to verify the spatial structure stability of the model under complex input changes.

[0033] This invention weakens the influence of background knowledge brought by real event names or real place names by entity abstraction and knowledge conflict testing, thereby analyzing whether the model truly infers based on explicit spatiotemporal relationships, rather than relying solely on entity memory in the pre-training corpus.

[0034] In addition to the time translation, direction inversion, distance symmetry, entity abstraction, and premise order permutation, further mutation operators such as time scaling, time granularity conversion, east-west direction transformation, spatial rotation, spatial mirroring, path length preservation, multi-hop chain compression, or multi-hop chain expansion can be designed.

[0035] According to the above technical solution, the natural language prompt template system is used to convert multiple types of temporal reasoning tasks and multiple types of spatial reasoning tasks into natural language question-and-answer format, and embeds reasoning logic constraints and standardized output constraints in the prompts, so that the large language model under test can complete spatiotemporal reasoning output according to the rules.

[0036] Specifically, for the direction judgment task, time point calculation task, duration comparison task, and event sorting task, the large language model under test is required to output a limited label; for the distance judgment task, the large language model under test is required to output a direct answer containing numerical values; and for the Allen interval relationship task, the large language model under test is required to output a specified interval relationship label.

[0037] According to the above technical solution, the evaluation indicators include original accuracy, variation consistency, de-entityization accuracy, relational consistency, and error distribution.

[0038] Original accuracy: Whether the output of the statistical model on the original samples is consistent with the standard answer generated by the knowledge graph.

[0039] Variation consistency: Whether the output of the statistical model on the original sample and the variant sample satisfies the expected metamorphic relationship.

[0040] Relational self-consistency: Whether the statistical model outputs the corresponding inverse relation under relational inversion or directional inversion conditions.

[0041] Error distribution: For distance judgment tasks, the mean, median, quartiles and maximum value of the relative error are statistically analyzed.

[0042] This invention not only measures the accuracy of the model on the original sample, but also measures the consistency of the model under conditions such as time shift, direction inversion, relation inversion, de-entityization, and scrambling of premise order. It can discover the potential defect of the model that "answers the original question correctly but becomes unstable after mutation".

[0043] A test report on the model's spatiotemporal reasoning capability is generated based on the above indicators. The report includes the model's accuracy, consistency, de-entityization performance, relational consistency, error distribution, and performance changes under different mutation conditions on different tasks. The test report can be used to compare the spatiotemporal reasoning capabilities of different models and to identify potential deficiencies in models during spatiotemporal reasoning tasks.

[0044] The present invention includes a technical solution, an electronic device comprising: one or more processors; and a storage device for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors execute any of the above technical solutions a spatiotemporal reasoning testing method for a large language model based on knowledge graphs and metamorphic testing.

[0045] The invention includes a technical solution, a storage medium storing at least one instruction, which is loaded and executed by a processor to implement the spatiotemporal reasoning testing method for large language models based on knowledge graphs and metamorphic testing as described in any of the above technical solutions.

[0046] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: This invention uses structured spatiotemporal facts in knowledge graphs as its data foundation, automatically constructs temporal and spatial reasoning tasks, and generates variant samples through numerical, logical, geometrically symmetric, and structurally perturbative mutation operators. Subsequently, the original and variant samples are input into the large language model under test, and the model output is automatically evaluated through rule extraction, numerical soft constraints, and relation consistency determination. This enables the evaluation of the model's accuracy, consistency, and structural stability in temporal and spatial reasoning tasks, achieving end-to-end automated testing and reducing the costs of manual annotation and evaluation. Furthermore, this invention can be used to compare the capabilities of different large language models in spatiotemporal reasoning tasks, and also for regression testing between different versions of the same model, thus providing a basis for the deployment, updating, and quality assurance of large language model software systems. Attached Figure Description

[0047] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of the steps of the spatiotemporal reasoning testing method for a large language model based on knowledge graphs and metamorphic testing, as described in this invention. Figure 2 This is a flowchart of the answer extraction and relation determination process for the large language model under test. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Using Wikidata knowledge graphs (wherein, knowledge graphs can also be replaced by DBpedia, YAGO, domain knowledge graphs, enterprise internal knowledge bases, or other structured knowledge bases containing time and space attributes) to construct test samples, this invention's spatiotemporal reasoning testing method for large language models based on knowledge graphs and metamorphic testing is applied to evaluate closed-source commercial large language models. Specific steps (…) Figure 1 )include: S1. Knowledge Graph Fact Extraction: Extracting entities with explicit temporal or spatial attributes from the knowledge graph to form temporal fact sets and spatial fact sets. Specifically: For time-series facts, attributes such as start time, end time, and occurrence time of events or entities are extracted and represented as follows: ; in, Indicates the name of the event or entity. Indicates the start time. Indicates the end time. For an event that contains only a single point in time, it can be represented as: ; For spatial facts, the latitude and longitude coordinates of geographic entities are extracted and represented as: ; in, Indicates the location name. and These represent latitude and longitude, respectively.

[0050] S2. Sample Filtering: Filtering the time-series and spatial fact sets, including filtering for valid time formats, invalid dates, valid coordinates, ambiguous entity names, and fuzzy relational boundaries.

[0051] S3. Task Sample Generation: Based on the set of temporal facts, construct multiple types of temporal reasoning tasks; based on the set of spatial facts, construct multiple types of spatial reasoning tasks.

[0052] The various temporal reasoning tasks include, but are not limited to, time calculation tasks, duration comparison tasks, event sequence tasks, and Allen interval relation tasks. Simultaneously, temporal mutation operators corresponding to these tasks are determined, including, but not limited to, time translation operators, duration-preserving mutation operators, sequence-preserving mutation operators, entity abstraction operators, and inverse relation transformation operators.

[0053] Specifically, the time translation operator is used to apply a fixed offset to all time points in a time calculation task.

[0054] The duration-preserving mutation operator is used to synchronously increase the end time of two events by the same length.

[0055] The order-preserving mutation operator is used to synchronously shift the start times of two events by the same length.

[0056] The entity abstraction operator replaces real historical event names with abstract symbols while keeping the underlying time constraints unchanged.

[0057] The inverse relation transformation operator is used to swap the order of two events in the Allen interval relation task and requires the model to output the corresponding inverse relation.

[0058] Based on a set of spatial facts, we construct multiple types of spatial reasoning tasks. These tasks include direction determination, distance determination, and multi-hop direction reasoning.

[0059] Simultaneously, spatial mutation operators corresponding to various types of spatial reasoning tasks are identified, including direction inversion operators, distance symmetry operators, spatial entity abstraction operators, premise order permutation operators, and composite structure perturbation operators.

[0060] The direction inversion operator is used to swap the positions of two locations in the problem.

[0061] Distance symmetric operators are used to exchange the starting and ending points in distance problems.

[0062] Spatial entity abstraction operators are used to replace real place names with abstract symbols.

[0063] The premise order substitution operator is used to randomly shuffle the order of premise statements in multi-hop directional reasoning tasks, while keeping the underlying logical relationships unchanged.

[0064] The composite structure perturbation operator is used to simultaneously execute the spatial entity abstraction operator and the premise order permutation operator.

[0065] S4. Mutant Sample Generation: Based on the temporal mutation operator, corresponding operations such as time shift, duration preservation, order preservation, entity abstraction, and inverse relation change are performed to construct mutant test samples. Among them, all mutant samples have undergone temporal constraint verification and shortcut filtering to remove invalid samples that can be answered directly through language priors. This ensures that all test samples require the model to obtain results through logical deduction and temporal calculation. Each sample is configured with complete context, test question, unique standard answer, and difficulty label.

[0066] Based on the spatial mutation operator, corresponding entity abstraction direction inversion, distance symmetry, premise order permutation, and conformity structure perturbation are performed on temporal facts and spatial facts to construct mutation test samples. Among them, the mutation samples are all subject to temporal constraint verification and shortcut filtering to remove invalid samples that can be answered directly by language priors, ensuring that all test samples require the model to obtain results through logical deduction and temporal calculation. Each sample is configured with complete context, test question, unique standard answer and difficulty label.

[0067] S5. Large Language Model Invocation: Based on the natural language prompt template system, the original samples and variant samples are input into the closed-source commercial large language model for inference testing.

[0068] Among them, the natural language prompt template system is used to convert multiple types of temporal reasoning tasks and multiple types of spatial reasoning tasks into natural language question-and-answer formats, and embeds reasoning logic constraints and standardized output constraints in the prompts, so that the large language model under test can complete the spatiotemporal reasoning output according to the rules.

[0069] Specifically, for the direction determination task, time point calculation task, duration comparison task, and event sorting task, the closed-source commercial large language model is required to output limited labels. For example, in the direction determination task, only "North" or "South" is allowed to be output; in the time point calculation task, only 'Month, Year' formats such as 'January, 1990' are allowed to be output; and in the duration comparison task and the event sorting task, only "Event_A" or "Event_B" is allowed to be output.

[0070] For distance judgment tasks, the large language model under test is required to output a direct answer that includes numerical values.

[0071] For the Allen interval relation task, the large language model under test is required to output a specified interval relation label. For example, for the Allen interval relation task, the large language model under test is required to output a specified interval relation label, such as "Before", "After", "During", "Starts" or "Overlaps" (13 Allen interval relations).

[0072] For example: For a direction determination task, the prompt template could be: Location A is at latitude Location B is at latitude . IsLocation B to the North or South of Location A? Output ONLY North or South.”

[0073] For multi-hop spatial reasoning tasks, the hint template can be: "Premise 1: B is North of A. Premise 2: C is North of B. Question:What is the direction of C relative to A? Output ONLY North or South."

[0074] For Allen's interval relation task, the prompt template can be: “Event A lasted from to Event B lasted from to . What is the Allen interval relation between Event A and Event B? Output ONLY onerelation label.”

[0075] S6. Answer Extraction: The output of the closed-source commercial large language model is used for answer extraction and determination. The specific steps include: S601. Perform data cleaning on the output of the closed-source commercial large language model. Data cleaning includes removing noise and redundancy, and standardizing the format.

[0076] S602. Extract labels and values ​​from the output of the closed-source business large language model after data cleaning.

[0077] Specifically, for closed-label tasks, regular expressions and keyword matching are used to extract answer labels from the model output. For example, labels such as North, South, Before, After, and During are extracted from the output.

[0078] For distance determination tasks, regular expressions are used to extract numerical values.

[0079] S603, based on extracted label and numerical rule matching and numerical soft constraints.

[0080] For example, for distance judgment tasks, regular expressions are used to extract numerical values, and the relative error between the output distances of the original problem and the mutated problem is calculated. ;in, This represents the distance output of the model in the original problem. This represents the distance output of the model in the mutation problem. express and The maximum of the two.

[0081] For relation inversion tasks, construct a relation mapping table. For example: Before corresponds to After, After corresponds to Before, Starts corresponds to Started-by, and Started-by corresponds to Starts. If the model's output on the original sample and the inverted sample satisfies this mapping relationship, then the sample is considered to have passed the relationship consistency test.

[0082] In addition, the results of the large language model under test can also be judged by combining a procedural validator, a knowledge graph query validator, a constraint solver, or a calibrated semantic matching model.

[0083] S7. Indicator Calculation: Based on the results of extracted label and numerical rule matching and numerical soft constraint, determine the corresponding evaluation indicators such as original accuracy, variation consistency, de-entityization accuracy, relational consistency and error distribution.

[0084] S8. Test Report Generation: Generate a test report on the model's spatiotemporal reasoning ability based on the above indicators. The report includes the model's accuracy, consistency, de-entityization performance, relational consistency, error distribution, and performance changes under different mutation conditions on different tasks. The test report can be used to compare the spatiotemporal reasoning abilities of different models and to identify potential defects in the model's spatiotemporal reasoning tasks.

[0085] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0086] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A spatiotemporal reasoning testing method for large language models based on knowledge graphs and metamorphic testing, characterized in that: The steps include: Entities with explicit temporal or spatial attributes are extracted from the knowledge graph to form a temporal fact set and a spatial fact set; Based on the set of temporal facts, we construct multiple types of temporal reasoning tasks and determine the temporal mutation operators corresponding to these tasks. Based on the set of spatial facts, we construct multiple types of spatial reasoning tasks and determine the spatial mutation operators corresponding to these tasks. Construct a natural language prompt template system; Mutated samples are obtained based on temporal mutation operators and spatial mutation operators; Based on the completed natural language prompt template system, the original samples and variant samples are input into the large language model to be tested for reasoning. The reasoning test is evaluated by extracting and judging answers and quantifying the evaluation indicators. Based on the quantified evaluation indicators, a test report on the spatiotemporal reasoning ability of the model is generated.

2. The spatiotemporal reasoning testing method for large language models based on knowledge graphs and metamorphic testing according to claim 1, characterized in that, The time-series fact set and the spatial fact set are filtered, including filtering for time format validity, invalid dates, coordinate validity, entity name ambiguity, and fuzzy relation boundary samples.

3. The spatiotemporal reasoning testing method for large language models based on knowledge graphs and metamorphic testing according to claim 1, characterized in that, The various types of temporal reasoning tasks include time calculation tasks, duration comparison tasks, event sequence tasks, and Allen interval relationship tasks; the various types of spatial reasoning tasks include direction judgment tasks, distance judgment tasks, and multi-hop direction reasoning tasks.

4. The spatiotemporal reasoning testing method for large language models based on knowledge graphs and metamorphic testing according to claim 3, characterized in that: The temporal mutation operators include time translation operators, duration preservation mutation operators, order preservation mutation operators, entity abstraction operators, and inverse relation transformation operators; The time translation operator is used to apply a fixed offset to all time points in the time calculation task; The duration-preserving mutation operator is used to synchronously increase the end time of two events by the same length; The order-preserving mutation operator is used to synchronously shift the start times of two events by the same length. The entity abstraction operator replaces real historical event names with abstract symbols while keeping the underlying time constraints unchanged. The inverse relation transformation operator is used to swap the order of two events in the Allen interval relation task and requires the model to output the corresponding inverse relation.

5. The spatiotemporal reasoning testing method for large language models based on knowledge graphs and metamorphic testing according to claim 3, characterized in that, The spatial mutation operators include the direction inversion operator, the distance symmetric operator, the spatial entity abstraction operator, the premise order permutation operator, and the composite structure perturbation operator; The direction inversion operator is used to swap the positions of two locations in the problem; The distance-symmetric operator is used to exchange the start and end points in the distance problem; The spatial entity abstraction operator is used to replace real place names with abstract symbols; The premise order substitution operator is used to randomly shuffle the order of premise statements in multi-hop directional reasoning tasks, but keep the underlying logical relationship unchanged. The composite structure perturbation operator is used to simultaneously execute the spatial entity abstraction operator and the premise order permutation operator.

6. The spatiotemporal reasoning testing method for large language models based on knowledge graphs and metamorphic testing according to claim 4, characterized in that, The natural language prompt template system is used to convert various types of temporal reasoning tasks and various types of spatial reasoning tasks into natural language question-and-answer formats, and embeds reasoning logic constraints and standardized output constraints in the prompts, so that the large language model under test can complete spatiotemporal reasoning output according to the rules.

7. The spatiotemporal reasoning testing method for large language models based on knowledge graphs and metamorphic testing according to claim 1, characterized in that, The evaluation metrics include original accuracy, consistency of variation, de-entityization accuracy, relational consistency, and error distribution.

8. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method according to any one of claims 1-7.

9. A storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to implement the method as described in any one of claims 1-7.

Images