Method and device for automatically evaluating credibility of metaverse simulation system

Abstract

This invention provides an automated method and apparatus for evaluating the credibility of a metaverse simulation system, relating to the field of computer software testing and evaluation technology. The method includes: acquiring multimodal features of the metaverse simulation system to be evaluated; generating multiple evaluation agents corresponding to various virtual roles based on the application requirements of the metaverse simulation system; evaluating the metaverse simulation system based on the multimodal features using each evaluation agent, generating multiple corresponding evaluation reports; and integrating the evaluation reports to generate a credibility evaluation result for the metaverse simulation system. The automated method and apparatus for evaluating the credibility of a metaverse simulation system provided by this invention achieves full automation from feature input to report output in the credibility evaluation process, supports large-scale rapid quantitative evaluation, and improves the efficiency and accuracy of credibility evaluation for metaverse simulation systems.

Description

Technical Field

[0001] This invention relates to the field of computer software testing and evaluation technology, and in particular to an automated method and apparatus for evaluating the credibility of a metaverse simulation system. Background Technology

[0002] With the deepening application of metaverse technology in fields such as industrial simulation and virtual training, the quantitative evaluation of the credibility of high-fidelity simulation systems has become a key link in ensuring the reliability of virtual-real fusion applications.

[0003] Existing technologies often employ the Analytic Hierarchy Process (AHP) to construct evaluation systems, determining indicator weights through expert scoring and combining multi-attribute decision-making to achieve credibility assessment. However, such methods rely excessively on expert subjective experience, making them susceptible to cognitive biases and leading to inaccurate evaluation results. Furthermore, the manual scoring and feature extraction processes are time-consuming and labor-intensive, resulting in low evaluation efficiency for metaverse simulation systems and making it difficult to meet the high-frequency evaluation requirements of dynamic iteration in metaverse systems. Summary of the Invention

[0004] This invention provides an automated method and apparatus for evaluating the credibility of a metaverse simulation system, thereby solving the technical problems of inaccurate and inefficient credibility evaluation of metaverse simulation systems in the prior art.

[0005] This invention provides an automated method for evaluating the credibility of a metaverse simulation system, comprising the following steps: Obtain the multimodal characteristics of the metaverse simulation system to be evaluated; Based on the application requirements of the metaverse simulation system to be evaluated, multiple evaluation agents corresponding to various virtual roles are generated; each virtual role corresponds to one evaluation agent. The metaverse simulation system to be evaluated is evaluated by each evaluation agent based on the multimodal features, generating multiple corresponding evaluation reports; By integrating the aforementioned evaluation reports, a credibility evaluation result for the metaverse simulation system to be evaluated is generated.

[0006] According to the present invention, an automated credibility evaluation method for a metaverse simulation system is provided, wherein the multimodal features include at least one of data features, image features, and descriptive features; The data features include at least one of the following: the number of model triangles, texture map size, and model accuracy of the metaverse simulation system to be evaluated. The image features are cropped images of the main objects in the scene of the metaverse simulation system to be evaluated during operation; The descriptive features are the interaction flow information of the metaverse simulation system to be evaluated during operation.

[0007] According to the present invention, an automated credibility evaluation method for a metaverse simulation system is provided, wherein each evaluation agent evaluates the metaverse simulation system to be evaluated based on the multimodal features, generating multiple corresponding evaluation reports, including: The evaluation agent assigns weights to multiple preset evaluation indicators to obtain the indicator weights corresponding to each evaluation indicator. Each evaluation agent performs a multi-dimensional evaluation based on the multimodal features and the evaluation indicators to obtain a credibility score corresponding to the evaluation indicators. Based on the weights of the indicators, the credibility scores are weighted and fused to obtain the overall score; The evaluation report is generated based on the overall score.

[0008] According to the present invention, an automated credibility evaluation method for a metaverse simulation system is provided, wherein the evaluation agent performs weight allocation on multiple preset evaluation indicators to obtain the indicator weight corresponding to each evaluation indicator, including: The evaluation agent performs pairwise comparisons of the evaluation indicators to generate a judgment matrix. Based on a preset consistency index, the consistency score of the judgment matrix is ​​calculated. Based on the consistency score and the judgment matrix, the indicator weights corresponding to each evaluation indicator are obtained.

[0009] According to the present invention, an automated credibility evaluation method for a metaverse simulation system is provided, wherein obtaining the index weight corresponding to each evaluation index based on the consistency score includes: If the consistency score is less than or equal to a preset threshold, the indicator weight corresponding to each evaluation indicator is generated based on the judgment matrix. If the consistency score is greater than the preset threshold, the judgment matrix is ​​modified based on the preset arbitration rules until the consistency score of the judgment matrix is ​​greater than or equal to the preset threshold.

[0010] According to the present invention, an automated credibility evaluation method for a metaverse simulation system is provided, wherein the evaluation indicators include geometric credibility, illumination credibility, behavioral credibility, and spatiotemporal credibility. The evaluation sub-items of geometric reliability include dimensional accuracy, surface fitting accuracy, and number of polygons; The evaluation sub-items for lighting credibility include shadow distribution and color distribution; The evaluation sub-items for behavioral credibility include interaction relationships and dynamic simulation; The evaluation sub-items of spatiotemporal credibility include the distribution of ground features, vegetation distribution, light and shadow changes, and occlusion relationships.

[0011] This invention also provides an automated credibility evaluation device for a metaverse simulation system, comprising the following modules: The acquisition module is used to acquire the multimodal characteristics of the metaverse simulation system to be evaluated; The generation module is used to generate multiple evaluation agents corresponding to various virtual roles based on the application requirements of the metaverse simulation system to be evaluated; the virtual roles and the evaluation agents correspond one-to-one. The evaluation module is used to evaluate the metaverse simulation system to be evaluated based on the multimodal features by each evaluation agent, and generate multiple corresponding evaluation reports. An integration module is used to integrate the evaluation report and generate a credibility evaluation result for the metaverse simulation system to be evaluated.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement an automated credibility assessment method for any of the metaverse simulation systems described above.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an automated credibility assessment method for the metaverse simulation system as described above.

[0014] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements an automated credibility assessment method for any of the metaverse simulation systems described above.

[0015] The present invention provides an automated method and apparatus for evaluating the credibility of a metaverse simulation system. By acquiring the multimodal features of the metaverse simulation system to be evaluated, it overcomes the limitations of traditional single data sources and achieves comprehensive automated feature acquisition of complex simulation systems. Based on the application requirements of the metaverse simulation system to be evaluated, multiple evaluation agents corresponding to various virtual roles are generated. Each virtual role corresponds one-to-one with the evaluation agent, thereby simulating the evaluation perspectives of users with different professional backgrounds and usage preferences in the real world through multiple different evaluation agents. This replaces the traditional single expert scoring mode, avoids individual subjective bias, and improves the objectivity and efficiency of the evaluation. Each evaluation agent evaluates the metaverse simulation system to be evaluated based on the multimodal features, generating multiple corresponding evaluation reports. Through differentiated analysis driven by different agents, detailed evaluation reports are generated, ensuring the transparency, traceability, and reasonableness of the evaluation process. The evaluation reports are integrated to generate the credibility evaluation result of the metaverse simulation system to be evaluated, achieving full-process automation from feature input to report output. This supports large-scale, rapid quantitative evaluation and improves the efficiency and accuracy of credibility evaluation of metaverse simulation systems. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the automated credibility assessment method for the metaverse simulation system provided by this invention.

[0018] Figure 2 This is a schematic diagram of the structure of the automated credibility evaluation device for the metaverse simulation system provided by the present invention.

[0019] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0022] In this application's embodiments, "determine B based on A" means that factor A must be considered when determining B. It is not limited to "B can be determined based solely on A," but should also include: "determine B based on A and C," "determine B based on A, C, and E," "determine C based on A, and further determine B based on C," etc. Additionally, it can include using A as a condition for determining B, for example, "when A meets the first condition, determine B using the first method"; another example, "when A meets the second condition, determine B," etc.; another example, "when A meets the third condition, determine B based on the first parameter," etc. Of course, it can also be a condition where A is a factor in determining B, for example, "when A meets the first condition, determine C using the first method, and further determine B based on C," etc.

[0023] It should also be noted that the terms "first," "second," etc., used in this invention are used to distinguish similar objects, and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, and the number of objects is not limited; for example, the first object can be one or more.

[0024] In this invention, the term "multiple" refers to two or more kinds, and other quantifiers are similar.

[0025] The following is combined with Figures 1 to 3 This invention describes an automated reliability assessment method and apparatus for a metaverse simulation system.

[0026] Figure 1 This is a flowchart illustrating the automated credibility assessment method for the metaverse simulation system provided by this invention, as shown below. Figure 1As shown, the method includes the following steps: Step 101: Obtain the multimodal characteristics of the metaverse simulation system to be evaluated; Specifically, the metaverse simulation system to be evaluated can be an industrial digital twin training system for industrial training, a metaverse system for earthquake rescue contingency plan simulation, or a virtual concert immersive performance system for entertainment. Multiple modal information, including structured numerical values, visual images, and semantic descriptions, are simultaneously acquired from the metaverse simulation system to obtain multimodal features, thereby comprehensively characterizing the system's geometric, physical, and interactive properties.

[0027] Furthermore, the multimodal features include at least one of data features, image features, and descriptive features; The data features include at least one of the following: the number of model triangles, texture map size, and model accuracy of the metaverse simulation system to be evaluated. The image features are cropped images of the main objects in the scene of the metaverse simulation system to be evaluated during operation; The descriptive features are the interaction flow information of the metaverse simulation system to be evaluated during operation.

[0028] Specifically, multimodal features include one or more of the following: data features, image features, and descriptive features.

[0029] For example, in one embodiment, the metaverse simulation system to be evaluated is an industrial digital twin training system. The server automatically retrieves the number of triangles (e.g., 500,000 faces for a single device model) and texture map size (e.g., 2K resolution) from the resource files of the industrial digital twin training system to obtain data features. When the industrial digital twin training system is running, a virtual camera (e.g., the Camera component in Unreal Engine) is controlled by a script to capture the main objects in the scene at different time slices and from different perspectives using a random strategy (e.g., capturing 10 images containing equipment, personnel, and the environment) to obtain image features. At the same time, interaction flow information is extracted from the project configuration file to form complete descriptive features. Among them, the interaction flow information may be: when the user pushes the joystick forward, the virtual camera moves forward, footprints appear on the ground, and the corresponding scene is displayed.

[0030] This invention, through the structured definition of multimodal features, enables the standardized implementation of the feature extraction process, avoiding evaluation biases caused by feature ambiguity in traditional methods. Simultaneously, by extracting multimodal features, it achieves comprehensive and automated data collection of complex metaverse systems, breaking through the limitations of traditional single data sources and providing a data foundation for subsequent credibility assessment.

[0031] Step 102: Based on the application requirements of the metaverse simulation system to be evaluated, generate multiple evaluation agents corresponding to various virtual roles; the virtual roles and the evaluation agents correspond one-to-one. Specifically, using a Large Language Model (LLM), virtual characters with different professional backgrounds, usage preferences, and evaluation tendencies are dynamically generated based on the application requirements of the metaverse simulation system to be evaluated (such as industrial training and emergency drills). Each virtual character corresponds to an independent evaluation agent, simulating the evaluation perspectives of different users in the real world. The LLM can be a generative artificial intelligence model with complex reasoning capabilities, such as GPT-4, DeepSeek-V3, or Claude 3.5 Sonnet.

[0032] For example, in one embodiment, the application requirements of the metaverse simulation system to be evaluated are input into the LLM, which is then asked to identify a virtual user role that is interested in the metaverse simulation system. This role includes attribute information such as age, professional field, and attitude towards the metaverse simulation system. The LLM's Temperature is set to a high value (e.g., 0.7) to enhance the diversity of the evaluation agent. For the workflow output, it is ensured that the evaluation agent records its attribute information in the memory stream or state function.

[0033] This invention generates multiple different evaluation agents through LLM, replacing the traditional single expert scoring mode. This avoids individual subjective bias and can simulate massive user evaluations in a short time, thereby improving the objectivity and efficiency of the evaluation.

[0034] Step 103: Evaluate the metaverse simulation system to be evaluated by each evaluation agent based on the multimodal features, and generate multiple corresponding evaluation reports; Specifically, each evaluation agent, based on the attribute information corresponding to its own virtual role, calls the preset evaluation indicators (such as geometric credibility, illumination credibility, etc.), and uses multimodal features to independently analyze and score the metaverse simulation system to be evaluated, and generates an evaluation report containing sub-item scores, overall scores and corresponding scoring basis, thus realizing multi-dimensional evaluation.

[0035] This invention enables intelligent agents to perform differentiated evaluations based on the attribute information corresponding to their virtual roles, making the evaluation reports more closely resemble real user feedback. At the same time, by utilizing multimodal features for comprehensive evaluation, it avoids misjudgments caused by missing features and ensures the transparency and traceability of the evaluation process.

[0036] Step 104: Integrate the evaluation report to generate the credibility evaluation result of the metaverse simulation system to be evaluated.

[0037] Specifically, the overall scores in the evaluation reports output by all evaluation agents are statistically aggregated (e.g., averaged, confidence filtered, etc.) to obtain the final score, which serves as the credibility evaluation result of the metaverse simulation system to be evaluated.

[0038] For example, in one embodiment, after collecting evaluation reports from three evaluation agents, some relevant features are first randomly screened from the evaluation reports to reduce dimensionality and adapt to the context window capacity of the evaluation agents. Then, the evaluation agents perform a second screening of the initially screened features to identify content highly relevant to the evaluation object, and then perform the final evaluation. It should be noted that if the overall score is abnormally low, it means that neither the initial random screening nor the second screening extracted features strongly related to the evaluation object, causing the agents to output extremely low scores without valid evidence (in some embodiments, this may also be due to extremely low scores generated by a certain evaluation agent due to fluctuations in the temporary application programming interface (API). In this case, it is necessary to remove the extremely low scores and average the remaining scores to obtain the final score.

[0039] This invention, through automated integration of evaluation reports, achieves full-process automation from feature input to report output, enabling large-scale and rapid quantitative evaluation and improving the efficiency and accuracy of credibility assessment for metaverse simulation systems.

[0040] The present invention provides an automated credibility assessment method for metaverse simulation systems. By acquiring the multimodal features of the metaverse simulation system to be assessed, it overcomes the limitations of traditional single data sources and achieves comprehensive automated feature acquisition of complex simulation systems. Based on the application requirements of the metaverse simulation system to be assessed, multiple assessment agents corresponding to various virtual roles are generated. Each virtual role corresponds one-to-one with an assessment agent, thereby simulating the evaluation perspectives of users with different professional backgrounds and usage preferences in the real world through multiple different assessment agents. This replaces the traditional single expert scoring mode, avoids individual subjective bias, and improves the objectivity and efficiency of the assessment. Each assessment agent evaluates the metaverse simulation system to be assessed based on the multimodal features, generating multiple corresponding assessment reports. Through differentiated analysis driven by different agents, detailed assessment reports are generated, ensuring the transparency, traceability, and reasonableness of the assessment process. The assessment reports are integrated to generate the credibility assessment result of the metaverse simulation system to be assessed. This achieves full-process automation from feature input to report output, supports large-scale rapid quantitative assessment, and improves the efficiency and accuracy of credibility assessment for metaverse simulation systems.

[0041] Furthermore, the evaluation of the metaverse simulation system to be evaluated is performed by each evaluation agent based on the multimodal features, generating multiple corresponding evaluation reports, including: The evaluation agent assigns weights to multiple preset evaluation indicators to obtain the indicator weights corresponding to each evaluation indicator. Each evaluation agent performs a multi-dimensional evaluation based on the multimodal features and the evaluation indicators to obtain a credibility score corresponding to the evaluation indicators. Based on the weights of the indicators, the credibility scores are weighted and fused to obtain the overall score; The evaluation report is generated based on the overall score.

[0042] Furthermore, the step of assigning weights to multiple preset evaluation indicators through the evaluation agent to obtain the indicator weight corresponding to each evaluation indicator includes: The evaluation agent performs pairwise comparisons of the evaluation indicators to generate a judgment matrix. Based on a preset consistency index, the consistency score of the judgment matrix is ​​calculated. Based on the consistency score and the judgment matrix, the indicator weights corresponding to each evaluation indicator are obtained.

[0043] Specifically, the evaluation agent uses the Analytic Hierarchy Process (AHP) to compare the relative importance of each evaluation indicator and its sub-items pairwise, forming a judgment matrix. Based on a preset Consistency Index (CI), a consistency score is calculated for the judgment matrix. The judgment matrix is ​​a square matrix representing the relative importance of each element, and the CI measures the degree to which the judgment matrix deviates from consistency. A higher consistency score indicates a more severe inconsistency in the judgment matrix.

[0044] Furthermore, obtaining the indicator weight corresponding to each evaluation indicator based on the consistency score includes: If the consistency score is less than or equal to a preset threshold, the indicator weight corresponding to each evaluation indicator is generated based on the judgment matrix. If the consistency score is greater than the preset threshold, the judgment matrix is ​​modified based on the preset arbitration rules until the consistency score of the judgment matrix is ​​greater than or equal to the preset threshold.

[0045] Specifically, when the consistency score is less than or equal to a preset threshold, the weight of each evaluation indicator is directly output based on the judgment matrix; when the consistency score is greater than the preset threshold, a re-comparison or the introduction of an arbitration agent swarm to correct the judgment matrix is ​​triggered until the consistency score of the judgment matrix is ​​greater than or equal to the preset threshold. The arbitration agent swarm is generated by LLM based on preset arbitration rules.

[0046] For example, in one embodiment, if the CI of the judgment matrix output by the evaluation agent is 0.15 (>0.1), the arbitration mechanism is triggered: an arbitration agent group containing 24 different roles and 3 different base models (8 different roles represent neutral agents such as "neutral algorithm engineers", and the 3 base models include GPT-4, DeepSeek-V3 or Claude 3.5Sonnet) is invoked to recompare contradictory terms (such as whether the assignment of "geometric credibility vs. behavioral credibility" is reasonable, where the context window of the arbitration agent group only contains the credibility score of "geometric credibility vs. behavioral credibility" and does not include the scores of other groups), and attempts are made to modify the original assignment, such as adjusting the original assignment of 0.5 to 0.8. After recalculation, CI = 0.08, and the consistency check is passed.

[0047] This invention, through consistency verification, ensures the rationality of the weight allocation for each evaluation indicator, avoiding distortion of evaluation results due to logical contradictions. Simultaneously, by designing an arbitration mechanism, it addresses the issue of consistency failure caused by misunderstandings of a single agent, further enhancing the reliability of the evaluation.

[0048] Each evaluation agent calculates a credibility score for each evaluation indicator based on multimodal features. Then, based on the assigned indicator weights, all credibility scores are weighted and fused to obtain an overall score. Finally, an evaluation report is generated based on the overall score. The evaluation report may include the credibility score (i.e., sub-item score) for each evaluation indicator, the indicator weight for each evaluation indicator, the judgment matrix, the consistency score, the scoring basis, and the overall score.

[0049] This invention, through evaluating the autonomous allocation of index weights by intelligent agents, retains the systematic nature of the AHP method while avoiding the limitations of subjective weighting by experts in traditional AHP, thus improving the objectivity of weight allocation. This enables a comprehensive evaluation of complex simulation systems and enhances the interpretability and traceability of the evaluation process by generating detailed evaluation reports.

[0050] Furthermore, the evaluation metrics include geometric credibility, illumination credibility, behavioral credibility, and spatiotemporal credibility; The evaluation sub-items of geometric reliability include dimensional accuracy, surface fitting accuracy, and number of polygons; The evaluation sub-items for lighting credibility include shadow distribution and color distribution; The evaluation sub-items for behavioral credibility include interaction relationships and dynamic simulation; The evaluation sub-items of spatiotemporal credibility include the distribution of ground features, vegetation distribution, light and shadow changes, and occlusion relationships.

[0051] Specifically, the evaluation indicators include at least geometric credibility, illumination credibility, behavioral credibility, and spatiotemporal credibility.

[0052] When invoking the evaluation agent to calculate the credibility score corresponding to each evaluation indicator based on multimodal features, the prompt must include at least the name and corresponding definition of each evaluation indicator.

[0053] In this embodiment of the invention, geometric credibility can be defined as the virtual object having a similar geometric appearance to an objective object existing in the real world; lighting credibility can be defined as the color and brightness of the virtual object conforming to the human eye's perception of environmental elements under natural lighting conditions, including texture color, texture, brightness and darkness effects, etc.; behavioral credibility can be defined as the behavioral characteristics of entities in the virtual environment, such as changes in position, collisions, and deformations, conforming to the dynamics and kinematics of objects moving in the real world; spatiotemporal credibility can be defined as the scene elements of the virtual environment conforming to the real environment in terms of spatial distribution and temporal evolution, and meeting the requirements of human eye recognition in terms of scene scale and real-time rendering.

[0054] Each evaluation indicator also includes one or more evaluation sub-items. The relative importance of each evaluation sub-item can be compared using the AHP method, thereby assigning a corresponding weight to each evaluation sub-item. Then, the evaluation sub-items are weighted and fused to obtain the credibility score of each evaluation indicator.

[0055] In this embodiment of the invention, the evaluation sub-items of geometric reliability may include dimensional accuracy, surface fitting accuracy, and polygon count. Dimensional accuracy characterizes the error in geometric dimensions between the virtual model and the real object; surface fitting accuracy characterizes the degree of fit between the virtual model's surface and the real object's surface; and polygon count characterizes the polygon complexity of the virtual model.

[0056] The evaluation sub-items of lighting credibility can include shadow distribution and color distribution. Shadow distribution is used to characterize whether the position, shape, and intensity of shadows in the virtual scene conform to the logic of real lighting; color distribution is used to characterize the naturalness and harmony of the surface colors of objects in the virtual scene.

[0057] The evaluation sub-items of behavioral credibility can include interaction relationships and dynamic simulation. Interaction relationships are used to characterize the logical consistency between user actions and system feedback; dynamic simulation is used to characterize the physical realism of the motion of virtual objects (such as whether there is reasonable bounce or deformation upon collision).

[0058] The evaluation sub-items of spatiotemporal credibility can include the distribution of ground features, vegetation distribution, light and shadow variations, and occlusion relationships. Among them, the distribution of ground features is used to characterize whether the position and layout of objects (such as buildings, roads, and equipment) in the virtual scene conform to real-world laws; the distribution of vegetation is used to characterize the consistency of plant types, density, and growing environments in the virtual scene; the light and shadow variations are used to characterize the dynamic evolution of lighting in the virtual scene over time (such as whether the angle and intensity of shadows transition naturally from sunrise to sunset, and whether there is reasonable light decay at night); and the occlusion relationships are used to characterize the spatial occlusion logic between objects in the virtual scene (such as foreground objects should occlude background objects, and transparent materials should partially transmit behind objects).

[0059] In some embodiments, the prompt may also include the following content, so that the evaluating agent can iterate through all possible scores for any two evaluation metrics based on its attribute information: if {a} is more important than {b}, the score is greater than 1 point; if {b} is more important than {a}, the score is less than 1 point; if {a} and {b} are equally important, the score is 1 point. Here, {a} and {b} represent any two different evaluation metrics.

[0060] In some embodiments, when invoking the evaluation agent to score, it may also be required to output the basis for the score.

[0061] In some embodiments, since the attribute information of the virtual character corresponding to the evaluation agent includes the character's personality and viewpoint on the simulation, the format of the prompt can be strictly constrained in order to avoid the illusionary output of the large model. For example, the prompt can include the following: The result format is: Because {X}, therefore {a}: <credibility score>, and an output example is given according to the application requirements of the metaverse simulation system to be evaluated. Here, {X} represents the specific scoring criteria, and {a} represents any evaluation index.

[0062] This invention transforms abstract credibility into measurable, concrete indicators by designing multiple evaluation metrics and multiple evaluation sub-items corresponding to each metric, thereby achieving comprehensive and fine-grained evaluation of complex simulation systems.

[0063] The following describes the automated credibility evaluation device for the metaverse simulation system provided by the present invention. The automated credibility evaluation device for the metaverse simulation system described below can be referred to in correspondence with the automated credibility evaluation method for the metaverse simulation system described above.

[0064] Figure 2 This is a schematic diagram of the structure of the automated credibility evaluation device for the metaverse simulation system provided by the present invention, as shown below. Figure 2As shown. An embodiment of the present invention provides an automated credibility evaluation device for a metaverse simulation system, comprising an acquisition module 201, a generation module 202, an evaluation module 203, and an integration module 204, wherein: The acquisition module 201 is used to acquire the multimodal features of the metaverse simulation system to be evaluated; the generation module 202 is used to generate multiple evaluation agents corresponding to various virtual roles based on the application requirements of the metaverse simulation system to be evaluated; the virtual roles correspond one-to-one with the evaluation agents; the evaluation module 203 is used to evaluate the metaverse simulation system to be evaluated based on the multimodal features through each evaluation agent, and generate multiple corresponding evaluation reports; the integration module 204 is used to integrate the evaluation reports and generate the credibility evaluation result of the metaverse simulation system to be evaluated.

[0065] The automated credibility assessment device for a metaverse simulation system provided by this invention overcomes the limitations of traditional single data sources by acquiring the multimodal features of the metaverse simulation system to be assessed, thus achieving comprehensive automated feature acquisition of complex simulation systems. Based on the application requirements of the metaverse simulation system to be assessed, multiple assessment agents corresponding to various virtual roles are generated. Each virtual role corresponds one-to-one with the assessment agent, thereby simulating the evaluation perspectives of users with different professional backgrounds and usage preferences in the real world through multiple different assessment agents, replacing the traditional single expert scoring mode, avoiding individual subjective bias, and improving the objectivity and efficiency of the assessment. Each assessment agent evaluates the metaverse simulation system to be assessed based on multimodal features, generating multiple corresponding assessment reports. Through differentiated analysis driven by different agents, detailed assessment reports are generated, ensuring the transparency, traceability, and reasonableness of the assessment process. The assessment reports are integrated to generate the credibility assessment result of the metaverse simulation system to be assessed, achieving full-process automation from feature input to report output, supporting large-scale rapid quantitative assessment, and improving the efficiency and accuracy of credibility assessment for metaverse simulation systems.

[0066] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute an automated credibility assessment method for the metaverse simulation system, which includes: Obtain the multimodal characteristics of the metaverse simulation system to be evaluated; Based on the application requirements of the metaverse simulation system to be evaluated, multiple evaluation agents corresponding to various virtual roles are generated; each virtual role corresponds to one evaluation agent. The metaverse simulation system to be evaluated is evaluated by each evaluation agent based on the multimodal features, generating multiple corresponding evaluation reports; By integrating the aforementioned evaluation reports, a credibility evaluation result for the metaverse simulation system to be evaluated is generated.

[0067] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0068] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of executing the automated credibility assessment method for the metaverse simulation system provided by the above methods, the method comprising: Obtain the multimodal characteristics of the metaverse simulation system to be evaluated; Based on the application requirements of the metaverse simulation system to be evaluated, multiple evaluation agents corresponding to various virtual roles are generated; each virtual role corresponds to one evaluation agent. The metaverse simulation system to be evaluated is evaluated by each evaluation agent based on the multimodal features, generating multiple corresponding evaluation reports; By integrating the aforementioned evaluation reports, a credibility evaluation result for the metaverse simulation system to be evaluated is generated.

[0069] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an automated reliability assessment method for the metaverse simulation system provided by the methods described above, the method comprising: Obtain the multimodal characteristics of the metaverse simulation system to be evaluated; Based on the application requirements of the metaverse simulation system to be evaluated, multiple evaluation agents corresponding to various virtual roles are generated; each virtual role corresponds to one evaluation agent. The metaverse simulation system to be evaluated is evaluated by each evaluation agent based on the multimodal features, generating multiple corresponding evaluation reports; By integrating the aforementioned evaluation reports, a credibility evaluation result for the metaverse simulation system to be evaluated is generated.

[0070] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0071] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automated credibility assessment method for a metaverse simulation system, characterized in that, include: Obtain the multimodal characteristics of the metaverse simulation system to be evaluated; Based on the application requirements of the metaverse simulation system to be evaluated, multiple evaluation agents corresponding to various virtual roles are generated. Each virtual character corresponds one-to-one with the evaluation agent; The metaverse simulation system to be evaluated is evaluated by each evaluation agent based on the multimodal features, generating multiple corresponding evaluation reports; By integrating the aforementioned evaluation reports, a credibility evaluation result for the metaverse simulation system to be evaluated is generated.

2. The automated credibility evaluation method for the metaverse simulation system according to claim 1, characterized in that, The multimodal features include at least one of data features, image features, and descriptive features; The data features include at least one of the following: the number of model triangles, texture map size, and model accuracy of the metaverse simulation system to be evaluated. The image features are cropped images of the main objects in the scene of the metaverse simulation system to be evaluated during operation; The descriptive features are the interaction flow information of the metaverse simulation system to be evaluated during operation.

3. The automated credibility assessment method for the metaverse simulation system according to claim 1, characterized in that, The evaluation process involves each evaluation agent assessing the metaverse simulation system based on the multimodal features, generating multiple corresponding evaluation reports, including: The evaluation agent assigns weights to multiple preset evaluation indicators to obtain the indicator weights corresponding to each evaluation indicator. Each evaluation agent performs a multi-dimensional evaluation based on the multimodal features and the evaluation indicators to obtain a credibility score corresponding to the evaluation indicators. Based on the weights of the indicators, the credibility scores are weighted and fused to obtain the overall score; The evaluation report is generated based on the overall score.

4. The automated credibility evaluation method for the metaverse simulation system according to claim 3, characterized in that, The step of assigning weights to multiple preset evaluation indicators through the evaluation agent to obtain the indicator weight corresponding to each evaluation indicator includes: The evaluation agent performs pairwise comparisons of the evaluation indicators to generate a judgment matrix. Based on a preset consistency index, the consistency score of the judgment matrix is ​​calculated. Based on the consistency score and the judgment matrix, the indicator weights corresponding to each evaluation indicator are obtained.

5. The automated credibility evaluation method for the metaverse simulation system according to claim 4, characterized in that, The step of obtaining the indicator weight corresponding to each evaluation indicator based on the consistency score includes: If the consistency score is less than or equal to a preset threshold, the indicator weight corresponding to each evaluation indicator is generated based on the judgment matrix. If the consistency score is greater than the preset threshold, the judgment matrix is ​​modified based on the preset arbitration rules until the consistency score of the judgment matrix is ​​greater than or equal to the preset threshold.

6. The automated credibility evaluation method for the metaverse simulation system according to claim 3, characterized in that, The evaluation metrics include geometric credibility, illumination credibility, behavioral credibility, and spatiotemporal credibility. The evaluation sub-items of geometric reliability include dimensional accuracy, surface fitting accuracy, and number of polygons; The evaluation sub-items for lighting credibility include shadow distribution and color distribution; The evaluation sub-items for behavioral credibility include interaction relationships and dynamic simulation; The evaluation sub-items of spatiotemporal credibility include the distribution of ground features, vegetation distribution, light and shadow changes, and occlusion relationships.

7. An automated credibility evaluation device for a metaverse simulation system, characterized in that, include: The acquisition module is used to acquire the multimodal characteristics of the metaverse simulation system to be evaluated; The generation module is used to generate multiple evaluation agents corresponding to various virtual roles based on the application requirements of the metaverse simulation system to be evaluated; the virtual roles and the evaluation agents correspond one-to-one. The evaluation module is used to evaluate the metaverse simulation system to be evaluated based on the multimodal features by each evaluation agent, and generate multiple corresponding evaluation reports. An integration module is used to integrate the evaluation report and generate a credibility evaluation result for the metaverse simulation system to be evaluated.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the automated credibility assessment method for the metaverse simulation system as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the automated credibility assessment method for the metaverse simulation system as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the automated credibility assessment method for the metaverse simulation system as described in any one of claims 1 to 6.

Images