A method and system for forced role attribution feedback calibration for multi-role large language model simulation

Abstract

The application discloses a forced role attribution feedback calibration method and system for multi-role large language model simulation, and belongs to the technical field of artificial intelligence simulation and feedback processing. When simulation is created, all role versions and scene configuration versions of a role library are read as fixed fields for persistent storage with simulation records, and are not affected by subsequent version upgrades; when a user submits an unauthentic feedback, it is forced to check whether an attribution role exists in a version snapshot bound to the simulation record, and after passing, the user dimension idempotent key is used to record and bind the exact version number of the role; according to scene configuration, version number, role and feedback type, grouping, counting, sorting and outputting a calibration priority signal are performed. The application solves technical problems such as lack of role attribution in user feedback, disconnection between feedback records and version states and lack of aggregated calibration signals, and can be used for an intelligent simulation decision platform.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence simulation and feedback data processing technology, specifically relating to a user feedback attribution verification, idempotent deduplication, and multi-dimensional aggregation calibration method and system for multi-role large language model simulation systems. It is particularly suitable for intelligent simulation decision-making platforms that need to continuously calibrate the behavioral parameters of simulation roles based on feedback from a group of users. Background Technology

[0002] In simulation systems based on multi-role large language models, multiple virtual characters engage in game-like reasoning around a given scenario and output simulation conclusions. Since the character behavior logic is generated through free reasoning by the large language model rather than deterministic formula calculations, the authenticity and rationality of the output conclusions lack objective verification methods and typically require post-event evaluation and iterative calibration based on user subjective feedback. However, existing technologies have significant shortcomings in effectively collecting and utilizing this type of user feedback. The main solutions in the existing technology include: Option 1: Overall Rating Feedback Scheme. Users rate the overall output of the simulation as satisfied / unsatisfied or gave a thumbs-up / dislike. This is a common practice in reinforcement learning human feedback (RLHF) scenarios. However, this scheme only collects feedback on the overall output. When the simulation involves multiple virtual characters collaboratively generating conclusions, it cannot identify which character's judgment logic is flawed, making it impossible to pinpoint the specific character and its parameters that need adjustment during subsequent calibration. Option 2: Fake Content Detection Solution. This solution trains a machine learning model by analyzing the writing style characteristics of the content (such as the frequency of capitalization, repetition of specific words or topics) to identify and attribute the possible authors of fake comment content. This solution addresses the problem of post-event detection of content authenticity, processing user-generated comment text. It does not involve the behavioral calibration of virtual characters in a multi-role simulation system, nor does it involve version binding mechanisms for simulation records. Option 3: Crowdsourced Preference Aggregation Training Scheme. This scheme collects large-scale user preference comparison data on model candidate outputs, aggregates it, and uses it to train a reinforcement learning reward model. The training objective of this scheme is to optimize the generation strategy of the large language model itself. The aggregation dimension is usually based on model version or data batch, and it does not involve the joint aggregation of scenario-configured version and role version in multi-agent simulation scenarios, nor does it introduce a mandatory role attribution verification mechanism in the feedback submission process. Option 4: Feedback Recording Solution Without Version Binding. Some existing simulation evaluation systems record user feedback, but the feedback records are only associated with the current system state. When the role configuration or scenario configuration on which the simulation depends is upgraded or changed after the feedback is submitted, the feedback record can no longer accurately correspond to "which version of the configuration produced the fed-out simulation conclusion," causing the feedback data to gradually lose its calibration value as the system iterates. The aforementioned existing technologies all have the following drawbacks: (1) User feedback indicates a lack of mandatory attribution mechanism for specific roles in multi-role simulations, making it impossible to locate the specific objects that need calibration; (2) The feedback records are not fixedly bound to the role configuration and scene configuration version status when the simulation is generated. After the configuration is upgraded, the calibration value of the historical feedback decreases or becomes invalid. (3) There is a lack of a mechanism for jointly aggregating feedback data by scenario configuration version and role version, which makes it impossible to generate calibration priority signals for specific versions and specific roles. Summary of the Invention

[0003] Purpose of the invention The purpose of this invention is to provide a forced role attribution feedback calibration method and system for multi-role large language model simulation, so as to solve the technical problems in the prior art such as lack of role attribution feedback, disconnect between feedback records and version status, and lack of multi-dimensional aggregated calibration signals. Technical solution The core idea is to solidify the version status snapshot of the role library and scenario configuration when the simulation record is created, and to force the attribution of "unrealistic" feedback to the specific role that actually exists in the snapshot of that simulation record when the user submits feedback. This binds the feedback record to the exact version number of the role in that simulation, thereby supporting multi-dimensional aggregation by scenario configuration version, role, and feedback type, and producing priority signals that can be directly used for manual calibration decisions. The technical solution provided by this invention includes the following core mechanisms: Mechanism 1: Binding of Simulation Record Version Snapshots When a simulation is created, the system reads the version numbers of all roles in the current role library and the version numbers of the scene configurations used, and persists them as fixed fields along with the simulation record. Any subsequent version upgrades of the role library or scene configuration will not affect the historical version status bound to the created simulation record, ensuring that any historical simulation record can be accurately traced back to the configuration status it depended on when it was generated. Mechanism 2: Feedback Validation of Forced Role Attribution When a user marks a simulation output as "unrealistic," the system mandates that the user specify a particular role identifier in the feedback request and verifies that this role identifier must exist in the role version mapping set bound to that simulation record itself—the verification object is the actual version snapshot of that simulation, not a generalized list of roles, thus preventing attribution to roles unrelated to that simulation. Feedback requests that fail verification are rejected and do not proceed to subsequent recording processes. Mechanism 3: Idempotent Deduplication Based on User Dimension Feedback records use a combination of simulation identifier, user identifier, feedback type, and feedback target content as the idempotent key to avoid count inflation caused by repeated submissions from the same user, while retaining the independent feedback counts for each user to maintain the statistical significance of the group feedback signal. When a feedback record is written, the exact version number of the attributed role is simultaneously retrieved from the version snapshot bound to that simulation record and stored together. This ensures that even if the role library is subsequently upgraded, historical feedback records still accurately correspond to the role version at the time the conclusion addressed by the feedback was generated. Mechanism 4: Multi-dimensional aggregation of calibration signals Feedback data is grouped and counted based on four dimensions: scene configuration identifier, scene configuration version number, role identifier, and feedback type. The results are then sorted and a priority signal is output that directly indicates "which scene configuration, which version, which role, and which type of feedback problem is most concentrated," allowing for manual decision-making on whether and how to calibrate and adjust the behavior parameters of that role. Beneficial effects (1) Precise positioning of calibration objects: Compared with the overall scoring feedback scheme, the present invention makes each "unreal" feedback precisely point to a specific role, and the calibration decision can be directly located to the role that needs to be adjusted, avoiding unfounded parameter adjustments to irrelevant roles; (2) Version traceability of feedback data: Compared with feedback record schemes without version binding, the present invention ensures that the feedback record always corresponds to the exact role version at the time of generation. Even if the role library is upgraded multiple times, the historical feedback data still retains calibration value and does not become invalid due to version iteration. (3) Automatic generation of calibration priority signals: Compared with the practice of relying on manual review and feedback, the multi-dimensional aggregation mechanism of the present invention automatically generates a list of calibration candidates sorted by quantity, which significantly reduces the time cost required for manual screening of calibration objects. Attached Figure Description Figure 1 The flowchart for binding simulation record version snapshots shows how the role version and scene configuration version are fixed fields in the simulation record when the simulation is created. Figure 2 To enforce role attribution feedback verification flowcharts, this document demonstrates the role identifier verification process when submitting "inauthentic" feedback. Figure 3 The flowchart for writing feedback idempotent deduplication and version binding is provided. Figure 4 This is a schematic diagram of the multi-dimensional aggregated calibration signal generation mechanism, showing the process of grouping, counting, and sorting feedback data according to scenario configuration version, role, and feedback type. Figure 5 A comparative diagram illustrating the impact of version snapshot binding on feedback traceability shows that the version correspondence of historical feedback records remains unchanged before and after the character library upgrade. Detailed Implementation The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Example 1: A scenario where users label simulation conclusions as unrealistic and attribute them to specific roles. A simulation platform performed a multi-role simulation for a user regarding the event of "rising raw material costs." When creating the simulation record, the system reads the current role library status, generates a set of role version mappings (e.g., customer role version 3, finance role version 5, sales team role version 2), and reads the current version number (e.g., version 1.2) of the scenario configuration used (e.g., the "manufacturing" scenario package). Both are written as fixed fields into the simulation record. Upon completion of the simulation, it outputs simulation conclusions containing the game-theoretic judgments of each role. After reading the simulation results, the user felt that the judgment given by the finance role—"gross profit margin decreased by 15 percentage points"—was inconsistent with their actual business experience. Therefore, the user clicked the "This is not true" button on the front-end interface and selected "Finance Role" as the role corresponding to this judgment. The system received a feedback request with the feedback type being "Not True" and the role identifier in the feedback target information being "Finance Role". The system verified whether this role identifier existed in the role version mapping set bound to this simulation record—since the finance role did participate in this simulation, the verification passed. The system used the combination of (this simulation identifier, current user identifier, not true identifier, and normalized representation of feedback target information) as an idempotent key, determined that this key had not been recorded, and thus recorded the feedback. The system also retrieved the finance role's version number "5" from the version mapping set bound to this simulation record and stored it along with the feedback record. If the user mistakenly enters the role identifier in the feedback target information as another role that did not participate in this simulation (such as a role that does not exist in a certain scenario configuration), the system will verify that the role identifier is not in the role version mapping set bound to the simulation record and directly reject the feedback request to avoid generating invalid attribution data that cannot be located to the specific simulation basis. Example 2: Scenario where historical feedback remains traceable after a character library version upgrade. One month after the occurrence of Example 1, the simulation platform operators discovered, based on the calibration dashboard, that the financial role version 5 was frequently flagged as "inaccurate." They decided to adjust the financial role's behavioral parameters and release it as version 6. After the role library version upgrade, the system's current role version mapping (read when a new simulation is created) has been updated to financial role version 6. However, the historical simulation records generated in Example 1 and their associated feedback records still retain the role version field as version 5, unaffected by the role library version upgrade. Multi-dimensional aggregation calibration signal generation: When operators invoke the aggregated calibration signal generation function, the system reads all recorded feedback data and counts it using (scene configuration identifier, scene configuration version number, role identifier, feedback type identifier) ​​as the grouping key, resulting in the following sorted results (partial example): Manufacturing Scenarios Package v1.2 | Finance Role | Inaccurate Feedback | Count: 23 Manufacturing Scenarios Package v1.2 | Customer Roles | Inauthentic Feedback | Count: 6 Retail Scenarios Package v2.0 | Sales Team Roles | Path Selection Feedback | Count: 4 The sorting results clearly indicate that "the finance role under the manufacturing scenario package version 1.2 received the most inaccurate feedback." Based on this, operations personnel can prioritize reviewing whether there are systematic deviations in the behavior parameter settings of this role in version 5, without having to go through each original feedback record one by one. If the existing technical solution four (a feedback record solution without version binding) is adopted, when the finance role is upgraded to version 6, the historical feedback records will be unable to distinguish whether the evaluation was generated for version 5 or version 6. The aggregated statistical results will incorrectly merge and count the feedback from different versions, causing the operations staff to misjudge whether the problem of version 5 has been reported in the current version 6. The solution of this invention, because the feedback records are fixedly bound to the exact role version at the time of generation, can accurately distinguish the feedback received by version 5 and version 6 in the aggregation results.

Claims

1. A version snapshot binding method for multi-role large language model simulation, characterized in that, Includes the following steps: (1) Receive a simulation creation request, wherein the simulation creation request specifies the scene configuration identifier to be used; (2) Read the current version number of all characters in the current character library and generate a mapping set of character identifiers to version numbers; (3) Read the current version number of the scene configuration corresponding to the scene configuration identifier; (4) Create a simulation record, write the mapping set described in step (2) and the scene configuration version number described in step (3) as fixed fields into the simulation record and persist them, so that from the moment the simulation record is created, the role version mapping set and scene configuration version number bound to it will not change with subsequent version changes of the role library or scene configuration. (5) Perform multi-role large language model simulation, generate simulation conclusions, and associate and store the simulation conclusions with the simulation records.

2. The method according to claim 1, characterized in that, The mapping set from role identifier to version number mentioned in step (2) covers all roles in the current role library, and is not limited to the subset of roles actually called in this simulation.

3. The method according to claim 1, characterized in that, Also includes: In response to the query request, the fixed set of character version mappings and scene configuration version numbers in the simulation record are returned, so that the simulation can be restored to the character configuration and scene configuration state on which it was generated based on the fixed fields at any subsequent time.

4. A forced role attribution feedback verification and idempotent deduplication method for multi-role large language model simulation, applicable to the simulation record with bound version snapshots as described in claim 1, characterized in that, Includes the following steps: (1) Receive feedback request, the feedback request includes simulation identifier, feedback type identifier and feedback target information, the feedback type identifier is taken from a preset finite enumeration set including false identifier; (2) When the feedback type identifier is an untrue identifier, extract the role identifier from the feedback target information and verify whether the role identifier exists in the role version mapping set of claim 1 bound to the simulation record corresponding to the simulation identifier; if it does not exist, reject the feedback request and return the verification failure result; (3) After the verification is passed, the combination of the simulation identifier, the feedback source user identifier, the feedback type identifier and the normalized representation of the feedback target information is used as the idempotency key to determine whether the idempotency key has been recorded; if it has been recorded, the duplicate identifier is returned and the record is not repeated. (4) When the idempotent key is not recorded, record the feedback. The recorded content includes the role identifier in the feedback target information and the version number taken from the role version mapping set bound to the simulation record according to the role identifier; mark the idempotent key as recorded.

5. The method according to claim 4, characterized in that, The idempotent key in step (3) includes the user identifier of the feedback source, so that different users count independently for the same simulation, the same feedback type, and the same feedback target information, while duplicate submissions by the same user are deduplicated.

6. The method according to claim 4, characterized in that, The preset finite enumeration set also includes path selection identifiers and parameter rerun identifiers, which correspond to the user's selection of candidate decision paths given by the simulation and the user's request to re-execute the simulation with adjusted parameters, respectively.

7. A method for generating multi-dimensional aggregated calibration signals based on scene configuration version and role version, applicable to the recorded feedback data as described in claim 4, characterized in that, Includes the following steps: (1) Read the feedback data set recorded by the method of claim 4, wherein each record in the feedback data set contains the scene configuration identifier of the simulation record to which it belongs, the scene configuration version number of claim 1, the role identifier in the feedback target information, and the feedback type identifier; (2) The feedback data set is grouped and counted using a combination of scene configuration identifier, scene configuration version number, role identifier and feedback type identifier as the grouping key; (3) Sort the group count results in descending order and output the sorted group count list as the priority sorting signal for calibration candidates.

8. The method according to claim 7, characterized in that, Also includes: Receive the filtering conditions of the specified scene configuration identifier. In step (2), only the feedback data records whose scene configuration identifier matches the filtering conditions are grouped and counted.

9. A multi-role large language model simulation feedback calibration system for implementing the methods of claims 1 to 8, characterized in that, include: - Version snapshot binding module: responsible for reading the role library version and scene configuration version, generating the mapping set, and persistently binding the simulation record as described in claim 1; - Feedback Receiving and Attribution Verification Module: Responsible for performing the feedback type verification, mandatory role attribution verification for false types, and verification failure rejection as described in claim 4; - Idempotent deduplication module: responsible for performing the idempotent key construction, duplicate judgment and version number binding and writing of the feedback record as described in claim 4; - Aggregate calibration signal module: responsible for performing the multi-dimensional group counting and sorting output as described in claim 7.

10. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images