Subject evaluation method and device based on heterogeneous artificial intelligence platform and storage medium

Abstract

The present disclosure relates to the technical field of artificial intelligence, and provides a subject evaluation method and device based on a heterogeneous artificial intelligence platform and a storage medium. An evaluation request carrying a target subject and a scene label is obtained and mapped to a lane matching a resource constraint; after obtaining a slot, an execution attempt is formed passively, and an evidence set including an answer, a source, and execution evidence is collected. A semantic extraction model is used to extract candidate semantics from the original answer, and after verification, formal semantics belonging to the target subject are obtained. The formal semantics of the same answer generate answer semantics, which are combined with the evidence set to form an execution result of the platform task. Effective answer semantics are screened and aggregated according to observed problems to obtain cross-platform semantic conclusions; execution results are collected, and output target subject performance evaluation signals and structured evaluation conclusions in overall coverage, scene performance, platform consistency, and the like are analyzed and output, and the cross-platform semantic conclusions are used to provide explanation clues. The present disclosure has the advantages of bounded parallel scheduling of heterogeneous platforms, evidence trace, conclusion explanation, and semantic verification to avoid illusions.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence analysis technology, and in particular to subject evaluation methods, devices and storage media based on heterogeneous artificial intelligence platforms. Background Technology

[0002] Traditional brand AI understanding assessments largely rely on manual extraction and statistical analysis of publicly available text information. This requires manually sifting through massive amounts of text material, identifying instances where the target entity mentions the information, its location, and feedback differences across various business scenarios, manually verifying evidence fragments, and compiling consensus viewpoints across different sources. The overall process is cumbersome, has limited sample coverage, long evaluation cycles, and low iteration efficiency, making it difficult to conduct multi-business scenario and multi-dimensional brand AI understanding assessments in batches and at scale. Summary of the Invention

[0003] In view of the shortcomings of the prior art described above, the purpose of this disclosure is to provide a subject evaluation method, device and storage medium based on a heterogeneous artificial intelligence platform to solve the problems in the related technology.

[0004] The first aspect of this disclosure provides a subject evaluation method based on heterogeneous artificial intelligence platforms, comprising: acquiring at least one observation question containing target subject information and associated scene tags, and evaluation requests from multiple observation platforms; determining the platform task for each observation question to be evaluated on one observation platform, wherein the multiple observation platforms are different generative artificial intelligence platforms; acquiring a platform capability profile of each observation platform and mapping it to a lane matching resource constraints, wherein the resource constraints include a capacity representing the number of concurrent execution slots of the lane; creating an execution attempt for the platform task in a queuing state waiting for a slot in the mapped lane; in response to the platform task obtaining a slot in the lane, updating the queuing state of the execution attempt to an execution state and creating an execution context to submit the observation question to each observation platform under the execution budget constraint of the slot and collecting platform responses to form an evidence set, wherein the evidence set includes response evidence, source evidence, and execution evidence, wherein the response... The evidence includes the original answer and the execution process information of the platform task recorded by the execution evidence record. Candidate semantics and candidate subjects are extracted from the original answer through a semantic extraction model, and the formal semantics that correctly belong to the target subject are determined through preset semantic verification. Answer semantics are formed based on the formal semantics of the same original answer, and the answer semantics and the evidence set form the execution result of the platform task. Valid answer semantics are selected based on a set of execution results, and the valid answer semantics associated with each observation question are aggregated to form a cross-platform semantic conclusion. Valid evaluation samples are selected based on a set of execution results, and the evidence set and scene labels of the observation questions in the valid evaluation samples are collected. Based on this, the strength evaluation signal representing the target subject's performance in the dimensions of overall cognitive coverage, scene performance, and platform consistency is obtained. This signal is used to form a structured evaluation conclusion and is output together with the cross-platform semantic conclusion. The cross-platform semantic conclusion is used as an explanatory clue for the performance strength evaluation signal.

[0005] In an embodiment of the first aspect, mapping the platform capability profile to a lane that matches resource constraints includes: mapping the platform capability profile to a lane that matches the interaction capabilities, including at least one of the following: if the observation platform supports API interface access, then it is mapped to a first type of lane using API access; if the observation platform allows independent backend webpage access, then it is mapped to a second type of lane using backend webpage access; if the observation platform requires foreground visibility / interaction, then it is mapped to a third type of lane that allows foreground visibility / interaction to access the observation platform.

[0006] In an embodiment of the first aspect, the platform capability profile further includes login or session requirements, and the method further includes: determining whether the observation platform meets the platform task execution conditions and scheduling priority based on the login or session requirements.

[0007] In an embodiment of the first aspect, the platform capability profile further includes a timeout policy; the method further includes: in response to obtaining a slot, recording the start time and calculating the execution duration of the platform task accordingly; and checking whether the execution duration has timed out based on the platform execution budget determined by the timeout policy of the observation platform corresponding to the platform task.

[0008] In an embodiment of the first aspect, the subject evaluation method further includes: recording the queuing time of the platform task; and performing congestion detection on the lane based on the interval between the queuing time and the start time when the execution attempt of the platform task is created.

[0009] In an embodiment of the first aspect, the platform capability profile further includes a timeout policy; the method further includes: determining the number of execution waves corresponding to each lane based on the number of platform tasks allocated to each lane in a batch of platform tasks and the capacity of the lane; determining the upper limit of queuing time for each lane based on the number of execution waves corresponding to each lane and the single-wave execution budget; wherein the single-wave execution budget is determined by the maximum value among the platform execution budgets of the platform tasks allocated to the lane; determining the global queuing time upper limit for each lane based on the maximum value among the queuing time upper limits of each lane; setting a watchdog based on the global queuing time upper limit, the watchdog being used to accumulate the execution time of the batch of platform tasks, and stopping the continued queuing of platform tasks that have not yet started execution when the execution time reaches the global queuing time upper limit.

[0010] In an embodiment of the first aspect, the main evaluation method further includes: setting an initial index number associated with each observation platform based on the initial order of the multiple observation platforms in the evaluation request, and binding the initial index number of each result slot of a group of platform tasks belonging to the same observation problem to the initial index number of the corresponding platform in the group; and storing the execution result of each platform task in the corresponding result slot based on the initial index number to restore the index order consistent with the initial order.

[0011] In an embodiment of the first aspect, the answer evidence, source evidence, and execution evidence each include an answer state, source state, and execution state that are independently determined and associated through tracing information. The method further includes: setting the answer state to available, partially available, or unavailable based on whether the collected original answer is complete, partially valid, or invalid; setting the source state to complete, partially successfully collected, failed collected, or not supported by the platform based on the different levels of completeness of the collected source evidence; and setting the execution state to successful or failed based on whether the platform task is executed successfully or failed. The failed execution state is associated with reason information.

[0012] In the first aspect of the embodiment, the step of forming answer semantics based on the formal semantics of the same original answer includes: setting the semantic state of the generated answer semantics to available, partially available, or unavailable according to the coverage of formal semantics in each candidate semantics of the same original answer; the step of filtering effective answer semantics based on a set of the execution results and aggregating the effective answer semantics associated with each observation question includes: filtering answer semantics with a semantic state of available or partially available from the various answer semantics associated with the observation question; and aggregating the filtered effective answer semantics to form the cross-platform semantic conclusion.

[0013] In an embodiment of the first aspect, the cross-platform semantic conclusion includes at least one of consensus terms, divergence terms, and common omission terms. The consensus terms / divergence terms are generated based on information extracted from semantic consensus / semantic divergence between the semantic responses of at least two of the observation platforms. The common omission terms are generated based on information extracted from situations where the semantic responses of at least two of the observation platforms omit the same preset content of the target subject.

[0014] In the first aspect of the embodiment, the subject evaluation method further includes: selecting only the latest execution result from the execution results obtained from multiple execution attempts corresponding to the same platform task to form the cross-platform semantic conclusion and the structured evaluation conclusion.

[0015] In an embodiment of the first aspect, the step of filtering valid evaluation samples based on the evidence set in a set of execution results, and collecting the evidence set of observation questions and scene labels in the valid evaluation samples to analyze and obtain evaluation signals characterizing the strength of the target subject's performance in the dimensions of overall cognitive coverage, scene performance, and platform consistency includes: determining valid evaluation samples based on the availability of the answer evidence indications; obtaining an overall coverage signal representing the maximum loss value of the target subject when switching between different recommendation rankings based on the visibility results of the answer evidence extracted by rules in the valid evaluation samples; obtaining a scene breadth signal representing the strength / weakness of the overall recommendation intention of multiple observation platforms for the target subject in different scenarios based on the results collected according to the observation question labels corresponding to the visibility results of the answer evidence extracted by rules in the valid evaluation samples; obtaining a platform consistency signal representing the discreteness of the target subject's performance on each observation platform in the same question dimension based on the visibility results of the answer evidence extracted by rules in the valid evaluation samples; and determining a main signal and an auxiliary signal based on the signal strength of the overall coverage signal, scene breadth signal, and platform consistency signal; wherein the main signal corresponds to the main conclusion in the structured evaluation conclusion.

[0016] In an embodiment of the first aspect, determining the primary and secondary signals based on the signal strength values ​​of the overall coverage signal, scene breadth signal, and platform consistency signal includes: screening qualified signals based on threshold conditions including the minimum number of effective platforms, the minimum number of effective scenes, and the minimum number of effective evaluation samples; screening candidate signals among the qualified signals based on the condition that the intensity value reaches a significant threshold; determining the primary signal among the candidate signals based on signal strength comparison; if the difference in intensity values ​​among the multiple candidate signals with the highest signal strength values ​​is close to or below a threshold, determining the primary signal among the multiple candidate signals based on the condition that at least one of the number of effective platforms, the number of effective scenes, and the number of effective evaluation samples covered by the candidate signal tends to be higher, and the remaining candidate signals are secondary signals.

[0017] A second aspect of this disclosure provides a subject evaluation device based on a heterogeneous artificial intelligence platform, comprising: an input module, configured to acquire at least one observation problem including target subject information and associated scene tags, and evaluation requests from multiple observation platforms, and determine a set of platform tasks for each observation problem to be evaluated on one observation platform, wherein the multiple observation platforms are different generative artificial intelligence platforms; a mapping module, configured to acquire a platform capability profile of each observation platform and map it to a lane with resource constraints, wherein the resource constraints include a capacity representing the number of concurrent execution slots of the lane; and a task scheduling module, configured to create an execution attempt for the platform task in a queuing state waiting for a slot in the mapped lane, and in response to the platform task obtaining a slot in the lane, update the queuing state of the execution attempt to an execution state and create an execution context for execution in the slot. The system executes platform tasks under computational constraints and obtains a set of execution results, including submitting observation questions to various observation platforms and collecting platform responses to form an evidence set, as well as forming answer semantics through preset semantic verification; a semantic aggregation module is used to filter valid answer semantics based on a set of execution results, aggregate valid answer semantics associated with each observation question, and form cross-platform semantic conclusions for each observation question; a subject evaluation module is used to filter valid evaluation samples based on a set of execution results, collect the evidence set and scene labels of the observation questions in the valid evaluation samples, and analyze them to obtain evaluation signals representing the strength of the target subject's performance in the dimensions of overall cognitive coverage, scene performance, and platform consistency, so as to form structured evaluation conclusions and output them together with the cross-platform semantic conclusions, wherein the cross-platform semantic conclusions are used as explanatory clues for the performance strength evaluation signals.

[0018] The third aspect of this disclosure may also provide a computer-readable storage medium storing a computer program or instructions, which, when executed, perform the subject evaluation method as described in any one of the first aspects.

[0019] As described above, this disclosure relates to the field of artificial intelligence technology, providing a method, apparatus, and storage medium for subject evaluation based on a heterogeneous artificial intelligence platform. The process involves acquiring evaluation requests carrying target subject and scene tags, mapping them to lanes that match resource constraints, and then lazily forming execution attempts after obtaining a slot, collecting an evidence set containing answers, sources, and execution evidence. A semantic extraction model is used to extract candidate semantics from the original answers, which are then verified to obtain formal semantics belonging to the target subject. Answer semantics are generated from the formal semantics of the same answer, and combined with the evidence set to constitute the execution result of the platform task. Valid answer semantics are filtered and aggregated according to the observed questions to obtain cross-platform semantic conclusions. The execution results are collected, and the output includes evaluation signals and structured evaluation conclusions on the target subject's performance in three dimensions: overall cognitive coverage, scene performance, and platform consistency. The cross-platform semantic conclusions are used to provide explanatory clues. This disclosure combines bounded parallel scheduling of heterogeneous platforms, evidence tracking, conclusion explanation, and semantic verification to avoid illusions. Attached Figure Description

[0020] Figure 1 A flowchart illustrating the subject evaluation method for the response of an artificial intelligence platform in one embodiment of this disclosure is shown.

[0021] Figure 2 A schematic diagram illustrating an application scenario according to one embodiment of this disclosure is shown.

[0022] Figure 3 This illustration shows a schematic diagram of the phased process of platform task execution in one embodiment of the present disclosure.

[0023] Figure 4 A schematic diagram illustrating the specific process of step S105 in one embodiment of this disclosure is shown.

[0024] Figure 5 A schematic diagram illustrating the specific process of step S106 in one embodiment of this disclosure is shown.

[0025] Figure 6 A schematic diagram illustrating the specific method flow of step S1065 in one embodiment of this disclosure is shown.

[0026] Figure 7 A schematic diagram of data flow is shown in one embodiment of this disclosure.

[0027] Figure 8 A schematic diagram of the main evaluation device based on a heterogeneous artificial intelligence platform is shown in one embodiment of this disclosure.

[0028] Figure 9 A schematic diagram of the circuit structure of a computer device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0029] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed herein. This disclosure can also be implemented or applied through other different specific embodiments, and various details in this disclosure can also be modified or changed according to different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this disclosure can be combined with each other.

[0030] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, so that those skilled in the art to which this disclosure pertains can readily implement it. This disclosure may be embodied in many different forms and is not limited to the embodiments described herein.

[0031] In the description of this disclosure, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or a group of embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this disclosure, as well as the features of those different embodiments or examples.

[0032] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a set" means two or more, unless otherwise explicitly specified.

[0033] For the purpose of clarity, devices unrelated to the description are omitted, and the same or similar components throughout the specification are given the same reference numerals.

[0034] Throughout this specification, when it is said that a device is "connected" to another device, this includes not only "direct connection" but also "indirect connection" by placing other components in between. Furthermore, when it is said that a device "comprises" a certain constituent element, unless otherwise stated otherwise, this does not exclude other constituent elements, but rather implies that other constituent elements may be included.

[0035] While the terms first, second, etc., are used in some examples herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, descriptions such as first interface and second interface, etc. Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, step, operation, element, module, item, kind, and / or group, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, modules, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition occur only when combinations of elements, functions, steps, or operations are inherently mutually exclusive in some way.

[0036] The technical terms used herein are for reference only to specific embodiments and are not intended to limit the scope of this disclosure. The singular form used herein includes the plural form unless the statement explicitly states otherwise. The word "comprising" as used in this specification means to specify a particular feature, area, step, operation, element, and / or component, and does not exclude the presence or addition of other features, areas, steps, operations, elements, and / or components.

[0037] Although not explicitly defined, all terms, including technical and scientific terms used herein, are associated with the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms defined in commonly used dictionaries are further interpreted as being associated with meanings consistent with relevant technical literature and the message of the present disclosure, and should not be over-interpreted as ideal or overly formulaic unless otherwise defined.

[0038] Traditional brand AI understanding assessments largely rely on manual extraction and statistical analysis of publicly available text information. This requires manually sifting through massive amounts of text material, identifying instances where the target entity mentions the information, its location, and feedback differences across various business scenarios, manually verifying evidence fragments, and compiling consensus viewpoints across different sources. The overall process is cumbersome, has limited sample coverage, long evaluation cycles, and low iteration efficiency, making it difficult to conduct multi-business scenario and multi-dimensional brand AI understanding assessments in batches and at scale.

[0039] In view of this, this disclosure provides a subject evaluation method based on heterogeneous artificial intelligence platforms. It utilizes the same batch of observation questions to generate answers on multiple generative AI-type observation platforms for semantic extraction and analysis, effectively improving efficiency compared to traditional analysis methods. Furthermore, considering the potential heterogeneity and varying capabilities of the observation platforms, a parallel scheduling mechanism for platform tasks is designed to adapt to the capabilities of different observation platforms, improving acquisition efficiency and avoiding resource congestion. Moreover, while generating cross-platform structured evaluation conclusions about the target subject based on the answers, the candidate semantics extracted from the original answers are verified to avoid illusion problems. The verified formal semantics are used to form cross-platform semantic conclusions that complement the structured evaluation conclusions, thereby providing an explanatory index for the structured evaluation conclusions. Thus, it solves various problems associated with manual evaluation in related technologies.

[0040] like Figure 1 The diagram shown illustrates a flowchart of a subject evaluation method for an artificial intelligence platform response in one embodiment of this disclosure.

[0041] exist Figure 1 The process includes:

[0042] Step S101: Obtain at least one observation problem containing target subject information and associated scene tags, as well as evaluation requests from multiple observation platforms, and determine the platform task for each observation problem to be evaluated on one observation platform.

[0043] In some embodiments, the target entity is a brand, product, organization, technology subject, or other entity to be analyzed. For merchants, the target entity can be their own brand, product, organization, technology subject, or other entity to analyze brand awareness. Alternatively, the target entity can be a competitor's brand, product, organization, technology subject, or other entity to analyze competitor awareness. The target entity information may include the target entity's official name and a list of aliases, such as "XX", "XX software", "XX APP", etc.

[0044] In some embodiments, each observation question may be assigned a unique question ID (question_id) for identification and subsequent tracing. In some embodiments, the number of observation questions may be a batch of multiple questions.

[0045] It is understandable that the observation questions may not directly mention the target entity's name, but rather indirectly observe whether the observation platform mentions the entity, in order to evaluate the target entity's AI understanding dimension of cognitive information. For example, observation questions such as "Please recommend a card trading platform," "Where can I buy cheap cards," and "Which platforms have low transaction fees" can be set, and the platform's answers can be used to indirectly determine its tendency to mention and recommend specific entities. In addition, each observation question can be associated with a business scenario tag, such as price inquiry, card purchase, high-price transaction, etc. (this is just an example, not limited to this), for subsequent analysis and evaluation results by scenario dimension.

[0046] In some embodiments, the observation platform is a generative artificial intelligence platform (AIGC), such as common large models like GPT, Deepseek, and Doubao. Multiple observation platforms may be different generative artificial intelligence platforms. These platforms possess human-computer dialogue capabilities. The dialogue principle of the generative artificial intelligence platform is explained in detail below. The generative artificial intelligence platform can obtain input information submitted by the user, including but not limited to textual queries, instructions, question descriptions, and contextual dialogue history. For example, after receiving the input information, the generative artificial intelligence platform parses and understands it, extracting semantic intent, key entities, constraints, and dialogue context relationships. Based on the parsed semantics, logical reasoning is performed, including question decomposition, intent identification, retrieval of relevant knowledge, and logical deduction, resulting in corresponding answer logic and answer elements. Then, based on the answer logic and answer elements, text is organized, completing sentence splicing, sentence structure adjustment, and expression polishing, forming the answer content fed back to the user.

[0047] You can refer to them together. Figure 2 The diagram shown illustrates an application scenario according to one embodiment of this disclosure. Figure 2 In this embodiment, the main evaluation method can be applied to an evaluation system 201. In some embodiments, the evaluation system 201 can be implemented on a computer device. The computer device can be implemented as a server or a server group, or it can be implemented as a user terminal, such as a computer, laptop, tablet, etc. The evaluation system 201 is connected to each observation platform 1 to n via network communication. The network can be wired, wireless, or a hybrid of both.

[0048] The evaluation system 201 can respond to the evaluation request by submitting observation questions related to the target subject to various observation platforms 1 to n. Utilizing the response generation capability of the generative AI platform, it obtains responses 1 to n generated by each observation platform 1 to n to the observation questions. Furthermore, based on the responses from each observation platform 1 to n, its understanding of the tested platform can be analyzed across platforms, such as visibility (e.g., mentions and recommendations), thereby generating an evaluation report. The evaluation system 201 can also display the evaluation report on the front-end interface of the observation terminal 202 (e.g., the terminal that initiated the evaluation request) upon user request.

[0049] In some embodiments, each platform task corresponds to the execution of an observation process on an observation platform for an observation problem. For example, such as... Figure 3 As shown, the execution process of each platform task can include stages such as request, collection, analysis, and storage. Simply put, in the request stage, a request can be established with the observation platform or an interactive interface can be accessed to submit the observation question to the platform. In the collection stage, the original answer, as well as source evidence (such as the observation platform and its related interfaces) and execution evidence (i.e., execution status information) of the original answer can be collected from the observation platform. Source evidence is used to determine whether the source of the answer is traceable and whether the content is complete, providing source-level support for the reliability of the answer. Execution evidence is used to determine whether the execution process was completed normally (such as whether it timed out or whether any anomalies occurred), providing process-level support for the reliability of the answer. In the analysis stage, analysis can be performed based on the original answer, specifically including two analysis paths: semantic analysis and evaluation analysis. The semantic analysis extracts semantic information from the original answer, representing the "viewpoint" of a single observation platform in response to the observation question. This semantic information can be used as the basis for subsequently forming a multi-platform overall viewpoint on the target subject across multiple platforms.

[0050] The evaluation analysis is based on observable facts extracted from the original answers by a single observation platform, such as mentions and recommendations of the target subject. These observable facts include one or more of the following: relevant snippets about the target subject, whether the target subject was mentioned, whether the target subject entered the Top 3, whether the target subject was the first recommended answer, and the recommending role of the target subject. These observable facts can be used as the basis for a subsequent cross-platform overall evaluation of the target subject's "performance."

[0051] In some embodiments, optionally, a scoring phase may be included after the analysis phase for single-platform scoring based on the target subject's performance on a single observation platform. As an optional example, the single-platform scoring may include visibility scoring and health scoring.

[0052] As an example, the visibility score and health score can both be a weighted result of the scores of one or more sub-items.

[0053] In optional examples, the visibility score can be a weighted result of sub-scores for multiple items, including intent matching, visibility, recommendation, accuracy, and content gap. Intent matching characterizes the degree to which the answer responds to the intent of the observed question, such as matching the intent type of the observed question (definition, comparison, recommendation, etc.) with the actual response type of the answer; or using semantic similarity to calculate the alignment between the question and the answer. Visibility characterizes the visibility of the target subject in the answer, and can be calculated based on its position, frequency, and proportion of content. For example, it involves statistically analyzing the position, frequency, and proportion of the target subject in the answer; the earlier the position, the higher the frequency, and the longer the content, the higher the visibility. Recommendation characterizes the strength of the recommendation of the target subject, and can be identified based on the ranking of explicit recommendation statements, such as identifying recommendation statements in the answer ("recommend," "suggest," "preferred," etc.), distinguishing between explicit recommendations (directly stating "Recommend A") and implicit recommendations ("A performs excellently"), and assigning scores according to the strength of the recommendation. Accuracy characterizes the correctness of information about the target subject. For example, it compares key facts about the target subject in the answer with a pre-defined benchmark knowledge base or trusted sources to calculate factual consistency; or it checks for obvious contradictory statements. Content gap characterizes the degree to which the answer covers the key attribute dimensions of the target subject. For example, it predefines key attribute dimensions of the target subject (such as function, advantages, application scenarios, etc.) and checks whether the answer covers these dimensions; the lower the coverage, the larger the gap, and the lower the score. Some or all of the above sub-items can be selected according to actual evaluation needs, and a comprehensive score can be obtained through weighted summation. As an example, the weights of intent matching, visibility, recommendation, accuracy, and content gap can be similar, with visibility relatively higher and content gap relatively lower based on importance. This can be configured according to needs rather than being a limitation.

[0054] In an optional example, the health score can be a weighted result of self-scoring in multiple sub-items such as attitude, accuracy, reputational risk, and information completeness. The attitude corresponds to the polarity of the response's sentiment towards the target subject—whether positive, neutral, or negative. This polarity can be obtained through sentiment analysis, identifying sentimental words related to the target subject in the response ("excellent," "leading" vs. "problematic," "insufficient"), calculating the proportion of positive / neutral / negative statements, and then determining the overall sentiment score. Optionally, this polarity information can also be obtained during semantic analysis (e.g., rule extraction or semantic extraction model extraction) and incorporated into the semantics of the response. The accuracy of the information about the target subject can be determined through fact-checking, such as comparing key factual statements about the target subject in the response with a benchmark knowledge base or trusted sources to calculate the factual consistency rate; and checking for obvious errors, outdated information, or contradictory statements. The reputational risk assessment, concerning whether the answer contains content that could potentially damage the target entity's reputation, can be obtained through risk identification and detection, such as detecting negative associations (association with controversial events / negative entities), inappropriate expressions (discriminatory / insulting language), and false information (content that may lead to misunderstanding). The more risk items, the lower the score. The information completeness assessment, concerning whether the answer adequately covers the target entity's key information, can be obtained through, for example, dimensional coverage detection. Predefined key attributes of the target entity (such as core functions, advantages, application scenarios, comparisons with competitors, etc.) can be used to check whether the answer covers these dimensions; the higher the coverage, the higher the score. As an example, the weights of attitude, accuracy, reputational risk, and information completeness can decrease in order of their importance, and can be configured according to needs rather than being a limitation.

[0055] It should be noted that the multiple dimensions of single-platform scoring given in this embodiment are only examples and can be selected and changed according to different analysis fields. The weight settings can also be changed according to actual needs and are not limited to the above.

[0056] In some embodiments, for cognitive observation questions that inherently require discussion of the target subject, the system does not provide additional visibility rewards for the target subject being forcibly mentioned. In such scenarios, the recommendation rating is tied to the question's semantics, meaning it is only counted in natural mentions or positive recommendation scenarios. For example, if the question directly asks "Is AAA reliable?" regarding the target subject AAA, then mentions of AAA should not be counted, thus incorrectly increasing the number of times AAA is mentioned.

[0057] In some embodiments, to make the quantitative scoring interpretable and verifiable, the basic facts of a single sample, rule hit status, model-assisted labels, combined conclusions, and confidence levels can be saved, along with sub-items, weights, capping reasons, sample range, and snapshot time. This allows the aggregated score to be traced back to the specific observation question, the observation platform's response, and the source status. Thus, any scoring result can reconstruct the sample range, rule evidence, and parameter configuration upon which its calculation was based, meeting audit and recalculation requirements.

[0058] In some embodiments, during the saving phase, various types of information and intermediate results generated during the execution of the current platform task can be persistently saved to the corresponding result slot for subsequent cross-platform aggregation.

[0059] It is understandable that, since each "problem-platform" corresponds to a platform task, if the evaluation request contains n observation problems and m observation platforms, then submitting each of the n observation problems to the m observation platforms can form m×n platform tasks. Therefore, a corresponding number of result slots can be allocated to each platform task for use in the storage phase. These result slots are storage resources allocated to the corresponding platform tasks, such as arrays.

[0060] In some embodiments, a master task can be created for each evaluation request and assigned a master task identifier (task_id), and a platform task identifier (task_item_id) can be assigned to each created platform task. In some embodiments, to distinguish different observation questions and to trace sources, each observation question can be assigned a question_id. In some embodiments, to distinguish different observation platforms and to trace sources, each observation platform can be assigned a platform identifier (platform). These identifiers can be associated based on the establishment of upstream and downstream relationships between the corresponding objects.

[0061] It is understandable that, since the various observation platforms may belong to generative AI platforms from different vendors, there are significant differences in their platform capabilities, such as interfaces, browser sessions, foreground interactions, login status, waiting behavior, and reference presentation methods. This disclosure refers to these differences as "heterogeneity." For example, some observation platforms may allow API access. Some web-based platforms further differentiate between allowing sessions with headless background processes and requiring foreground visibility and focus interaction. Additionally, some platforms require maintaining a valid login session and cookie context to obtain complete response output. The capabilities, constraints, and execution behaviors of these various observation platforms are not uniform. If batch platform tasks are executed concurrently in a unified manner, it can lead to scheduling imbalances such as contention for browser or account resources.

[0062] Therefore, corresponding to the execution of platform tasks for each "problem-observation platform", the method in this embodiment can provide a scheduling strategy adapted to the platform capabilities of different observation platforms, thereby avoiding the aforementioned scheduling imbalance problem.

[0063] Step S102: Obtain the platform capability profile of each observation platform and map it to the lane that matches the resource constraints.

[0064] In some embodiments, a platform capability profile may include one or more combinations of access methods, interactive resources, login / session requirements, and timeout policies. In some embodiments, the platform capability profile may be formed by static configuration (e.g., based on publicly available configuration information from the observation platform), dynamic detection (e.g., detecting and dynamically generating the profile based on the observation platform's interface responses and browser behavior), or a combination of both. When using dynamic detection to generate the platform capability profile, the "platform capability profile" is no longer just a configuration table, but a technical mechanism that adapts to changes in the observation platform, allowing for a more accurate determination of the current actual capabilities of the observation platform compared to static configuration.

[0065] The login or session requirement is used to determine whether the platform has the conditions for execution and its scheduling priority. For example, if no login is required, the corresponding item can be recorded as "none".

[0066] "Lane" is a logically isolated scheduling domain, with each lane maintaining its own task queue and capacity. A lane can be associated with at least one resource constraint, such as access capability, interaction resources, login session requirements, capacity, and execution budget. These resource constraints are used to determine observation platforms with compatible platform capability profiles, ensuring that platform tasks corresponding to those platforms are allocated to that lane for execution. Resources in different lanes are independent of each other, preventing platform tasks belonging to observation platforms with different platform capability profiles from competing for browser or account resources.

[0067] "Capacity" indicates the number of execution slots that can be occupied simultaneously in the same lane. Each "problem-platform" task occupies one slot, and after completion, it is released and the next platform task in the queue is processed. For example, a lane capacity of 2 means that the lane has two concurrent slots, allowing two platform tasks to be executed in parallel.

[0068] The number of slots currently available for execution in a lane is controlled by a lane semaphore. The lane semaphore is a concurrency control counter whose count corresponds to changes in the number of currently available execution slots for a lane. A platform task can only occupy a slot and execute after obtaining permission from the lane semaphore; the permission is released upon completion.

[0069] In some embodiments, an adaptation lane can be selected based on hard constraints such as access methods and interactive resources in the platform capability profile, and then the execution method within that lane can be configured using session requirements and timeout policies.

[0070] As an example, the access methods may include API access, headless backend webpage access, and foreground visual interface access. Correspondingly, lane types with different resource constraints include first-type lanes, second-type lanes, and third-type lanes.

[0071] Specifically, if the observation platform supports API access, it is mapped to the first type of lane using API access. The API lane is associated with the interface session, rate, and response budget, and does not occupy the foreground window. This type of lane is suitable for observation platforms that have provided structured API interfaces. Observation questions are submitted and answers are obtained through interface calls, and the execution process is not affected by the browser window state.

[0072] If the observation platform allows independent background webpage access, it is mapped to the second type of lane using the background webpage access method. The background webpage lane is associated with the browser session, page stabilization waiting, and source collection. The second type of lane is suitable for observation platforms that support webpage interaction but do not require foreground window focus, and can independently complete page loading, question input, and answer collection in the background.

[0073] If the observation platform requires foreground visibility / interaction, it is mapped to the third type of lane that allows access to the observation platform using foreground visibility / interaction. The third type of lane is suitable for observation platforms that require a user-visible window or foreground interactive focus.

[0074] In some embodiments, the capacity of different types of lanes can be the same or different. For example, the capacity of the first type of lane is equal to that of the second type of lane, and greater than that of the third type of lane. This is because the third type of lane requires a visible real browser window and cannot be opened multiple times, while the first and second types of lanes are background lanes, requiring less and less browser resources, respectively. In a further example, the capacities of the first, second, and third types of lanes are 2, 2, and 1, respectively. It should be noted that the capacity can be configured according to one or more of the following: device, account, platform restrictions, and compliance requirements, and is not limited to this example.

[0075] Each lane provides parallel slots according to its capacity, and each lane executes in parallel relatively independently. Platform tasks queue in their respective lanes waiting for an available slot, and begin execution once an available slot is obtained. Different lanes do not interfere with each other. For example, the API lane and the backend webpage lane can execute their respective concurrent tasks simultaneously, while the visible webpage lane strictly protects the frontend interaction from being executed sequentially.

[0076] Therefore, each lane can execute platform tasks in parallel within the same round, up to the sum of its capacity, thus limiting the peak concurrency of the acquisition phase to an upper limit determined by the capacity of each lane. For example, if one observation problem is input from eight observation platforms, it can be split into eight platform tasks. Three platforms support API interfaces, and their corresponding tasks are mapped to the first type of lane (capacity 2, executing two platform tasks simultaneously, with the remaining one queued). Four platforms allow access to the background webpage and enter the background webpage lane (capacity 2, executed in two rounds). One platform requires foreground interaction and enters the visible webpage lane (capacity 1, exclusively occupying the foreground window for execution). Thus, a maximum of 2 + 2 + 1 = 5 tasks can be executed in parallel in the first round, meaning the peak concurrency of the acquisition phase is 5.

[0077] This enables bounded concurrent scheduling between lanes, preventing them from competing for resources and improving the efficiency of platform task execution.

[0078] In some embodiments, each lane may be assigned a lane identifier that can be associated upstream with an associated platform task identifier for traceability.

[0079] In some alternative embodiments, during the analysis phase following the acquisition phase, the concurrency of platform tasks attempting to perform analysis can also be limited by analyzing control semaphores.

[0080] In some alternative embodiments, during the saving phase following the analysis phase, it may be necessary to write the execution results of the platform task into the database. Alternatively, a single mutex lock can be used to serialize the write operations of each platform task execution attempt, so that only one write operation can be executed at a time, avoiding the conflict problem caused by multiple concurrent write operations.

[0081] Therefore, the acquisition phase uses lane capacity control for concurrency, the analysis phase uses analysis semaphores for concurrency control, and the storage phase uses a single mutex lock for concurrency control. Thus, each of the three phases uses an independent "lock" instead of a shared lock, implementing a multi-phase independent locking strategy. This multi-phase independent locking strategy avoids the situation where slow model analysis (which may involve large language model calls and is time-consuming) or database transactions (which may involve disk I / O and index updates) occupy the platform's interaction lane resources for extended periods, ensuring that the platform's request sending and response receiving can always proceed in a timely manner.

[0082] Therefore, the peak concurrency of some or all of the stages—front-end visible interaction, back-end web page collection, semantic analysis, and database writing—can be limited within a preset upper limit. This shortens the overall evaluation time while avoiding interaction contention, analysis resource congestion, and database write conflicts.

[0083] Step S103: Create an execution attempt for the platform task in a queued state waiting for the slot in the mapped lane.

[0084] It is understandable that a number of platform tasks corresponding to "n questions × m platforms" corresponds to the creation of a corresponding number of platform tasks. For example, platform tasks are created before queuing and can be associated with scheduling information such as a platform identifier, lane, initial index number, and platform execution budget. The platform identifier identifies the observation platform involved in the platform task, and the lane identifies the lane the platform task needs to enter. In the evaluation request, multiple observation platforms may have an initial order, and an initial index number is assigned to each observation platform according to the initial order. Furthermore, each platform task has a corresponding observation platform for which an observation question needs to be submitted, and the initial index number of that observation platform can be assigned to the platform task. It is understood that each platform task can uniquely associate a pair of observation questions and observation platforms, and can be uniquely associated with subsequent events between that observation platform and the observation question, as well as the generated data. For example, the submission of the observation question to the observation platform, the collection of the observation platform's answer to the observation question, the extraction of semantics from the answer, semantic analysis data, single-platform cognitive analysis data, etc.

[0085] It is worth noting that because the response speeds of different observation platforms may vary, the order in which the original responses are collected in subsequent acquisition phases may not be fixed. This inconsistency can lead to errors in information dependent on this order when fusing data across platforms. Specifically, even with substantially the same data, different data orders can result in different fusion outcomes, such as cached fingerprints or aggregations of the original responses or their extracted semantics from various observation platforms. Therefore, using the initial index number, the initial order can be restored before fusing data across platforms.

[0086] When queuing, a corresponding queued execution attempt can be created for the "queuing" work of the platform task. "Attempt" means attempt, which means a single attempt at a specific operation or request, and is not guaranteed to succeed.

[0087] Step S104: In response to the platform task obtaining a slot in the lane, update the queuing state of the execution attempt to the execution state and create an execution context to execute the platform task under the execution budget constraint of the slot and obtain a set of execution results.

[0088] In some embodiments, the determination of a lane slot by the platform task can be based on lane semaphore. The value of the lane semaphore corresponds to the number of available slots, and can dynamically change accordingly. After the platform task acquires a slot, it begins execution and releases the slot upon completion, making it available for subsequent tasks in the same lane.

[0089] The execution attempt refers to an actual execution attempt of a platform task. This execution attempt can be updated by the queued execution attempts and an execution context can be created for actual execution. In some embodiments, multiple execution attempts may be created under a single platform task due to execution failure. When this execution attempt fails, an exception message is generated. The exception message may include the failed execution state and the corresponding reason information. Retrying can then be performed by re-establishing the execution state. The platform task can record the log information of each of its subordinate execution attempts. The main purpose of distinguishing between the queued and execution states of execution attempts is to decouple "queuing" from "execution," facilitating separate persistence, auditing, and retry management.

[0090] In some embodiments, each platform task has a platform task identifier (task_item_id). Each created attempt may include fields such as a unique attempt identifier (attempt_id) under the corresponding platform task, a queuing time indicating when the queue to enter the lane (queue_at), a start time indicating when the slot was obtained (started_at), a completion time indicating when the slot was finished (finished_at), execution status (status), and reason information (reason_code). Optionally, the attempt may also include an execution record identifier (run_id) field. The execution record identifier is assigned corresponding to the execution result after completion and is written back to this field.

[0091] The platform task can be associated with each of its attempts by directly or indirectly (through a data record containing both) its task_item_id and attempt_id.

[0092] It's worth noting that the "laziness" refers to the fact that when a platform task is enqueued, it is already created but is not in an execution state (i.e., forming an execution attempt) but is in a "queued state" (i.e., an execution attempt in a queue). It only becomes an execution attempt and creates an execution context (such as allocating the resources required for execution) after obtaining an available slot. Platform tasks that have not obtained a slot and have not actually started execution will not generate execution attempts, thus avoiding the generation of false execution attempt records for tasks that are still in the queue or have never submitted a request.

[0093] Therefore, the queuing time (queue_at) recorded when a platform task begins queuing is not used to determine whether its execution time has exceeded the time limit; the actual execution time is calculated from the start time (started_at). In some embodiments, the interval between the queuing time (queue_at) and the start time can be used for congestion detection. Based on the execution time calculated from the start time (started_at), it can be used to determine whether the platform task execution has exceeded the platform's execution budget. If it has, a timeout is determined, and exception information for this execution attempt can be generated, including the failed execution status and reason information. This hierarchical timing mechanism ensures that queuing waiting time will not trigger a false judgment of execution timeout for the platform task.

[0094] In some embodiments, the platform capability profile also includes a timeout policy, wherein the execution budget of a lane can be determined based on the timeout policy of the observation platform to which it is mapped.

[0095] In some embodiments, the various execution budgets of the lane include at least one of platform execution budget, single-wave execution budget, and watchdog.

[0096] The platform execution budget is used to control whether a single platform task times out. In some embodiments, in response to obtaining a slot, the start time (started_at) is recorded and the execution duration of the platform task is calculated accordingly. Further, the execution duration can be checked for timeout based on the platform execution budget determined by the timeout policy of the observation platform corresponding to the platform task. If the execution duration reaches the platform execution budget (other execution duration limits can also be added), the platform task is individually determined to have timed out and failed, which will not affect the execution of platform tasks in the same lane or other lanes. That is, local task execution anomalies do not affect global task execution, improving reliability and efficiency.

[0097] The single-wave execution budget is determined by the maximum value among the platform execution budgets of the platform tasks assigned to the lane. This single-wave execution budget is used to estimate the maximum duration required for a single wave of concurrent tasks in each lane. Specifically, the estimated number of waves can be calculated by combining the number of platform tasks and capacity in the lane, thereby determining the upper limit of the queuing time for that lane.

[0098] The watchdog timer serves as the total execution time limit for a batch of platform tasks (such as those generated based on multiple observation issues in a single evaluation request), and can be set according to a single-wave execution budget. When the watchdog timer is triggered, queued tasks that have not yet started execution cease dispatching and are recorded as scheduling interruptions, while tasks that have already started execution are still controlled by their respective platform execution budgets.

[0099] like Figure 3 The diagram shows a flowchart illustrating the watchdog setting in one embodiment of this disclosure.

[0100] exist Figure 3 The process includes:

[0101] Step S301: Determine the number of execution waves corresponding to each lane based on the number of platform tasks allocated to each lane and the lane capacity of a batch of platform tasks.

[0102] The number of platform tasks can be obtained by counting the execution attempts of their corresponding queue states.

[0103] As an example, the number of execution waves can be obtained by dividing the number of platform tasks assigned to that lane by the lane capacity and rounding up. For example, if lane A has a capacity of 2 and is assigned 5 platform tasks, 5 / 2 = 2 waves, leaving 1 platform task. Therefore, the number of execution waves can be determined to be 2 + 1 = 3.

[0104] Step S302: Determine the upper limit of queuing time for each lane based on the number of execution waves and the single-wave execution budget corresponding to each lane.

[0105] In some embodiments, the single-wave execution budget can be determined based on the maximum value of the platform execution budget in the timeout strategy of each platform task assigned to the same lane, and is used to estimate the upper limit of the execution time of a single wave of platform tasks executed in parallel according to capacity. For example, in a batch of 5 platform tasks, lane A is assigned 3 platform tasks, corresponding to three observation platforms a, b, and c respectively, and the platform execution budgets of a, b, and c are x, y, and z respectively. Then, the maximum value among x, y, and z is taken as the single-wave execution budget of lane A.

[0106] In some embodiments, the upper limit of the queuing time for a lane can be obtained by multiplying the number of execution waves for each lane by the single-wave execution budget.

[0107] Step S303: Determine the global queuing time limit for each lane based on the maximum value among the queuing time limits for each lane.

[0108] In some embodiments, the maximum value of the queuing time limits for each lane can be added to a safety margin to form the global queuing time limit.

[0109] Since the execution budgets of different observation platforms within the same lane may differ, taking the maximum value allows for the most conservative estimation of the time limit for each wave. Multiplying the number of execution waves by the single-wave execution budget yields the queuing time limit for that lane. Then, taking the maximum value among the queuing time limits for all lanes gives the global queuing time limit.

[0110] Step S304: Set a watchdog timer based on the global queuing time limit. The watchdog timer is used to accumulate the execution time of the batch of platform tasks and stop the platform tasks that have not yet started execution from queuing when the execution time reaches the global queuing time limit.

[0111] Therefore, the watchdog can be used to prevent the entire batch from waiting indefinitely.

[0112] Let's illustrate this process with an example. Suppose lane A (type 1) has 5 platform tasks, a capacity of 2, and a single-wave execution budget of 120 seconds. If 3 waves are expected, then the maximum queuing time for lane A is 120 × 3 = 360 seconds. Suppose lane B (type 2) has 2 platform tasks, a capacity of 2, and a single-wave budget of 180 seconds. If 1 wave is expected, then the maximum queuing time for lane B is 180 seconds. Taking the larger of 360 seconds and 180 seconds (360 seconds), and adding a 120-second safety margin, we get a global queuing time limit of 480 seconds. Therefore, the watchdog timer is set to 480 seconds.

[0113] It is worth noting that the watchdog in this embodiment is not a simple timeout mechanism, but a technical means of determining the computable batch monitoring time limit based on the capacity of heterogeneous resources. In practical application scenarios, when the actual execution time of a lane exceeds the expectation, the system can dynamically adjust the monitoring time limit of subsequent batches (feedback loop), which is more accurate than a purely static timeout judgment mechanism.

[0114] In some embodiments, step S104 further includes step S1041, submitting an observation question to each of the observation platforms and collecting platform responses to form an evidence bundle, the evidence bundle including response evidence, source evidence, and execution evidence.

[0115] The answer evidence, source evidence, and execution evidence can all contain an execution record identifier (run_id), a platform identifier (platform_id), and a question identifier (question_id). These three types of evidence can be linked through these identifiers and are stored independently without overwriting each other. In some embodiments, the execution record identifier can be assigned after a successful attempt is saved. Therefore, a platform task may retry multiple attempts due to failed attempts, generating multiple attempt_ids, while a successful attempt will only have one execution record identifier (run_id). This execution record identifier can then be written back to the corresponding successfully executed attempt for marking. In some embodiments, when execution fails, a corresponding reason code can also be generated, appended to the execution status "failed," and recorded in the corresponding attempt's field.

[0116] In some embodiments, the answer evidence includes the original answer. Additionally, the answer evidence may also include at least one observable element extracted from the original answer, such as target-related segments, mentions / recommendations, etc. In some embodiments, the answer evidence may further include an answer status indicating the availability of the original answer, including values ​​such as available, partially available, and unavailable. Thus, the answer status can be set to available, partially available, or unavailable based on whether the collected original answer is complete, partially valid, or invalid.

[0117] In some embodiments, the source evidence may include characteristic information of the observation platform that serves as the source, preferably characteristic information presented when the original answer was generated. For example, at least one of the following observable features in the observation platform's answer interface: title, link, domain name, number of tags, publisher information, and completeness of data collection. In some embodiments, the source evidence may also include a source status indicating the completeness of the source data collection, including values ​​such as complete, partial, failed, and unsupported.

[0118] In some embodiments, the execution evidence may include normal / abnormal execution information of the original response during execution, as well as corresponding source tracing information (such as upstream identifiers). For example, the execution evidence may include at least one of the following: lane, slot index, platform task identifier, attempt identifier, anomaly (such as failure status, failure stage, and reason information), degradation (such as the semantic state of the following response evidence changing from available to partial, the source status of the source evidence changing from complete to partial, etc.), and recovery order (such as how to recover step-by-step after observing platform errors). In some embodiments, the execution evidence includes an execution status, which is success or failure and its reason information (success / failed + reason_code). Thus, based on whether the platform task execution is successful or failed, the execution status is set to success or failure respectively; wherein, the failed execution status is associated with reason information.

[0119] It is worth mentioning that during the collection phase of raw responses from various observation platforms, the raw responses and the sources cited were set as two related but independent collection objects, and their completion status was determined separately.

[0120] Once the original answer has been collected, source evidence can continue to be collected within an independent grace period. If the source collection times out (as determined by the lane execution budget in the previous embodiment) but the answer has already been successfully extracted, the original answer is retained, and stage evidence such as the original answer timing out before the source (answer_before_source_timeout) is recorded, and the source status is marked as failed separately. Through this dual-track determination, if, for example, the original answer body has been completed and persisted, the failure to load is only due to network anomalies, errors in referenced resource interfaces, or errors in referenced data parsing in the UI interface panel (a special type of UI element used to group, store, and isolate content), and not because the original answer is faulty, the completed original answer will not be incorrectly discarded. Thus, the partial failure stage of the source can be clearly located at the source collection stage, and the retained original answer body can still continue to enter the subsequent semantic extraction and verification processing. The loss of source integrity is reflected separately through the source status (i.e., "partial").

[0121] In some optional embodiments, the authenticity and completeness of the source can be verified by rules during the source acquisition phase. Specifically, only observable source evidence on the acquired page can be used, without fabricating URLs out of thin air or inferring a source title without a link as a real webpage. In some embodiments, the number of platform source markers and claims can be detected, and expansion, polling, and scrolling can be performed a limited number of times, stopping after several rounds of stabilization to avoid infinite waiting. For the acquired URLs, the system normalizes them to remove common tracking parameters, merges duplicates by source key, and filters out intra-site navigation links and page elements that do not constitute external evidence. In some optional embodiments, the source status can also record lower-level fine-grained statuses during actual acquisition. For example, lower-level fine-grained statuses can include complete acquisition (captured), partial acquisition (captured_partial), source marker detected but acquisition failed (marker_detected_capture_failed), no source provided by the platform (no_sources_provided), acquisition failed (failed), and source acquisition not supported (not_supported). Furthermore, the aforementioned lower-level fine-grained states can be normalized and mapped to values ​​such as complete, incomplete, failed, and unsupported source states. Source acquisition failure does not automatically invalidate the generated original response, but source completeness and source-type indicators only use data that conforms to the aforementioned evidence rules.

[0122] In some embodiments, an anomaly and degradation mechanism can be provided throughout the execution of platform tasks. For local anomalies occurring in individual observation platforms at certain stages, these platforms can be independently downgraded or rejected without affecting other observation platforms. For example, when an access method becomes unavailable, the access method can be switched or the platform task of that observation platform can be terminated based on the configuration, without affecting the collection of raw responses from other observation platforms. Another example is downgrading or discarding stage data from observation platforms experiencing anomalies at a certain stage. For instance, the response status set in the response evidence for raw responses collected during the collection stage can be downgraded from available to partially available, and the source status in the source evidence can be downgraded from complete to partially.

[0123] In some embodiments, the reliability of data collection can be calculated based on rules such as collection method, whether the response is too short, truncation, and mixing in historical content, and then the confidence level of the collected data can be determined based on the reliability of data collection.

[0124] Step S104 also includes a step of semantic analysis based on the original response from a single observation platform, specifically including:

[0125] Step S1042: Extract candidate semantics and candidate subjects from the original answer through a semantic extraction model, and determine the formal semantics that correctly belong to the target subject through a preset semantic verification.

[0126] In some embodiments, during the semantic extraction stage, candidate semantics, candidate subjects, and related subjects can be extracted from the original answer by a semantic extraction model that receives the target subject information. The semantic extraction model explicitly receives the target subject and allowed output structure and outputs the corresponding candidate semantics accordingly. The candidate semantics can be structured judgment candidates extracted by the semantic extraction model from the answer, and the output structure exemplarily includes at least a portion of the subject, judgment dimension, label, meaning, and original text evidence fragments. In some embodiments, the semantic extraction model can be implemented by a remote or local natural language processing (NLP) model (such as a large language model LLM), rule engine, or statistical model, but the same evidence verification contract must be agreed upon to implement the verification processing based on the evidence information.

[0127] The semantic extraction model can generate multiple candidate semantics from a single original answer. For example, for an answer like "AAA is a commonly used BBB trading platform with low transaction fees" (all positive evaluations), the semantic extraction model can extract multiple candidate semantics such as "Subject (AAA) - Judgment Dimension (Platform Type) - Label (BBB Trading Platform) - Meaning (Objective Fact: AAA's business type is a BBB trading platform, no sentiment bias) - Original Evidence Fragment (AAA is a commonly used BBB trading platform)", "AAA - Platform Popularity - Positive Evaluation: The platform has high usage popularity - AAA is a commonly used BBB trading platform", and "AAA - Transaction Fees - Low - Positive Evaluation: The platform has low transaction fees - Low transaction fees", etc.

[0128] In some embodiments, to prevent the illusion problem of semantic extraction models, during the candidate semantic extraction process, semantic extraction can be performed separately by deterministic structural rules (such as manually written judgment conditions, matching patterns, parsing logic, such as text regular expression matching, DOM, etc.) and semantic extraction models, and the semantics extracted by them can be fused together using deterministic merging logic to obtain candidate semantics.

[0129] As an example, basic semantics are extracted from the original answer and its observable content using deterministic structural rules, which can be categorized as answer evidence. The deterministic structural rules are explicit extraction rules based on content structure, such as at least one of pattern matching rules, structural parsing rules, entity recognition rules, and relation extraction rules. In some embodiments, the deterministic structural rules can be implemented based on at least one of a dictionary, pattern matching, statistical model, or rule engine.

[0130] In some embodiments, the basic semantics may include relevant fragments of the target subject and mention / recommendation information (such as recommendation relationships, ranking relationships, and recommendation bias). Optionally, the basic semantics may also include one or more dimensions of information such as competitors, attitude bias, accuracy, and completeness. In some embodiments, if a semantic extraction model is available, candidate semantics output by the semantic extraction model based on the original answer are obtained. If a semantic extraction model is unavailable, the basic semantics are directly used as candidate semantics to ensure that the evaluation can continue.

[0131] In some embodiments, the preset semantic verification may include subject attribution verification to confirm the subject to which the candidate semantic belongs. In some embodiments, the subject attribution verification may confirm the attribution of the candidate semantic to the target subject, competitor, platform, or general industry description based on one or more of the target subject's name, aliases, original evidence fragments in the original answer, and contextual clauses. This prevents incorrect attribution from causing inaccurate subsequent analysis. For example, it prevents negative reviews from other subjects from being mistakenly attributed to the target subject, or incorrectly binding general industry descriptions to a specific subject.

[0132] In some embodiments, the preset semantic verification may include original text verification, such as verifying whether the candidate semantic or its subject is supported by original text evidence in the original answer, and marking the candidate semantic as passed or rejected based on the verification result. In some embodiments, when there is a conflict between the original answer and the candidate semantic, the processing logic of evidence priority is followed, and the candidate semantic must not overwrite the strong textual evidence of the answer with a null value or a weaker conclusion. Optionally, for situations with low confidence, conflicting evidence, or serious information risks, corresponding review reasons can be generated.

[0133] In some embodiments, the preset semantic verification may also include integrity verification and enumeration / type verification. Integrity verification can determine whether necessary fields are missing, and enumeration / type verification can determine whether the type / enumeration is invalid.

[0134] In some embodiments, the semantic extraction model also returns an analytical confidence score for the extracted candidate semantics. The analytical confidence score is a numerical value returned synchronously by the semantic extraction model when outputting candidate semantics, representing the model's degree of confidence in the extraction result. This can be understood from two levels: model output and process control. At the model output level, when extracting semantics from the original answer, the semantic extraction model outputs the analytical confidence score, such as a scalar value between 0 and 1, in addition to outputting structured results like the text content and candidate subject of the candidate semantics. In optional embodiments, this analytical confidence score can vary depending on the different implementations of the semantic extraction model. In some optional examples, if the model itself is a sequence labeling or span extraction model, the softmax probability of the corresponding label or span can be directly taken as the analytical confidence score. In other optional examples, if the model is a generative large model, it can be obtained through the consistency ratio of multiple samples, the normalized value of the output logprobs, or by having the model self-evaluate an analytical confidence score field in the output structure. Regardless of the method, it can be normalized to a scalar between 0 and 1. The higher the value, the more confident the model is in the semantic extraction results.

[0135] After the semantic extraction model returns the results, the analysis confidence score is compared with the analysis confidence score threshold. When the analysis confidence score is greater than or equal to the analysis confidence score threshold, the semantic extraction result is considered reliable enough, and the candidate semantics directly enters the subsequent semantic verification process. When the analysis confidence score is lower than the threshold, it indicates that the semantic extraction model is not confident enough in the result, and there may be semantic ambiguity, subject identification ambiguity, or unclear original text expression. In this case, the candidate semantics does not directly enter the verification process, but is transferred to the review process.

[0136] In optional scenarios, the review process can be implemented in several ways. One option is to send candidate semantics with low analytical confidence and their corresponding original text fragments to a review queue for secondary confirmation by another model or manual review. Only those that pass the review proceed to semantic verification; those that fail are discarded or marked as insufficient semantics. Another option is to adjust the parameters of the semantic extraction model and re-extract candidate semantics, continuing with those that match the results from both reviews. Alternatively, results with low analytical confidence can be marked and allowed to proceed to the semantic verification stage for further filtering in subsequent verification steps, essentially replacing single-channel threshold interception with multi-channel verification. The specific review method used can be configured based on the system's trade-off between accuracy and throughput. The analytical confidence threshold is also a configurable parameter and is not limited to the examples above.

[0137] In addition, semantic dimensions that the semantic extraction model cannot recognize or whose analysis confidence is below the threshold are not forcibly filled; instead, candidate semantic extractions are left empty or marked as unrecognized.

[0138] Step S1043: Form answer semantics based on the formal semantics of the same original answer.

[0139] Understandably, during the semantic extraction stage, the semantic extraction model outputs candidate semantics only within the defined input, output, and rule boundaries, and its correctness is ensured by verification of the answer evidence during the verification stage. Therefore, the semantic extraction model does not acquire control over the acquisition of the aforementioned answer evidence, the selection of samples in subsequent cognitive analysis, or the generation of the final evaluation conclusion. This avoids the illusion of "creating something out of nothing" and ensures the reliability of the final evaluation conclusion.

[0140] Only after verifying the candidate semantics generated from an original response on an observation platform will the verified formal semantics be integrated to form the answer semantics corresponding to the original response.

[0141] Based on the number of formal semantics in the candidate semantics, a semantic state representing the confidence level of the answer's semantics can be determined. Specifically, the semantic state is determined as available, partially available, or not available based on whether all candidate semantics belonging to the same original answer pass, partially pass, or are all rejected. For example, if an answer extracts 3 candidate semantics, if the subject attribution, original text evidence fragment, type, and enumeration of all 3 candidate semantics pass verification, they are marked as available; if only 2 pass and 1 is rejected, they are marked as partially available; if all 3 are rejected, they are marked as not available. The reason for the rejected candidate semantics is retained and they are not included in the formal semantic list.

[0142] As an example, the response semantics may include: semantic state (values ​​are available, partial, or not_available), a list of verified formal semantics, original evidence fragments (evidence_text), and verification statistics for candidate semantics (e.g., any combination of the total number of candidate semantic entries, the number of available entries, and the number of not_available entries) and error reasons (validation_errors, reason_code for not_available candidate semantics). The list of formal semantics includes the subject to which each formal semantic belongs, the judgment dimension (claim_type), etc.

[0143] The execution results of the platform task, which forms the answer semantics and evidence set, are written to the respective result slots. When a scoring phase exists, the obtained single-platform score is also stored in the corresponding result slot, along with the execution results of the platform task, for use in subsequent cross-platform evaluations.

[0144] As described above, after obtaining the lane slot, the platform task transitions from a queuing state to an execution state and creates an execution context for execution. This process sequentially involves several stages: requesting (establishing a request with the corresponding observation platform or entering the target interaction interface), collecting (collecting responses and sources, with each response and source documented separately), analyzing (executive rule analysis and optional model analysis, constrained by analysis semaphores), scoring (calculating single-sample and aggregate indicators, an optional stage), and saving (persistently storing responses, sources, analysis, scores, and stage evidence, constrained by database write locks). This process results in the platform task's success or failure status and generates an execution result. Each stage retains stage evidence; outer timeouts, stage evidence, and anomaly degradation collectively ensure that local failures are locable and that successful execution is not negated by local failures.

[0145] In some optional embodiments, the system can provide anti-duplicate submission protection for platform tasks based on task keys. For example, the task key can be obtained based on slice feature information during the actual execution of the platform task, such as a combination of information including the observation platform, actual access method, and start time (start_at). This ensures that a platform task has been executed, preventing tasks with the same task key from being executed repeatedly, avoiding duplicate submissions or concurrent execution of the same observation problem on the same observation platform, and guaranteeing a stable correspondence between input (observation problem) and output (execution result of the platform task) within a batch.

[0146] In some embodiments, for anomalies occurring during platform task execution, such as execution timeouts, source collection failures, semantic validation rejections, or database write errors, corresponding failure records can be generated and normalized. As an example, regardless of the stage or cause of the anomaly, it can be uniformly converted into a record of cause information in the same data structure, which includes at least: the stage at which the anomaly occurred (e.g., request / collection / analysis / saving), cause information (e.g., timeout / source failure / validation rejection / write error), and a timestamp. This normalization eliminates the need for upper-level aggregation logic to write special processing branches for each failure type. The records of cause information can be accumulated into a cause information set, which can be used for auditing and analyzing the causes of anomalies.

[0147] Furthermore, based on the execution results obtained from the task execution on each platform, cross-platform aggregation can be performed to observe the overall views of multiple observation platforms on the target subject, including consensus / disagreements based on semantic aggregation of answers, and overall cognitive evaluation of observable content based on answer evidence.

[0148] In some embodiments, to ensure that the evaluation results obtained from each platform are not affected by the asynchronous completion order, the results may be restored according to the initial order of the observation platforms in the evaluation request based on the initial index number before performing cross-platform aggregation of the semantics of each answer.

[0149] For example, before each observation platform enters the lane queue, a fixed initial index number (original_index) is assigned according to the initial order of the observation platforms specified in the user's submitted evaluation request. For instance, if observation platforms A, B, and C are submitted, their original_index values ​​are assigned as 0, 1, and 2, respectively.

[0150] Furthermore, based on the initial index number, the execution results of each platform task are stored in the corresponding result slot.

[0151] For example, observation platforms A, B, and C are assigned original_index as 0, 1, and 2 respectively. The structured evaluation conclusions of the three observation platforms A, B, and C are obtained in the order of C, A, and B. Then, result[2], result[0], and result[1] are written respectively. Finally, the values ​​are taken according to result[0], result[1], and result[2], which restores the initial order of A, B, and C. In some embodiments, the aggregation key is a combination identifier used to identify the "platform-result" correspondence when batching cross-platform aggregation results (e.g., including the target subject around which the aggregation is located, the judgment dimension (claim_type), the standardized label (label), etc.). If the order of the results on the right is different from the order of the platforms on the left, the correspondence between the two sides of the entire aggregation key will be disordered, causing the aggregation results to be inconsistent. Through the above mechanism of restoring the initial order, the result index order in the aggregation key can be made consistent with the initial order of the platform, so that the order of the platform and the result on both sides of the "platform-result" combination identifier is consistent and corresponds one-to-one, thereby avoiding the inconsistency of the aggregation input caused by the asynchronous completion order change.

[0152] In some embodiments, when performing cross-platform data aggregation (such as aggregation semantics or answers), for multiple Attempts on the same question and the same observation platform task, only the latest valid Attempt record is selected to form the execution result, while older records are retained for auditing. This avoids the repeated accumulation of historical retests and the mixing of expired records, ensuring that the same input data can form a stable aggregation caliber.

[0153] Back Figure 1 The method also includes step S105: filtering effective answer semantics based on a set of execution results, aggregating the effective answer semantics associated with each observation question, and forming a cross-platform semantic conclusion for each observation question.

[0154] like Figure 4 The diagram shown illustrates the specific process flow of step S105 in one embodiment of this disclosure.

[0155] exist Figure 4 The process includes:

[0156] Step S1051: Filter the semantic states of responses that are available or partially available from the various response semantics associated with the observed question.

[0157] Understandably, when aggregating the semantic responses to the same observation question across different observation platforms, only valid, available or partially available semantic responses are used to ensure the accuracy and reliability of the aggregation results. Candidate semantic responses that fail validation are not included in the official semantic list; only the number of rejections and the reasons for the errors are retained.

[0158] Step S1052: Aggregate the selected effective answer semantics to form the cross-platform semantic conclusion.

[0159] In some embodiments, the semantics of valid answers to the same observation question on multiple platforms can be aggregated according to the association keys of "target subject", "judgment dimension" and "standardized label". For example, if two observation platforms A and B describe the platform type of the exchange as "card trading platform" and "platform for buying and selling cards" respectively, then the semantics of valid answers to the same question by A and B can be aggregated by the association key of "exchange platform" as the target subject, "business type" as the judgment dimension and "card trading platform" as the standardized label.

[0160] In some embodiments, the cross-platform semantic conclusion may include at least one aggregation item among consensus items, divergence items, and common omission items. Thus, each aggregation item can be used to explain which viewpoints are consistent across multiple observation platforms, which judgments are divergent, and which important content (such as content belonging to the target subject) is commonly ignored. The consensus item can be aggregated based on semantic consensus between the semantics of responses from at least two of the observation platforms. In some embodiments, the semantic consensus can be achieved based on semantic similarity determination. Each consensus item may also record the judgment dimension, variant labels (synonymous with the standardized label that reached the consensus), participating observation platforms, and original text evidence, which can identify which platforms share the same viewpoint. For example, if the response semantics of observation platform A and observation platform B both formally express "MMM can be used as a transaction price reference," a consensus item can be generated on the standardized label "price reference," recording the consensus observation platform ID set {A, B}, the corresponding run_id, and two original text evidence fragments or their indexes from platforms A and B. In some embodiments, the original text evidence index may be implemented as pointing to the position, fragment ID, or hash of the corresponding original text evidence fragment in the original response.

[0161] The divergence term can be obtained by aggregating semantic divergences between the semantics of responses from at least two of the observation platforms. In some embodiments, the divergence can be determined based on contradiction detection. Specifically, the semantic divergence can be that the valid response semantics have contradictory standardized labels for the same target subject. For example, if one observation platform describes the target subject as "a platform with low fees," and another platform describes the target subject as "a platform with high prices," then the two standardized labels point to high prices and low prices respectively, generating a contradictory divergence, and their formal semantic information can be aggregated to form a divergence term. The divergence term can also record a set of multiple platform IDs that generate the divergence, a set of corresponding run_ids, and a set of original text evidence fragments (or their indexes). The index can be a pointer to the position, fragment ID, or hash of the corresponding original text evidence fragment in the original response.

[0162] The common omission item is generated based on information extracted from situations where the semantics of the responses from at least two of the observation platforms both omit the same preset content (such as important content) of the target subject.

[0163] In some embodiments, the cross-platform semantic conclusion may include a set of IDs of the various observation platforms related to the aggregated semantics, in order to understand the scope of platforms participating in the aggregation.

[0164] Therefore, by setting a threshold of at least two observation platforms for aggregation, consensus, disagreement, or common omissions can be identified only when consistent and comparable evidence is provided by more than two observation platforms. When there is only one platform, single-platform observations are retained, but aggregations are not erroneously generated; however, the platform identifier and evidence of the single platform are still preserved.

[0165] Step S106: Based on the evidence set in a set of execution results, select valid evaluation samples, collect the evidence set of observation problems and scene labels in the valid evaluation samples, and analyze them to obtain evaluation signals that characterize the strength of the target subject's performance in the dimensions of overall cognitive coverage, scene performance and platform consistency, so as to form a structured evaluation conclusion.

[0166] In cases where the cross-platform semantic conclusions and structured evaluation conclusions are related in terms of the strength of the evaluation signals, they can be used as explanatory clues for the strength of the evaluation signals. If the cross-platform semantic conclusions are related to the structured evaluation conclusions, they can serve as supplementary evidence for those conclusions, such as being output together according to problem identifiers, scene labels, or sample ranges to provide explanatory clues, but without directly changing the signal strength value. For example, if the signal strength value of the scene breadth signal indicates "weak high-price transaction scene," and the corresponding cross-platform semantic conclusion expresses that multiple observation platforms have collectively omitted "high-price transaction guarantee," then the omission of "high-price transaction guarantee" can serve as an explanatory clue for the "weak high-price transaction scene" of the target subject, and can be used as a reference when determining the cause. Thus, by outputting a combined report of structured evaluation conclusions and corresponding cross-platform aggregated semantics for the same observation problem or observation problem range, users can more efficiently and easily understand the strength of the evaluation signals in the structured evaluation conclusions. In some more stringent optional examples, the structured semantic conclusions are not used as a direct cause of the weak scene, nor do they directly change the signal strength value.

[0167] It is understood that the evidence set and response semantics contained in the execution results all possess state information characterizing their completeness and reliability, including response evidence, source evidence, operational evidence, and response semantics. Therefore, this state information can be used to filter valid execution results as effective evaluation samples for aggregation.

[0168] In some embodiments, such as Figure 5 The diagram shown illustrates the specific process flow of step S106 in one embodiment of this disclosure.

[0169] exist Figure 5 In step S106, the following may be further included:

[0170] Step S1061: Determine the valid evaluation samples based on the availability of the response evidence indications.

[0171] In some embodiments, the determination of valid evaluation samples can be based on the confidence level of the response status of the response evidence. As an example, the determination criterion for valid evaluation samples can be configured such that the response status indicates the availability of the original response. For instance, the determination criterion can be set to the response status being either available or partially available.

[0172] Of course, the above screening criteria are merely examples and not intended to be limiting. It is understood that the screening criteria for valid evaluation samples are not fixed and can be adjusted according to the stringency of the evaluation scenario. For example, in evaluation scenarios with high reliability requirements, the criteria for valid evaluation samples can be limited to an "Available" response status. Alternatively, in exploratory evaluation scenarios with high sample coverage requirements, the criteria can be relaxed to an "partially available" response status. Through this status screening mechanism, valid evaluation samples undergo quality stratification before entering cross-platform aggregation, ensuring that the generation of subsequent performance strength evaluation signals is based on reliable samples.

[0173] Step S1062: Based on the visibility results extracted from the answer evidence in the valid evaluation samples according to the rules, obtain the overall coverage signal representing the maximum loss value of the target subject when switching between different recommendation rankings.

[0174] In some embodiments, the overall coverage signal can also be referred to as the "overall funnel signal." Based on progressively stricter hierarchies such as mentions in effective question-platform units (represented by the coverage rate of the target subject mentioned by each observation platform), entry into the recommendation set (e.g., the recommendation position within each subject when mentioned, such as Top N), and first-choice (i.e., ranked first in recommendations), it determines whether the target subject as a whole can be seen and whether it can enter a stronger recommendation position. The overall funnel determines the main loss from being mentioned to Top 3 and then to the first recommendation. For example, with a coverage rate of 75%, a Top 3 rate of 50%, and a first-choice rate of 20%, it can be determined that the main loss occurs in the recommendation position transformation stage, with a significant attenuation from being mentioned to entering Top 3 and then to the first recommendation. The signal strength can be calculated based on the maximum loss among the coverage rate, Top 3 rate, and first-choice rate.

[0175] Step S1063: Based on the results of the observation question labels corresponding to the visibility results extracted by rules from the answer evidence in the valid evaluation samples, a scene breadth signal representing the overall recommendation intention strength / weakness of multiple observation platforms for the target subject in different scenarios is obtained.

[0176] In some embodiments, the scenario breadth signal is based on the effective coverage, advantages, disadvantages, and sample sufficiency of each business scenario to determine whether advantages are concentrated in only a few scenarios and where the disadvantages lie. For example, scenarios include price inquiry scenarios, card purchase scenarios, and high-price transaction scenarios. Assuming the price inquiry scenario scores 70 points, the card purchase scenario scores 65 points, and the high-price transaction scenario scores 30 points, then the high-price transaction scenario can be identified as a significant disadvantage scenario. In some embodiments, the signal strength of the scenario breadth signal is calculated based on the difference between each scenario score and the scenario median. In some embodiments, scenarios are determined based on scenario tags pre-associated with the observation questions, and the platform task can directly obtain these tags. These tags are not regenerated by each observation platform based on the answer content, avoiding the introduction of illusions. Alternatively, in some embodiments, the scenario tags can also be pre-determined when the question enters the question database, and can be manually set or extracted from a fixed scenario list and then verified by rules. The evaluation request can directly carry the tag, without being regenerated by each observation platform based on the answer content.

[0177] Step S1064: Based on the visibility results extracted from the rules of the answer evidence in the valid evaluation samples, a platform consistency signal representing the discreteness of the target subject's performance on each observation platform in the same question dimension is obtained.

[0178] In some embodiments, the temperature of the same question can be determined based on the consistency of the question identifier (question_id). In a further embodiment, when the question_id is unavailable, identification can be performed based on the textual or contextual features of the observed question, such as identifying the observed question identity based on a question fingerprint. For example, the question fingerprint can be obtained based on the hash value of the observed question text, the standardized text it contains, or semantic similarity clustering. In some embodiments, the question fingerprint can normalize observed questions with different expressions but essentially the same meaning into "the same question," facilitating alignment according to the question dimension when aggregating data in cross-platform execution results.

[0179] In some embodiments, platform consistency can be determined based on the valid results and average dispersion of different observation platforms to assess the stability of the conclusions, i.e., cross-platform stability and whether they are influenced by a single platform. For example, if the average scores of four platforms on multiple issues are 30, 60, 65, and 70 respectively, with large dispersion, it indicates that the conclusions are significantly affected by platform differences, and it is not appropriate to draw definitive conclusions directly. Therefore, the platform consistency can be calculated based on the average dispersion of each platform on multiple issues.

[0180] It should be noted that the three signals in steps S1062, S1063, and S1064 do not need to be in any particular order. The overall coverage signal, scene breadth signal, and platform consistency signal (hereinafter referred to as the "three signals") obtained respectively serve as the main conclusions in the structured evaluation conclusions.

[0181] In some embodiments, after filtering, the observation problems and observation platforms can be standardized (i.e., key normalization), sorted from newest to oldest according to the creation time of the execution attempt. For each combination of standardized observation problems and observation platforms, only the latest record is retained to represent the execution result for three-signal calculation. Replaced or excluded old records are retained in the audit information, recording the replacement relationship, replacement time, and execution record identifier.

[0182] In some embodiments, based on the latest valid records, the system can first stratify the valid evaluation samples according to the type of observed problem, such as dividing them into core problems, adjacent problems, and directional problems. Furthermore, different analysis views can be designed to correspond to the aforementioned problem types. Each problem type can limit the sample range of the corresponding valid evaluation samples. Each view only performs statistics and scoring on the valid evaluation samples of the corresponding type of observed problem within its own sample range, avoiding duplicate scoring across ranges and ensuring that the evaluation conclusions at different levels have clear sample boundaries and verifiable standards.

[0183] Step S1065: Determine the main signal and auxiliary signal based on the signal strength of the overall coverage signal, scene breadth signal and platform consistency signal.

[0184] The primary signal is the signal dimension upon which the main evaluation conclusions are based, reflecting the most prominent cognitive characteristics or problem manifestations of the target subject in the current evaluation batch. Secondary signals are used to supplement dimensions not covered by the primary signal, or to verify or correct the judgment direction of the primary signal from another perspective.

[0185] like Figure 6 The diagram shown illustrates the specific process flow of step S1065 in one embodiment of this disclosure.

[0186] exist Figure 6 The process includes:

[0187] Step S10651: Filter qualified signals based on threshold conditions including the minimum number of valid platforms, the minimum number of valid scenarios, and the minimum number of valid evaluation samples.

[0188] First, eligibility is determined by checking the minimum number of valid evaluation samples, minimum number of platforms, and minimum number of scenarios required for each signal. The state of the evidence can also be considered, such as the individual states of the three types of evidence and one or more combinations of the semantic states of cross-platform semantics. Signals that do not qualify do not participate in the competition for the main conclusion.

[0189] In some embodiments, the minimum number of valid platforms can be set according to different levels of strictness. For example, it can be one or more combinations of the following: the number of observation platforms whose execution status is successful, the number of platforms whose response status is available / partially available, the number of platforms whose source status is available / partially available, the number of platforms whose semantic status of the response semantics is available / partial, or the number of observation platforms required by preset rules. It can be configured according to the actual scenario.

[0190] In some embodiments, the minimum number of valid scenarios can be the number of valid scenarios that meet the requirements of a minimum preset number of issues and a minimum preset number of platforms. The minimum preset number of issues and the minimum preset number of platforms can be set according to preset rules and can be configured according to changes in actual scenarios.

[0191] In some embodiments, the minimum number of valid evaluation samples can also be set according to preset rule requirements.

[0192] Step S10652: Based on the condition that the intensity value reaches a significant threshold, filter candidate signals from the qualified signals.

[0193] The signal strength value of the overall coverage signal is determined based on the maximum loss value during the conversion between recommended rankings. The signal strength of the scene breadth signal is calculated based on the difference between the score of each scene and the median of the scene. The signal strength value of the platform consistency signal is calculated based on the dispersion score. For example, a signal strength value of 10 is considered a significant candidate signal.

[0194] Step S10653: Based on signal strength comparison, determine the main signal among the candidate signals.

[0195] In some embodiments, qualified signals are included in the strength calculation. Comparable index values ​​are formed based on the original signal strength values ​​of each qualified signal. For example, the original signal strength values ​​can be used, or they can be converted to level-normalized signal strength values, while retaining the original values ​​before conversion and the reason for threshold hit. Furthermore, qualified signals with higher comparable index values ​​can be selected as primary signals, and the remaining qualified signals can be secondary signals.

[0196] In some embodiments, if the difference between the multiple candidate signals with the highest index values ​​is close to or below a threshold, a primary signal is determined among the multiple candidate signals based on the condition that at least one of the number of effective platforms, the number of effective scenarios, and the number of effective evaluation samples covered by the candidate signals tends to be higher, and the remaining candidate signals are secondary signals. For example, the strongest one is designated as the "primary signal," and other obvious signals are designated as "auxiliary signals."

[0197] Specifically, when comparable indicator values ​​are similar (e.g., the difference is less than a preset proximity threshold), an evidence sufficiency comparison is performed, prioritizing signals that cover more platforms, scenarios, and effective units. In some examples, the signal with the highest indicator value can be prioritized as the primary signal. In other examples, if multiple signals reach the threshold and have equal strength, a fixed priority rule can be used to determine the primary signal, with another signal serving as an auxiliary signal, without random selection. When no signal is qualified, controlled degradation is performed, outputting "Insufficient evidence, no trend / consensus conclusion formed at this time," rather than forcibly drawing conclusions with a low sample size. In some embodiments, multi-level conclusions and preset strategies for judging controlled degradation can be designed based on the number of qualified signals, such as from "formal conclusion" to "reference conclusion" and then to "insufficient evidence." This is particularly important in heterogeneous platform scenarios, as the observation platform may be unavailable at any time.

[0198] In some embodiments, if the primary and secondary signals cannot be distinguished after comparing the sufficiency of evidence, the primary signal is determined according to the default priority. For example, overall coverage takes precedence over scenario breadth, scenario breadth takes precedence over platform consistency, and random selection is not used to ensure the certainty and reproducibility of the results. After the primary signal is determined, the remaining signals that reach the significance threshold are used as secondary signals; qualified signals that do not reach the significance threshold are not included in the secondary signals, but can be recorded in the reason information. When all signals do not meet the threshold conditions or the evidence is insufficient, controlled degradation is performed: the primary signal is set to empty, the validity of the conclusion is marked as insufficient evidence, and the reason information, such as insufficient sample or the signal not reaching the threshold, is recorded in the reason information as reason_code.

[0199] Understandably, the distinction between primary and secondary signals is not fixed to a single signal dimension, but rather dynamically determined based on the eligibility criteria and intensity calculations of each signal within the current evaluation batch. When the overall coverage signal is significant, it serves as the primary signal, reflecting the overall cognitive level of the target subject. When the scene breadth signal is more prominent, it becomes the primary signal, revealing the performance differentiation of the target subject in different scenarios. Regardless of which signal becomes the primary signal, the remaining qualified signals serve as secondary signals in the conclusion formation, thus ensuring that the evaluation conclusions are both focused and comprehensive.

[0200] The primary signal corresponds to the main conclusion in the final structured evaluation results, while the secondary signals serve as supplementary reference information. For example, in the presentation of the structured evaluation results, the evaluation conclusion corresponding to the primary signal is displayed prominently, serving as the core conclusion of the evaluation report for users to focus on first. The secondary conclusions corresponding to the secondary signals are listed as supplementary reference information, helping users to comprehensively understand the target entity's performance in other dimensions and avoiding oversimplification based on a single dimension.

[0201] It should be noted that the aforementioned signal strength, comparable index values, and thresholds are preset and configurable judgment parameters, which can be calibrated based on historical data, expert annotations, and actual verification results, and are not limited to fixed values. As can be seen from the above scheme, this disclosure provides evidence records at each stage, enabling traceability of inspections during audits or reviews.

[0202] like Figure 7 The diagram shown illustrates the data flow in one embodiment of this disclosure.

[0203] The evaluation request includes a target subject (with a subject type), scene labels, observation questions (assigned question_id), and observation platforms (assigned platform), each assigned an original index number (original_index). Based on the evaluation request, a total task (assigned task_id) is generated, and a platform task (assigned task_item_id) is generated for each observation question on each observation platform. A result slot is set and bound for each platform task. Execution attempts (assigned attempt_id) corresponding to the platform task are queued and obtained in lanes (in the example, three types of lanes, each assigned lanex, y, z). After obtaining a lane slot (or a slot number under lane), the attempt is lazily updated and an execution context is created to collect data from the corresponding observation platform and store it in the result slots (bound to original_index) according to the initial order of the platform. In the diagram, these are represented as m slots respectively.

[0204] During the analysis phase, candidate semantics are extracted from each raw answer and validated. Those that pass validation become official semantics and are integrated to form the answer semantics. Each candidate / official semantic can be associated with a `run_id`, `question_id`, and `platform` to indicate its affiliation, and also includes a `evidence_text` fragment from the original text to guide the reader to the original text. Candidate semantics that fail validation may also include validation errors, containing the reasons for failure. Therefore, the answer semantics constructed from the official semantics and error reasons are associated with the set of `run_id + question_id + platform + evidence_text` (and / or its original text evidence index) for each official and candidate semantic.

[0205] In some embodiments, the output object of the structured evaluation conclusion may include: a primary signal, supporting signals, and run IDs. Optionally, the structured evaluation conclusion may also include qualification information such as whether the signal is qualified or unqualified for auditing purposes.

[0206] In some optional embodiments, the structured evaluation conclusion may further include a conclusion validity (confidence_state). The conclusion validity can be assessed by a conclusion confidence evaluation logic based on a comprehensive judgment of signal strength, sufficiency of evidence, and sample coverage, reflecting the reliability of the current evaluation conclusion. For example, the conclusion validity may include multiple levels of states, which can be controlled and downgraded according to the level of reliability. For instance, there are three levels: formal, reference, and insufficient. Formal indicates that all formal sample thresholds are met and the main signal is selected normally. Reference indicates that a conclusion can be formed, but the formal matrix is ​​not yet fully satisfied; the conclusion is for reference but not considered a formal conclusion. Insufficient indicates that the basic judgment threshold has not been met. When the evidence is insufficient or the signal strength is unqualified, the conclusion validity is downgraded, and the output is "Insufficient evidence, no trend / consensus conclusion is formed at this time." The aforementioned signal strength and threshold are preset and configurable judgment parameters, which can be calibrated based on historical data backtesting, expert annotation verification, and actual implementation effects, and are not limited to fixed values. Therefore, the goal of grading the credibility of structured evaluation conclusions can be achieved, rather than a simple binary judgment of "present / absent", but a multi-dimensional comprehensive judgment based on signal strength, sufficiency of evidence and sample coverage.

[0207] The `run_ids` associated with the cross-platform semantic conclusions can be used for forward tracing. For example, when tracing the structured evaluation conclusions, the tracing information chain can be used: Structured Evaluation Conclusion → `run_ids` → (`ai_runs` → `ai_analysis` / `ai_run_sources`) → `attempt_id` → `task_item_id` → `task_id`, tracing upwards level by level, and the original text can be found based on `evidence_text`. Here, `ai_runs` represents the set of execution records indexed by `run_ids`, which can store questions (e.g., question ID or original question text), platforms, platform capability profiles, original answers, and execution decisions (e.g., lane mapping, timeout, etc.). `ai_analysis` represents the set of analysis records involved in `ai_runs`, storing mention / recommendation locations, single-platform scores, and semantic information (e.g., cross-platform semantic conclusions, single-platform answer semantics, and its formal and candidate semantics, etc.). ai_run_sources represents the set of source evidence involved in ai_runs, which stores source evidence (such as at least one of the following: title, link, domain name, number of tags, publisher information, and collection completeness).

[0208] To trace the source of cross-platform semantic conclusions, one can trace down through the following steps: cross-platform semantic conclusion → evidence reference information (evidence_refs, which is only a data structure representation and does not necessarily need to exist as actual fields in the database) → formal semantics in AnswerSemantics → attempt_id → task_item_id → task_id, tracing back to the execution source of each participating observation platform. The aggregated items in the cross-platform semantic conclusions carry the evidence reference information. In some embodiments, the evidence reference information includes platforms + run_ids + evidence_text (or its index). The platform set platforms and the execution record identifier set run_ids point to the observation platforms that generated the semantics and their related execution records. The original text evidence fragment evidence_text (or its index) can be precisely located in the original responses of each platform.

[0209] It is worth mentioning that the above-mentioned traceability is not one-way, but can also achieve two-way traceability applications. For example, it is possible not only to trace back from the evaluation conclusion to find the relevant original records (such as evidence sets, platform tasks, observation platforms, observation issues, etc.), but also to determine the conclusions affected by the original records. This two-way traceability has excellent technical effects in auditing, reproduction, and error correction scenarios.

[0210] like Figure 8 The diagram illustrates a module schematic of a subject evaluation device based on a heterogeneous artificial intelligence platform according to an embodiment of this disclosure. It should be noted that the principle and technical implementation of the subject evaluation device based on the heterogeneous artificial intelligence platform can be referred to the method embodiments in previous embodiments, therefore, they will not be repeated in this embodiment.

[0211] exist Figure 8 The main evaluation device 800 includes:

[0212] The input module 801 is used to obtain at least one observation problem containing target subject information and associated scene labels, as well as evaluation requests from multiple observation platforms, and to determine a set of platform tasks for each observation problem to be evaluated by an observation platform, wherein the multiple observation platforms are different generative artificial intelligence platforms.

[0213] The mapping module 802 is used to obtain the platform capability profile of each observation platform and map it to the lanes that match the resource constraints according to the platform capability profile. The lanes are configured with capacity, which refers to the number of slots that can be occupied concurrently in the lane.

[0214] The task scheduling module 803 is used to create a queuing state for platform tasks to wait for slots in the mapped lanes, and in response to the platform task obtaining a slot in the lane, to form an execution attempt for the platform task to submit the observation question to each observation platform under the execution budget constraint of the slot and collect the platform response to form an evidence set, the evidence set including answer evidence, source evidence and execution evidence; and to extract candidate semantics and candidate subjects from the original answer through a semantic extraction model, and determine the formal semantics that correctly belong to the target subject through preset semantic verification, and form answer semantics based on the formal semantics of the same original answer, and the answer semantics and the evidence set form the execution result.

[0215] The semantic aggregation module 804 is used to filter valid answer semantics based on a set of execution results, aggregate valid answer semantics associated with each observation question, and form cross-platform semantic conclusions for each observation question.

[0216] The subject evaluation module 805 is used to filter valid evaluation samples based on a set of execution results, collect evidence sets of observation problems and scene labels in the valid evaluation samples, and analyze them to obtain evaluation signals representing the strength of the target subject in the dimensions of overall cognitive coverage, scene performance and platform consistency. These signals are used to form structured evaluation conclusions and output them together with cross-platform semantic conclusions. The cross-platform semantic conclusions are used as explanatory clues for the evaluation signals representing the strength of the subject.

[0217] It should be noted that, in Figure 8 The various functional modules in the embodiments can be implemented, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, in the form of a program instruction product. A program instruction product includes one or a set of program instructions. When the program instructions are loaded and executed on a computer, all or part of the flow or function according to this disclosure is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0218] and, Figure 8 The apparatus disclosed in the embodiments can be implemented through other modular division methods. The apparatus embodiments described above are merely illustrative. For example, the module division is only a logical functional division, and in actual implementation, there may be other division methods. For example, a group of modules or modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces, and the indirect coupling or communication connection between devices or modules may be electrical or other forms.

[0219] in addition, Figure 8 The functional modules and sub-modules in the embodiments can be integrated into a single processing unit, or each module can exist physically separately, or two or more modules can be integrated into a single unit. The integrated unit can be implemented in hardware or as a software functional module. If the integrated unit is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0220] It should be specifically noted that the flowcharts describing the processes or methods in the embodiments of this disclosure can be understood as describing a module, segment, or portion of code comprising one or more sets of executable instructions configured to implement a specific logical function or process. Furthermore, the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved.

[0221] For example, Figure 1 The order of the steps in the method embodiments may vary in specific business scenarios and is not limited to the above description.

[0222] The technical mechanisms of the above embodiments can bring verifiable technical effects. The lane-bounded concurrent scheduling, analysis of control semaphores, and database single-lock limitation can be applied to the peak concurrency of webpage interaction, background data collection, analysis, and database writing. According to the applicant's simulation estimates, using the lane-bounded concurrent scheduling method of this disclosure, the total task completion time, theoretically lower bounded, can be significantly shortened compared to serial execution platform tasks. Therefore, the evaluation efficiency is greatly improved.

[0223] For example, when the platform task completion order changes, the initial platform order constrained by the initial index number is restored, ensuring a stable correspondence between input and output. For example, the independent state of response evidence and source evidence allows for the retention of completed original responses even when source loading times out or partially fails, locating the fault to the source acquisition stage. For example, checks on source authenticity and completeness can reduce the likelihood of text lists, site navigation, or meaningless short texts being mistakenly recorded as cited sources. For example, using only the latest valid attempt records to form execution results for subsequent structured evaluation conclusions and cross-platform semantic conclusions generated through cross-platform data aggregation / consolidation avoids the accumulation of historical retests and the mixing of expired records. For example, the combined use of strong evidence-first rule extraction and semantic extraction models can still output deterministic results and verification reasons even when the semantic extraction model is unavailable, has low confidence, or conflicts with explicit text, effectively constraining the randomness and illusion problems of the semantic extraction model. For example, scoring and risk capping can be configured, allowing scoring items, weights, thresholds, and capping reasons to be retrospectively and repeatedly calculated.

[0224] like Figure 9 The diagram shown illustrates the structure of a computer device according to an embodiment of the present disclosure.

[0225] The computer device may be a centralized or distributed computing system, a server group / server, a desktop computer, a user terminal, etc.

[0226] The computer device 900 includes a bus 901, a processor 902, and a memory 903. The processor 902 and the memory 903 can communicate via the bus 901. The memory 903 can store program instructions. The processor 902 executes the program instructions in the memory 903 to implement the method steps in the previous embodiments, such as... Figure 1 The method in the middle.

[0227] Bus 901 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of description, although only a single thick line is used in the diagram, it does not imply that there is only one bus or one type of bus.

[0228] In some embodiments, processor 902 may be implemented as a central processing unit (CPU), microprocessor module (MCU), system on chip (System on Chip), or field-programmable array (FPGA). Memory 903 may include volatile memory for temporary data storage during program execution, such as random access memory (RAM).

[0229] The memory 903 may also include non-volatile memory for data storage, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state disk (SSD).

[0230] In some embodiments, the computer device 900 may further include a communicator 904. The communicator 904 is used for communication with external devices. In specific examples, the communicator 904 may include one or more wired and / or wireless communication circuit modules. For example, the communicator 904 may include one or more of, such as a wired network card, a USB module, a serial interface module, etc. The wireless communication protocols followed by the wireless communication module include, for example, Nearfield Communication (NFC) technology, Infrared (IR) technology, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Bluetooth (BT), Global Navigation Satellite System (GNSS), etc.

[0231] This disclosure also provides a computer-readable storage medium storing program instructions that, when executed, perform the methods described in the previous embodiments.

[0232] That is, the method steps in the above embodiments are implemented as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as computer code that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium after being downloaded via a network, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special processor or programmable or special hardware (such as ASIC or FPGA).

[0233] Those skilled in the art should understand that there are many alternative implementations of the technical elements in the above embodiments.

[0234] In some embodiments, lane access to the observation platform may include at least one of the following: official interface, browser automation, controlled proxy, or other compliant access capabilities. The observation platform maps lanes to the corresponding type based on the configured access method.

[0235] In some embodiments, capacity, timeout, number of rounds, etc., can be configured, and are not limited to the values ​​in the current embodiment.

[0236] In some embodiments, the aggregation dimension can be expanded from a combination of observation problem and observation platform to a combination of dimensions such as observation problem, observation platform, region, language, model version, or observation window; the target subject is not limited to a brand, but can also be a product, organization, person, technology topic, or other entity to be observed. All the above alternative implementation methods, without departing from the spirit of this disclosure, are within the scope of protection of this disclosure.

[0237] In summary, this disclosure relates to the field of artificial intelligence technology, providing a method, apparatus, and storage medium for subject evaluation based on a heterogeneous artificial intelligence platform. The process involves acquiring evaluation requests carrying target subject and scene tags, mapping them to lanes with matching resource constraints, and then lazily forming execution attempts after obtaining a slot, collecting an evidence set containing answers, sources, and execution evidence. A semantic extraction model is used to extract candidate semantics from the original answers, which are then verified to obtain formal semantics belonging to the target subject. Answer semantics are generated from the formal semantics of the same answer, and combined with the evidence set to constitute the execution result of the platform task. Valid answer semantics are filtered and aggregated according to the observed questions to obtain cross-platform semantic conclusions. The execution results are collected, and the output is analyzed to provide evaluation signals and structured evaluation conclusions regarding the target subject's performance strength in three dimensions: overall cognitive coverage, scene performance, and platform consistency. The cross-platform semantic conclusions are used to provide explanatory clues. This disclosure combines bounded parallel scheduling of heterogeneous platforms, evidence tracking, conclusion explanation, and semantic verification to avoid illusions.

[0238] The above embodiments are merely illustrative of the principles and effects of this disclosure and are not intended to limit this disclosure. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this disclosure. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this disclosure should still be covered by the claims of this disclosure.

Claims

1. A subject evaluation method based on a heterogeneous artificial intelligence platform, characterized in that, include: The system acquires at least one observation question containing target subject information and associated scene tags, as well as evaluation requests from multiple observation platforms, and determines a set of platform tasks for each observation question to be evaluated on one observation platform; wherein, the multiple observation platforms are different generative artificial intelligence platforms. Obtain a platform capability profile for each observation platform and map it to lanes that match resource constraints; wherein, the resource constraints include the capacity representing the number of slots that can be executed concurrently in a lane; Creates a queued execution attempt for platform tasks, waiting in the mapped lane for a slot; In response to the platform task obtaining a lane slot, the queuing state of the execution attempt is updated to the execution state, and an execution context is created to execute the platform task under the execution budget constraint of the slot and obtain a set of execution results. This includes: submitting observation questions to each observation platform and collecting platform responses to form an evidence set, including response evidence, source evidence, and execution evidence; wherein the response evidence includes the original response; the execution evidence records the execution process information of the platform task; and extracting candidate semantics and candidate subjects from the original response through a semantic extraction model, and determining the formal semantics correctly belonging to the target subject through a preset semantic verification; forming response semantics based on the formal semantics of the same original response; wherein the response semantics and the evidence set form the execution result of the platform task. Based on a set of execution results, valid answer semantics are filtered, and the valid answer semantics associated with each observation question are aggregated to form a cross-platform semantic conclusion for each observation question. Based on the evidence set in a set of execution results, valid evaluation samples are screened, and the evidence set of observation problems and scene labels in the valid evaluation samples are collected. Based on this, the performance strength evaluation signals of the target subject in the dimensions of overall cognitive coverage, scene performance and platform consistency are analyzed to form structured evaluation conclusions and output them together with cross-platform semantic conclusions. The cross-platform semantic conclusions are used to provide explanatory clues for the performance strength evaluation signals.

2. The subject evaluation method according to claim 1, characterized in that, The process of mapping the lanes to resource constraints based on the platform capability profile includes: Based on the platform capability profile, the supported interaction methods are mapped to lanes with matching interaction capabilities, including at least one of the following: If the observation platform supports API interface access, then it will be mapped to the first type of lane that uses API access. If the observation platform allows independent webpage access in the background, it will be mapped to the second type of lane that uses the background webpage access method; If the observation platform requires foreground visibility / interaction, it is mapped to the third type of lane that uses foreground visibility / interaction to access the observation platform.

3. The subject evaluation method according to claim 1, characterized in that, The platform capability profile also includes login or session requirements, and the method further includes: The observation platform is determined based on the login or session requirements to determine whether it meets the platform task execution conditions and scheduling priority.

4. The subject evaluation method according to claim 1, characterized in that, The platform capability profile also includes a timeout policy; the method also includes: In response to obtaining a slot, the startup time is recorded and the execution time of the platform task is calculated accordingly; Based on the platform execution budget determined by the timeout policy of the observation platform corresponding to the platform task, check whether the execution time has exceeded the timeout.

5. The subject evaluation method according to claim 1, characterized in that, Also includes: Record the queuing time of platform tasks; Congestion detection is performed on the lane based on the interval between the queuing time and the start time when the execution attempt of the platform task is created.

6. The subject evaluation method according to claim 4, characterized in that, Also includes: Based on the number of platform tasks allocated to each lane and the lane capacity, the number of execution waves corresponding to each lane is determined. Based on the number of execution waves and the single-wave execution budget corresponding to each lane, the upper limit of the queuing time for each lane is determined; wherein, the single-wave execution budget is determined by the maximum value of the platform execution budget of the platform tasks assigned to the lane. The global queuing time limit for each lane is determined based on the maximum value among the queuing time limits for each lane. A watchdog timer is set based on the global queuing time limit. The watchdog timer is used to accumulate the execution time of the batch of platform tasks, and when the execution time reaches the global queuing time limit, the platform tasks that have not yet started execution are stopped from continuing to queue.

7. The subject evaluation method according to claim 1, characterized in that, Also includes: Based on the initial order of multiple observation platforms in the evaluation request, set the initial index number associated with each observation platform, and bind the initial index number of the corresponding platform to each result slot of a group of platform tasks belonging to the same observation problem within the group. Based on the initial index number, the execution result of each platform task is stored in the corresponding result slot to restore the index order to be consistent with the initial order.

8. The subject evaluation method according to claim 1, characterized in that, The response evidence, source evidence, and execution evidence each include a response state, source state, and execution state that are independently determined and associated through tracing information. The method further includes: Based on whether the collected original answers are complete, partially valid, or invalid, the answer status is set to available, partially available, or unavailable, respectively. Based on the different levels of completeness of the collected source evidence, the source status is set to complete, partially collected successfully, collected unsuccessfully, or the platform does not support source collection. Based on whether the platform task is executed successfully or failed, the execution status is set to successful or failed respectively; wherein, the failed execution status is associated with reason information.

9. The subject evaluation method according to claim 1, characterized in that, The various formal semantics based on the same original answer form the answer semantics, including: Based on the coverage of formal semantics in each candidate semantics of the same original answer, the semantic state of the generated answer is set to available, partially available, or unavailable; The process of filtering valid answer semantics based on a set of execution results and aggregating valid answer semantics associated with each observed question includes: Filter the semantic states of responses that are available or partially available from the semantic states of the responses associated with the observed question; The effective answer semantics selected through aggregation are used to form the cross-platform semantic conclusion.

10. The subject evaluation method according to claim 9, characterized in that, The cross-platform semantic conclusion includes at least one of consensus terms, divergence terms, and common omission terms. The consensus terms / divergence terms are generated based on information extracted from the semantic consensus / semantic divergence between the responses of at least two of the observation platforms. The common omission terms are generated based on information extracted from the case where the responses of at least two of the observation platforms omit the same preset content of the target subject.

11. The subject evaluation method according to claim 1, characterized in that, Also includes: Among the execution results obtained from multiple execution attempts for the same platform task, only the latest execution result is selected to form the cross-platform semantic conclusion and structured evaluation conclusion.

12. The subject evaluation method according to claim 1, characterized in that, The process involves filtering valid evaluation samples based on the evidence set from a set of execution results, compiling the evidence set of the observed issues and scene labels from the valid evaluation samples, and analyzing them to obtain evaluation signals characterizing the target subject's performance in the dimensions of overall cognitive coverage, scene performance, and platform consistency, including: The valid evaluation samples are determined based on the availability of the evidence in the responses. Based on the visibility results extracted from the answer evidence in the valid evaluation samples through rules, the overall coverage signal representing the maximum loss value of the target subject when switching between different recommendation rankings is obtained; Based on the results of the visibility results extracted from the answer evidence in the valid evaluation samples and the corresponding observation question labels, a scene breadth signal representing the overall recommendation intention strength / weakness of multiple observation platforms for the target subject in different scenarios is obtained. Based on the visibility results extracted from the rules of answer evidence in the valid evaluation samples, a platform consistency signal representing the discreteness of the target subject's performance on each observation platform in the same question dimension is obtained; The main signal and auxiliary signal are determined based on the signal strength of the overall coverage signal, the scene breadth signal, and the platform consistency signal; wherein, the main signal corresponds to the main conclusion in the structured evaluation conclusion.

13. The subject evaluation method according to claim 12, characterized in that, The determination of the primary and secondary signals based on the signal strength values ​​of the overall coverage signal, scene breadth signal, and platform consistency signal includes: Qualified signals are selected based on threshold conditions that include the minimum number of valid platforms, the minimum number of valid scenarios, and the minimum number of valid evaluation samples. Candidate signals are selected from qualified signals based on the condition that the intensity value reaches a significant threshold. Based on signal strength comparison, the primary signal among the candidate signals is determined; if the difference in strength values ​​among the multiple candidate signals with the highest signal strength values ​​is close to a threshold, the primary signal among the multiple candidate signals is determined based on the condition that at least one of the number of effective platforms, the number of effective scenes, and the number of effective evaluation samples covered by the candidate signals tends to be higher, and the remaining candidate signals are auxiliary signals.

14. A subject evaluation device based on a heterogeneous artificial intelligence platform, characterized in that, include: The input module is used to acquire at least one observation question containing target subject information and associated scene tags, as well as evaluation requests from multiple observation platforms, and to determine a set of platform tasks for each observation question to be evaluated by one observation platform; wherein, the multiple observation platforms are different generative artificial intelligence platforms. The mapping module is used to obtain the platform capability profile of each observation platform and map it to lanes that match resource constraints; wherein, the resource constraints include the capacity representing the number of slots that can be executed concurrently in a lane; The task scheduling module is used to create queuing states for execution attempts of platform tasks, waiting for slots in the mapped lanes. In response to a platform task obtaining a slot in a lane, the module updates the queuing state of the execution attempt to an execution state and creates an execution context to execute the platform task under the execution budget constraints of the slot and obtain a set of execution results. This includes: submitting observation questions to each observation platform and collecting platform responses to form an evidence set, including answer evidence, source evidence, and execution evidence; wherein the answer evidence includes the original answer; the execution evidence records the execution process information of the platform task; and extracting candidate semantics and candidate subjects from the original answer using a semantic extraction model, and determining the formal semantics correctly belonging to the target subject through preset semantic verification; forming answer semantics based on the formal semantics of the same original answer; wherein the answer semantics and the evidence set form the execution results of the platform task. The semantic aggregation module is used to filter valid answer semantics based on a set of execution results, aggregate valid answer semantics associated with each observation question, and form cross-platform semantic conclusions for each observation question. The subject evaluation module is used to filter valid evaluation samples based on the evidence set in a set of execution results, collect the evidence set of observation problems and scene labels in the valid evaluation samples, and analyze them to obtain evaluation signals representing the strength of the target subject's performance in the dimensions of overall cognitive coverage, scene performance and platform consistency, so as to form a structured evaluation conclusion and output it together with the cross-platform semantic conclusion; wherein, the cross-platform semantic conclusion is used to provide explanatory clues for the performance strength evaluation signals.

15. A computer-readable storage medium, characterized in that, It stores a computer program or instructions, which, when executed, perform the subject evaluation method as described in any one of claims 1 to 13.

Images