Natural language driven situation awareness control method and system based on task intention and medium

Abstract

The invention discloses a natural language driven situation awareness control method and system based on task intention and a medium. Relates to the technical field of artificial intelligence and man-machine interaction. Generating a natural language template library and a task intention field table based on the three-table knowledge; constructing an interface-data double-mapping knowledge graph by taking three-table knowledge as a construction basis and taking a natural language template library and a task intention field table as retrieval rearrangement tools, and then performing semantic retrieval and anaphora resolution to generate structured intention data; on the premise of not depending on a fixed layout, a user instruction is analyzed into a task intention, stable anchoring of an interface object is achieved through semantic positioning, and the task intention is automatically compiled into a system API call or interface event sequence; and meanwhile, the availability and compliance of execution are guaranteed through a closed-loop mechanism of credible execution scoring and minimization clarification, so that the operation complexity is effectively reduced, and the response efficiency and intelligent interaction level of the situation awareness system are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and human-computer interaction, and particularly relates to a natural language driven situation awareness control method and system based on task intention and a medium. BACKGROUND

[0002] The situation awareness system is widely used in city management, emergency response, command and dispatch, etc. Its core functions include real-time monitoring, data analysis and comprehensive command and dispatch to assist decision makers to quickly understand and respond to complex dynamic business environment. The existing system usually integrates multiple topics, multiple panels and a large number of controls, the interface contains hundreds of buttons, switches, drop-down filters and interactive cards, and there is a linkage relationship between different panels. When the staff performs continuous operations such as "query-locate-linkage-export / assignment", they need to frequently switch, find and set parameters one by one between multiple topics and interface objects, which has high learning and using cost, and the error rate and delay risk are particularly prominent in high-pressure shift scenarios.

[0003] However, the existing interactive mode still has obvious deficiencies: (1) Most solutions are centered on "control / component" for static mapping (such as relying on fixed position, DOM path or control ID), which is easy to fail when the interface is revised or the layout is changed; (2) Lack of unified expression for "task", which cannot stably decompose a user's natural language into executable operation micro-processes (macros), resulting in the need for manual concatenation of cross-panel and cross-topic linkage steps; (3) Lack of semantic alignment and evidence-based positioning mechanism between interface objects and business indicators / indicators, making it difficult to ensure that the "found is the right one object"; (4) Lack of quantifiable evaluation and gating of data freshness QoS and role or permission compliance, making it difficult to ensure execution reliability in emergency scenarios; (5) Existing large language model driven configuration / visual control ideas usually focus on directly mapping natural language to component control commands or configuration files to create or modify charts and interface properties, but pay little attention to the problems of "semantic positioning of interface objects, operation micro-process orchestration across panels, and reliable execution under data freshness QoS and permission constraints" in multi-topic situation systems, making it difficult to meet the requirements of low learning cost, low delay and high stability in real combat scenarios. SUMMARY

[0004] The technical problem to be solved by the present application is that with the development of artificial intelligence and natural language processing technology, large language models have the ability to understand user intent and task decomposition, but in a multi-topic situation awareness system, there is still a lack of a general method that can stably convert natural language into executable operation sequences across panels and controls, and reliably run under data freshness and permission constraints. The existing approach needs to rely on fixed layout or single component mapping, which is easy to fail when the interface is revised, the parameter caliber is different, or the data is temporarily unavailable, and it is difficult to meet the high-reliability, low-latency human-machine collaboration demand; The purpose of the present application is to provide a natural language driven situation awareness control method, system and medium based on task intent, which can parse user instructions into task intent containing topics, targets, actions and parameters without relying on fixed layout, use semantic positioning to stably anchor interface objects, and automatically compile task intent into system API calls or interface event sequences; At the same time, through the trusted execution scoring and the closed-loop mechanism of minimizing clarification, the availability and compliance of execution are guaranteed, thereby effectively reducing the operation complexity and improving the response efficiency and intelligent interaction level of the situation awareness system.

[0005] The present application is realized by the following technical solutions:

[0006] The present application provides a natural language driven situation awareness control method based on task intent, which includes:

[0007] Obtain perception data and extract three-table knowledge from the perception data, the three-table knowledge including: object table, capability table and parameter table;

[0008] Generate a natural language template library and a task intent field table based on the three-table knowledge; and construct an interface-data double-mapping knowledge graph based on the three-table knowledge as the construction basis, and the natural language template library and the task intent field table as the retrieval and rearrangement tools;

[0009] In response to a natural language instruction, perform semantic retrieval and anaphora resolution based on the natural language template library, the task intent field table and the interface-data double-mapping knowledge graph table to generate structured intent data; the structured intent data including task intent and a positioning candidate set;

[0010] Compile the task intent in the structured intent data into an operation microflow, and generate an operation instruction format list;

[0011] Perform step by step according to the operation instruction format list, and return the execution result summary;

[0012] Perform a trusted execution scoring on the execution result summary.

[0013] Further optimization scheme is that the perception data includes: a front-end structure definition document, a front-end interface description, a front-end construction product and an interface operation telemetry log;

[0014] The front-end build product includes a DOM tree, component metadata, accessibility semantic attributes, and stable selector identification.

[0015] Further optimization scheme is that the object table is an interface target main data, including: name, alias, label text, index name, unit of measurement, subject, and visibility condition.

[0016] The capability table is a list of executable actions, including: action name, target type, call method, parameter mode, preconditions, postconditions, linkage range, and impact domain.

[0017] The parameter table is a parameter specification, including: type, value range, default value, verification rule, and data caliber identification.

[0018] Further optimization scheme is that the task intent field table defines a field set, including: subject, target, action, parameter, data freshness, role or permission, and interaction constraint.

[0019] Further optimization scheme is that the task intent definition includes: subject, target, action, parameter, data freshness, role or permission, and interaction constraint.

[0020] The positioning candidate set definition includes: target object identification, positioning confidence, and semantic positioning evidence vector.

[0021] Further optimization scheme is that the structured intent data generation method includes:

[0022] Obtain a natural language instruction and perform hierarchical analysis on the text of the natural language instruction:

[0023] Based on the natural language template library and the task intent field table, first extract the field candidate values of the subject, target, action, and parameter from the natural language instruction;

[0024] Combine the interface-data double-mapping knowledge graph to perform semantic retrieval and anaphora resolution, map the user language in the natural language instruction to the controlled vocabulary and parameter enumeration, and retrieve the interface object consistent with the target semantics in the object table to generate a semantic positioning evidence vector.

[0025] Further optimization scheme is that the intent in the structured intent data is compiled into an operation micro-process, and an operation instruction format list is generated; including the method:

[0026] The execution plan engine compiles the task intention into an operation micro-process and generates a list of operation instruction formats in sequence, each of which contains at least the following fields: target object identification, action identification, parameter, precondition, post-check, rollback strategy, timeout and power token;

[0027] The execution plan engine selects an optimal execution channel according to the capability table, and the selection rules include: preferentially assembling API requests through native interface adapters; when the target only supports interface interaction, automatically switch to interface event adapters.

[0028] A further optimization scheme is to execute the operation instruction format list step by step and return the execution result summary, which includes:

[0029] The execution plan engine uses the trusted execution score of the scheduling plan as a gating mechanism to execute the operation instruction format list;

[0030] Before each action starts, the execution plan engine checks the pre-parameters and automatically fills in the missing parameters; the pre-parameters include object visibility, required roles / required permissions, and data freshness QoS;

[0031] After invoking the corresponding adapter to complete the action, the post-parameters are used to instantly check the action result; the post-parameters include: interface object state change, return data structure and quantity, and prompt message matching;

[0032] If the check fails or times out, the rollback path is started;

[0033] After all actions are completed, the execution result summary is returned.

[0034] A further optimization scheme is that the trusted execution score is obtained by:

[0035] Based on the trusted execution score gating mechanism, the semantic matching, interface positioning, data availability and permission compliance are nonlinearly fused to realize the trusted execution score.

[0036] The present scheme also provides a natural language driven situational awareness control system based on task intention, which is used to realize the natural language driven situational awareness control method based on task intention; the system includes:

[0037] A preprocessing module is configured to obtain perception data and extract three-table knowledge from the perception data, the three-table knowledge including: object table, capability table and parameter table;

[0038] A knowledge generation module is configured to generate a natural language template library and a task intention field table based on the three-table knowledge, and construct an interface-data double-mapping knowledge graph based on the three-table knowledge as a construction basis, the natural language template library and the task intention field table as retrieval and rearrangement tools.

[0039] A retrieval and resolution module is configured to perform semantic retrieval and reference resolution based on the natural language template library, the task intention field table and the interface-data double-mapping knowledge graph table to generate structured intention data in response to a natural language instruction, wherein the structured intention data includes a task intention and a positioning candidate set.

[0040] A compiling module is configured to compile the task intention in the structured intention data into an operation micro-process and generate an operation instruction format list.

[0041] An execution module is configured to perform step by step according to the operation instruction format list and return an execution result summary.

[0042] The present application also provides a computer readable medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the task intention-based natural language driven situation awareness control method as described above.

[0043] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0044] 1. The task intention-based natural language driven situation awareness control method, system and medium provided by the present application can parse user instructions into task intentions containing topics, targets, actions and parameters without relying on a fixed layout, realize stable anchoring of interface objects by semantic positioning, and automatically compile the task intentions into system API calls or interface event sequences; meanwhile, the usability and compliance of execution are guaranteed by a trusted execution scoring and minimum clarification closed-loop mechanism, thereby effectively reducing operation complexity and improving the response efficiency and intelligent interaction level of the situation awareness system.

[0045] 2. The task intention-based natural language driven situation awareness control method, system and medium provided by the present application are different from the existing idea of directly mapping natural language to a single component command, and take the task intention as the core to realize layout-independent target positioning in combination with an interface-data double-mapping knowledge graph and a semantic positioning evidence vector, and complete continuous operation arrangement by an operation micro-process, thereby significantly reducing the learning and use costs caused by interface complexity.

[0046] 3. The task-intent-based natural language driven trend perception control method, system and medium provided by the application, which compiles the task intent into the shortest executable path through an execution plan engine, preferentially adopts a native interface adapter, automatically switches an interface event adapter when the native interface adapter is missing and keeps a power token control; in cooperation with parameter normalization and default value completion, can obviously shorten the total time delay from "expressing needs" to "completing linkage, exporting and dispatching", and reduce the risk of cross-panel manual searching and error operation.

[0047] 4. The task-intent-based natural language driven trend perception control method, system and medium provided by the application, which relies on a three-table system of an object table, a capability table and a parameter table and a natural language template library, and only needs to import control capability and parameter specification to reuse when a new topic or a new system is connected; at the same time, uniformly restricts data freshness and role permissions, adapts to multi-form interfaces such as city operation screens, on-duty operators and Web, and has universality and scalability.

[0048] 5. The task-intent-based natural language driven trend perception control method, system and medium provided by the application, which realizes high robustness and full-link auditability through a gating-clarification-execution closed loop of trusted execution scoring and minimization of clarification, and automatically gives an alternative path or safe degradation and records operation logs when there is positioning ambiguity, data unavailability or permission limitation.

[0049] 6. The task-intent-based natural language driven trend perception control method, system and medium provided by the application, which, based on a semantic adaptation optimization mechanism, performs incremental learning on the natural language template library, the interface-data dual-mapping knowledge graph and the measurement parameters, deposits high-frequency successful processes into reusable operation macros and directly calls them in similar instructions; automatically evolves with business versioning, reduces maintenance cost and continuously improves recognition accuracy and interaction stability. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the application, and therefore should not be considered as a limitation to the scope. For those skilled in the art, other related drawings can also be obtained without creative labor. In the drawings:

[0051] Figure 1 It is a flowchart of the task-intent-based natural language driven trend perception control method.

[0052] Figure 2 It is a schematic diagram of the generation process of the three-table knowledge and the interface-data dual-mapping knowledge graph.

[0053] Figure 3Generate a process diagram for the structured intent data generation process;

[0054] Figure 4 Perform a process diagram for the operation instruction format list;

[0055] Figure 5 Structure diagram for a natural language driven situational awareness control system based on task intent. DETAILED DESCRIPTION

[0056] To make the purpose, technical solutions and advantages of the present application clearer, further detailed description of the present application is given below in combination with examples and drawings, the illustrative embodiments of the present application and their descriptions are only used to explain the present application, and do not limit the present application.

[0057] With the development of artificial intelligence and natural language processing technology, large language models have the ability to understand user intent and task decomposition, but in multi-topic situational awareness systems, there is still a lack of a general method that can stably convert natural language into executable operation sequences across panels and controls, and reliably run under data freshness and permission constraints. In view of this, the present application provides the following embodiments to solve the above technical problems:

[0058] Embodiment 1: The present embodiment provides a natural language driven situational awareness control method based on task intent, as shown in Figure 1 , which comprises:

[0059] Step one, as shown in Figure 2 , obtain the perception data, and extract the three-table knowledge from the perception data, the three-table knowledge includes: object table, capability table and parameter table;

[0060] The perception data includes: front-end structure definition document, front-end interface description, front-end construction product and interface operation telemetry log;

[0061] The front-end construction product includes: DOM tree, component metadata, accessibility semantic attribute and stable selector identifier.

[0062] Step two, generate a natural language template library and a task intent field table based on the three-table knowledge; and take the three-table knowledge as the construction basis, and take the natural language template library and the task intent field table as the retrieval and rearrangement tools, to construct an interface-data double-mapping knowledge graph;

[0063] The object table OTB is the interface target master data, including: name, alias, label text, index name, unit of measurement, subject and visibility condition;

[0064] The capability table ACT is a list of executable actions, including: action name, target type, invocation method (such as API / script / shortcut key), parameter mode, pre-condition, post-condition, linkage range and impact domain;

[0065] The parameter table PRM is a parameter specification, including: type, enumeration / value range, default value, verification rule and data caliber identification.

[0066] A layout-independent stable identifier (view identifier VID, object identifier OID) and a structural fingerprint are generated for each interface object, a interface-data double mapping knowledge graph UDKG is constructed, which is the basic knowledge for natural language and system function mapping.

[0067] Based on the object table OTB, the capability table ACT and the parameter table PRM, a natural language template library NLT and a task intent field table TTI-F are generated offline;

[0068] The task intent field table TTI-F defines a field set: topic Topic, target Target, action Action, parameter Params, data freshness QoS, role or permission Role, and interaction constraint Constraints;

[0069] The enumeration, value range and default value in the parameter table are written into the placeholder of the natural language template library to form a template instance that can be constrained and filled; at the same time, a semantic positioning evidence vector SEV is generated for each target, which is used for subsequent semantic positioning and confidence calculation. Thus, a bidirectional mapping library of "example-template-parameter" is obtained, and a subsequent operation microflow is compiled; optionally, the natural language template library and the synonym library are continuously updated based on online interaction logs.

[0070] Step three, in response to a natural language instruction, semantic retrieval and reference resolution are performed based on the natural language template library, the task intent field table and the interface-data double mapping knowledge graph table to generate structured intent data; the structured intent data includes a task intent and a positioning candidate set;

[0071] The task intent field table TTI-F defines a field set including: topic, target, action, parameter, data freshness, role or permission, and interaction constraint. The structured intent data serves as the intermediate decision basis for function call, which can map natural language to the correct interface object and its executable ability without relying on the fixed layout of the page; accordingly, it is convenient to compile the task intent TTI into an operation micro-process in the subsequent step, to realize automatic scheduling and precise triggering across panels and across topics. For example: "show the top ten congestion ranking this week and export Excel" corresponds to the task intent TTI which can be parsed as: topic Topic = traffic operation, target Target = congestion ranking table, action Action = filtering + sorting + exporting, data freshness QoS = online, role or permission Role = dispatch seat, and interaction constraint Constraints = shortest interaction path.

[0072] As shown in Figure 3 , after receiving the natural language instruction input by the user, the text of the natural language instruction is hierarchically parsed, and the task intent TTI and the positioning candidate set are generated according to the process of "template matching-semantic retrieval-parameter normalization-target positioning". Specifically: based on the natural language template library NLT and the task intent field table TTI-F, the candidate values of the fields such as topic, target, action, and parameter are first extracted from the instruction; then, combined with the interface-data double-mapping knowledge graph UDKG, semantic retrieval and anaphora resolution are performed, so that the user's language (including aliases and colloquial expressions) is mapped to the controlled vocabulary and parameter enumeration; then, the normalization of parameter unit / range and the completion of default values are completed, and the interface objects consistent with the "target" semantics are retrieved in the object table OTB, to generate a semantic positioning evidence vector SEV (such as text similarity, measurement unit matching, stable selector identifier matching, structural fingerprint consistency, historical click priori) and its confidence score, which are used for subsequent executability evaluation and minimization clarification.

[0073] Step four, as shown in Figure 4 , the intent in the structured intent data is compiled into an operation micro-process, and an operation instruction format list is generated;

[0074] The task intent definition includes: topic, target, action, parameter, data freshness, role or permission, and interaction constraint;

[0075] The positioning candidate set definition includes: target object identifier, positioning confidence, and semantic positioning evidence vector.

[0076] The generation method of the structured intent data includes:

[0077] The natural language instruction is obtained, and the text of the natural language instruction is hierarchically parsed:

[0078] Based on the natural language template library and the task intent field table, candidate values ​​for the topics, objectives, actions, and parameters are first extracted from the natural language instructions.

[0079] By combining the interface-data dual-mapping knowledge graph for semantic retrieval and referential resolution, user terms (including aliases and colloquial expressions) in natural language instructions are mapped to a controlled vocabulary and parameter enumeration. Interface objects that are consistent with the target semantics are retrieved from the object table to generate semantic localization evidence vectors.

[0080] The step of compiling the intents in the structured intent data into operation micro-flows and generating a list of operation instruction formats includes the following methods:

[0081] The execution plan engine compiles the task intent into operation micro-flows and generates a list of operation instruction formats to be executed in sequence. Each operation instruction format list contains at least the following fields: target object identifier, action identifier, parameters, preconditions, post-validation, rollback strategy, timeout, and idempotent token.

[0082] The execution plan engine selects the optimal execution channel based on the capability table. The selection rules include: prioritizing the assembly of API requests through the native interface adapter; and automatically switching to the interface event adapter (such as atomic event sequences like focus, expand, select, input, and click) when the target only supports interface interaction.

[0083] Step 5: Follow the operation instruction format list step by step, and return a summary of the execution results; this step specifically includes:

[0084] The execution plan engine uses the trusted execution score of the scheduling plan as a list of gating execution operation instruction formats;

[0085] Before each action begins, the execution plan engine checks the pre parameters and automatically fills in any missing parameters; the pre parameters include object visibility, required roles or permissions, and data freshness QoS.

[0086] After the corresponding adapter is invoked to complete the action, the action result is validated in real time using the post parameters; the post parameters include: changes in the state of the UI object, the returned data structure and quantity, and matching of the prompt message;

[0087] If the verification fails or times out, a fallback path is initiated (e.g., switching to a similar target candidate, downgrading to the default time window, or using a read-only interface) to ensure the process is robust.

[0088] After all actions are completed, a summary of execution results is returned. The summary includes: overall status, step-by-step status codes, key information extraction (such as exported file identifier, alarm handling order number), and interface change summary; and "original instruction, task intent TTI, selected target object identifier OID, semantic location evidence vector SEV, operation instruction format OCF sequence, and execution log" are written to the operation log OpLog; through the above micro-process orchestration and gating verification, stable automatic scheduling and precise triggering across panels and topics are achieved from natural language.

[0089] Both native interface adapters and UI event adapters are presented externally as a unified list of operation instructions. To ensure executability, the Execution Plan Engine (EPE) uploads the scheduling plan for trusted execution scoring before scheduling, using this score as a gating mechanism. When the score meets the threshold, step-by-step scheduling begins; otherwise, minimization clarification or candidate replacement is triggered. This process does not rely on fixed layouts or coordinates, utilizing a UI-data dual-mapping knowledge graph (UDKG) and semantic localization evidence vectors (SEVs) to ensure stable location of the target object and its executable capabilities even after UI redesigns.

[0090] The method for obtaining the trusted execution score includes:

[0091] The trusted execution score is obtained by nonlinearly fusing semantic matching, interface location, data availability and permission compliance based on the trusted execution score gating mechanism, which summarizes the current execution results.

[0092] Specifically, to improve the executability of natural language commands in multi-topic, multi-panel environments, this solution introduces a Trusted Execution Score (CES) gating mechanism; the Trusted Execution Score nonlinearly fuses four types of signals: semantic matching, interface location, data availability, and permission compliance. [0, 1] 4 ;in, These represent the retrieval or re-ranking score of the semantic matching score (derived from the Natural Language Template Knowledge (NLT) and the Interface-Data Dual Mapping Knowledge Graph (UDKG)), the locator confidence score (derived from the Semantic Localization Evidence Vector (SEV)), the data freshness QoS score, and the permission consistency score, respectively; the components are arranged in descending order as follows: ,remember Let be the set of "the first i indicators", where The overall score uses CI points:

[0093]

[0094] in, The trusted execution score is represented by μ; μ represents the fuzzy measure FM defined on {semantic matching index, locator confidence index, data freshness index, and permission consistency index}, which satisfies monotonicity and normalization; optionally, μ({semantic matching index, locator confidence index})>μ({semantic matching index})+μ({locator confidence index}) is set to reflect the synergistic gain of "semantic + location";

[0095] The specific gating rules of the execution plan engine include:

[0096] The preset first threshold τ1 < the second threshold τ2;

[0097] When the trustworthy execution score S CES When the value is ≥τ2, execute directly;

[0098] When τ1 ≤ Trusted Execution S CES When <τ2, trigger the minimization of the clarified MCS and only supplement key fields (such as target objects or key parameters).

[0099] When the trustworthy execution score S CES When <τ1, a list of candidate intents (including target object identifier OID, evidence summary, and expected action) is presented for the user to select;

[0100] If a permission conflict or data unavailability is detected, a compliant alternative path or security downgrade solution will be provided first.

[0101] This solution also continuously records information such as "original instructions, task intent (TTI), candidate and selected target object identifiers (OID), semantic localization evidence vectors (SEV), operation instruction format (OCF) sequence, step-by-step verification results, and final feedback," forming an operation log (OpLog). The operation log (OpLog) is used for semantic adaptation optimization (SAO): iteratively updating the template weights and thesaurus of the Natural Language Template Library (NLT) based on real interaction samples, fine-tuning the entity aliases and calibrated mappings of the Interface-Data Dual Mapping Knowledge Graph (UDKG), calibrating the measurement parameters and reordering model of the Fuzzy Measure (FM), and generating a regression test set for offline acceptance. At the same time, high-frequency successful micro-processes are precipitated as reusable operation macros (Macro), which can be directly reused in subsequent similar instructions to reduce latency.

[0102] This solution achieves robust execution, auditable traceability, and continuous optimization driven by natural language through a closed loop of Trusted Execution Score (CES) gating mechanism, Minimized Clarification Score (MCS), and Semantic Adaptation Optimization Learning (SAO).

[0103] Example 2

[0104] This embodiment provides a task intent-based natural language-driven situational awareness control system to implement the task intent-based natural language-driven situational awareness control method described in Embodiment 1; such as Figure 5 As shown, the system includes:

[0105] The preprocessing module is used to acquire the perception data and extract three tables of knowledge from the perception data, which include: an object table, a capability table, and a parameter table.

[0106] The knowledge generation module is used to generate a natural language template library and a task intent field table based on the knowledge of the three tables; and to construct an interface-data dual mapping knowledge graph using the knowledge of the three tables as the foundation and the natural language template library and the task intent field table as retrieval and rearrangement tools.

[0107] The retrieval and resolution module is used to respond to natural language commands by performing semantic retrieval and referential resolution based on a natural language template library, task intent fields, and an interface-data dual-mapping knowledge graph table to generate structured intent data; the structured intent data includes task intent and a location candidate set;

[0108] The compilation module is used to compile the intents in the structured intent data into operation micro-flows and generate a list of operation instruction formats;

[0109] The execution module executes the operation instructions step by step according to the list of format and returns a summary of the execution results.

[0110] Example 3

[0111] This embodiment provides a computer-readable medium storing a computer program. The computer program, when executed by a processor, can implement the task intent-based natural language-driven situational awareness control method as described in Embodiment 1; specifically, it performs the following steps:

[0112] Step 1: Acquire the perception data and extract three tables of knowledge from the perception data. The three tables of knowledge include: object table, capability table and parameter table.

[0113] Step 2: Generate a natural language template library and a task intent field table based on the knowledge of the three tables; and construct an interface-data dual mapping knowledge graph using the knowledge of the three tables as the foundation and the natural language template library and the task intent field table as retrieval and rearrangement tools.

[0114] Step 3: In response to natural language instructions, semantic retrieval and referential resolution are performed based on the natural language template library, task intent field table, and interface-data dual mapping knowledge graph table to generate structured intent data; the structured intent data includes task intent and location candidate set;

[0115] Step four: Compile the intents in the structured intent data into operation micro-flows and generate a list of operation instruction formats;

[0116] Step 5: Follow the operation instruction format list step by step and return the summary of execution results.

[0117] Specifically, this embodiment uses the situational awareness screen of the city operation management center as a typical scenario for illustration as follows:

[0118] Constructing a three-table knowledge and interface-data dual-mapping knowledge graph:

[0119] The large screen in the city operation management center includes multiple topics such as social governance, traffic management, culture and tourism, and environmental protection, and the interface integrates thousands of target objects and controls.

[0120] During the initialization phase, based on the front-end structure definition files and interface descriptions of each module, the object table OTB, capability table ACT, and parameter table PRM are extracted, and view identifier VID and object identifier OID are generated.

[0121] Taking the "Monthly Air Quality Report" as an example, the object table OTB records: Name = Air Quality Report, Alias ​​= Air Quality Trend, Report; Topic = Environmental Protection; Visibility Condition = Displayed after entering the Environmental Protection topic. The capability table ACT records: action_id = {Filter, Time Switch, Sort, Export}; Calling Method = {API, Interface Event}; Preconditions = {Target Visible, Read-Only Role Available}; Post-Validation = {Chart Rendering Completed, Data Points ≥ N}. The parameter table PRM records: Period ∈ {Current Day, Last 7 Days, Last 30 Days, Current Month, Last Month}; Format ∈ {png,xlsx}; Default Period = Current Month. Based on this, the system constructs a dual-mapping knowledge graph UDKG (interface-data), and generates a Natural Language Template Library (NLT) and a Task Intent Field Table (TTI-F), forming a bidirectional mapping library of "Example—Template—Parameter".

[0122] Semantic retrieval and referential resolution generate structured intent data: The command center operator issues the instruction via microphone: "Please retrieve the monthly air quality report from the environmental protection topic." The system converts the speech to text, extracts fields based on the Natural Language Template Library (NLT) and TTI-F, and retrieves the target from the Interface-Data Dual Mapping Knowledge Graph (UDKG) to obtain the task intent TTI: Topic=Environmental Protection, Target=Air Quality Report, Action=View, Params={period=Current Month}, QoS=Online, Role=Read-Only Access, Constraints=Shortest Interaction Path. Simultaneously, a location candidate set L={〈OID_a, Confidence 0.92, Evidence=[Name Match, Unit Match, Structural Fingerprint Consistency]〉, 〈OID_b, 0.61,…〉} is generated. The system performs a trusted execution score based on "semantic matching, location confidence, data freshness, and permission consistency"; in this scheme, S... CES If the threshold τ2 is greater than or equal to 2, proceed to the execution phase without further clarification.

[0123] Generate a list of operation instruction formats and execute them sequentially: The Execution Plan Engine (EPE) compiles the Task Intent (TTI) into an Operation Microflow (OpFlow), generating a list of operation instruction formats (OCFs) to be executed in sequence. Example:

[0124] 1) OCF-1: target_oid=OID_a, action_id=Focus on topic, params={topic=Environmental Protection}, pre={Target Visible}, post={Topic Page Activation}, fallback={Direct Navigation to Topic Route}.

[0125] 2) OCF-2: target_oid=OID_a, action_id=view, params={}, pre={chart container visible}, post={rendering start}, fallback={refresh container}.

[0126] 3) OCF-3: target_oid=OID_a, action_id=time switch, params={period=current month}, pre={existing period enumeration}, post={chart data points ≥ N}, fallback={period=last 30 days}.

[0127] The Execution Plan Engine (EPE) prioritizes fulfilling native API requests; if a specific action lacks an API, it automatically switches to a UI event sequence (focus, expand, select). Post-validation is performed after each step; if a failure or timeout occurs, a fallback strategy is executed. After the process concludes, the system returns a summary of execution results and writes it to the Operation Log (OpLog), which includes the selected object identifier (OID), the operation instruction format (OCF) sequence, and key outputs (such as the exported file identifier).

[0128] The specific gating rules of the execution plan engine include:

[0129] If semantic retrieval yields multiple similar targets, or if it detects that period=month is unavailable in the current data source, then S CES Falling into τ1≤S CES In the interval <τ2, the system triggers a minimal clarification MCS, providing only necessary options, such as "Please select report granularity: Monthly / Annual / Current Status". After the user selects "Monthly Report", execution continues, and the "Original Instruction, Task Intent (TTI), Candidate and Selected Targets, Evidence Summary, Operation Instruction Format Sequence, and Verification Result" are written to the Operation Log (OpLog). The system optimizes the SAO mechanism based on semantic adaptation, using the Operation Log (OpLog) to update the synonym weights of the Natural Language Template Library (NLT) and the alias mappings of the Interface-Data Dual Mapping Knowledge Graph (UDKG), thus accumulating this instruction as a reusable operation macro for direct invocation in subsequent similar requests, reducing latency.

[0130] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A natural language-driven situational awareness control method based on task intent, characterized in that, include: Acquire sensing data and extract three tables of knowledge from the sensing data, including: an object table, a capability table, and a parameter table; Based on the knowledge of the three tables, a natural language template library and a task intent field table are generated; and using the knowledge of the three tables as the foundation, and the natural language template library and the task intent field table as retrieval and rearrangement tools, an interface-data dual mapping knowledge graph is constructed. In response to natural language instructions, semantic retrieval and referential resolution are performed based on a natural language template library, a task intent field table, and an interface-data dual-mapping knowledge graph table to generate structured intent data; the structured intent data includes task intent and a location candidate set; The task intents in the structured intent data are compiled into operation micro-flows, and an operation instruction format list is generated; The operation instructions are executed step by step according to the list of instruction formats, and the summary of execution results is returned.

2. The task intent-based natural language-driven situational awareness control method according to claim 1, characterized in that, The sensing data includes: front-end structure definition document, front-end interface description, front-end build artifacts and interface operation telemetry logs; The front-end build artifacts include: DOM tree, component metadata, accessibility semantic attributes, and stable selector identifiers.

3. The task intent-based natural language-driven situational awareness control method according to claim 1, characterized in that, The object table is the main data of the interface target, including: name, alias, tag text, indicator name, unit of measurement, topic, and visibility conditions; The capability table is a list of executable actions, including: action name, target type, invocation method, parameter mode, preconditions, postconditions, linkage range, and influence domain. The parameter table is a parameter specification, including: type, value range, default value, verification rules, and data caliber identifier.

4. The task intent-based natural language-driven situational awareness control method according to claim 1, characterized in that, The task intent field table defines a set of fields, which includes: topic, goal, action, parameters, data freshness, role or permission, and interaction constraints.

5. The task intent-based natural language-driven situational awareness control method according to claim 4, characterized in that, The task intent in the structured intent data includes: topic, target, action, parameters, data freshness, role or permission, and interaction constraints; the location candidate set in the structured intent data includes: target object identifier, location confidence, and semantic location evidence vector. The method for generating the structured intent data includes: Obtain natural language instructions and perform layered parsing of the text of the natural language instructions: Based on the natural language template library and the task intent field table, candidate values ​​for the topics, objectives, actions, and parameters are first extracted from the natural language instructions. By combining the interface-data dual-mapping knowledge graph for semantic retrieval and reference resolution, user terms in natural language instructions are mapped to a controlled vocabulary and parameter enumeration, and interface objects consistent with the target semantics are retrieved from the object table to generate semantic localization evidence vectors.

6. The task intent-based natural language-driven situational awareness control method according to claim 5, characterized in that, The task intents in the structured intent data are compiled into operation micro-flows, and an operation instruction format list is generated; Including methods: The task intent is compiled into operation micro-flows based on the execution plan engine, and a list of operation instruction formats to be executed in sequence is generated. Each operation instruction format list must contain at least the following fields: target object identifier, action identifier, parameters, preconditions, post-validation, rollback strategy, timeout, and idempotent token; The execution plan engine selects the optimal execution channel based on the capability table. The selection rules include: prioritizing the assembly of API requests through the native interface adapter. When the target only supports UI interaction, automatically switch to the UI event adapter.

7. The task intent-based natural language-driven situational awareness control method according to claim 6, characterized in that, Execute the instructions step by step according to the list of operation instructions, and return a summary of the execution results; Including methods: The execution plan engine uses the trusted execution score of the scheduling plan as a list of gating execution operation instruction formats; Before each action begins, the execution plan engine checks the pre parameters and automatically fills in any missing parameters; the pre parameters include object visibility, required roles or permissions, and data freshness QoS. After the corresponding adapter is invoked to complete the action, the action result is validated in real time using the post parameter; The post parameters include: changes in the state of the interface object, the structure and quantity of the returned data, and matching of the prompt message; If the verification fails or times out, the fallback path will be initiated. After all actions are completed, return a summary of the execution results.

8. The natural language-driven situational awareness control method based on task intent according to claim 7, characterized in that, The method for obtaining the trusted execution score includes: Trusted execution scoring is achieved by nonlinearly fusing semantic matching, interface location, data availability, and permission compliance based on a trusted execution scoring gating mechanism.

9. A task intent-based natural language-driven situational awareness control system, characterized in that, Used to implement the task intent-based natural language-driven situational awareness control method according to any one of claims 1-8; The system includes: The preprocessing module is used to acquire the perception data and extract three tables of knowledge from the perception data, which include: an object table, a capability table, and a parameter table. The knowledge generation module is used to generate a natural language template library and a task intent field table based on the knowledge of the three tables; and to construct an interface-data dual mapping knowledge graph using the knowledge of the three tables as the foundation and the natural language template library and the task intent field table as retrieval and rearrangement tools. The retrieval and resolution module is used to respond to natural language commands by performing semantic retrieval and referential resolution based on a natural language template library, task intent fields, and an interface-data dual-mapping knowledge graph table to generate structured intent data; the structured intent data includes task intent and a location candidate set; The compilation module is used to compile the task intents in the structured intent data into operation micro-flows and generate a list of operation instruction formats; The execution module executes the instructions step by step according to the list of operation instructions and returns a summary of the execution results. The scoring module is used to perform a reliable execution score on the summary of the execution results.

10. A computer-readable medium having a computer program stored thereon, characterized in that, The computer program, when executed by a processor, can implement the task intent-based natural language-driven situational awareness control method as described in any one of claims 1-8.

Images