Intelligent application interaction method based on natural language processing

By constructing incremental text streams and dynamic semantic graphs in intelligent interaction systems, and combining bidirectional referential resolution and cross-attention alignment, the problems of data bloat and insufficient robustness caused by full-volume interface text are solved, achieving efficient and accurate cross-application interaction and self-verification.

CN123068577APending Publication Date: 2026-11-10HOHHOT LANGYING TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611457181.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-18
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

In intelligent interaction scenarios with frequent switching between multiple applications, existing technologies rely on full-text interfaces or full-screen screenshots, which leads to a sharp increase in the amount of text input to the model. This can easily cause context window truncation and lacks self-verification capabilities, resulting in insufficient system robustness.

Method used

By continuously monitoring interface changes through the operating system's accessibility service interface, an incremental text stream is constructed and a dynamic semantic graph indexed by intent, time, and space is created. Combined with bidirectional referential resolution and cross-attention alignment processing, parallel verification interaction actions are generated and anomalies are fed back, forming a closed-loop verification mechanism.

Benefits of technology

It significantly reduced the amount of input data to the model, improved object binding accuracy, avoided repeated and invalid operations, and enhanced system robustness and execution success rate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123068577A_ABST
    Figure CN123068577A_ABST
Patent Text Reader

Abstract

The application discloses an intelligent application interaction method based on natural language processing and relates to the technical field of natural language processing.The method obtains an incremental text stream of interface changes through an operating system barrier-free service interface, constructs a dynamic semantic graph in accordance with a three-dimensional indexing mode of intention, time and space, determines interface objects corresponding to time and space reference words in a user instruction in combination with bidirectional reference resolution and cross attention alignment, and generates an expected result semantic description text in parallel with an interaction action text generated by a large language model, and performs closed loop verification through semantic similarity comparison, generates a semantic anomaly abstract and triggers re-planning when an anomaly is detected, and the application reduces the amount of model input data and computing overhead, avoids time and space reference failure in cross-application interaction, and improves the self-checking and anomaly recovery capabilities of the system, and is mainly applied to natural language driven interface interaction scenes of intelligent terminals.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to an intelligent application interaction method based on natural language processing. Background Technology

[0002] Large language models have strong natural language understanding and generation capabilities. The industry combines them with graphical user interface automation technology to build intelligent agents that can transform natural language commands into interface clicks, inputs, swipes and other operations. In order for the model to recognize interface elements, it usually obtains the structured text tree of interface controls through the accessibility service interface provided by the operating system or document object model parsing tools, or identifies and locates interface elements by combining screenshots with multimodal visual models, and then generates the next interactive action to be executed.

[0003] The closest existing technology to this invention is an open-loop graphical user interface intelligent agent interaction method based on a multimodal large model. In each interaction step, the method takes a full-screen screenshot of the currently active window or exports the full control tree text, encodes it together with the historical interaction instructions, and inputs it into a large language model. The large language model generates interaction action instructions and delivers them to the execution module for execution. After the execution is completed, the above process is repeated after a preset fixed time until the task is completed.

[0004] The aforementioned existing technologies have the following shortcomings: On the one hand, as the number of cross-application interaction steps increases, the amount of text input to the model increases dramatically due to the full amount of interface text input, which can easily cause the model's context window to be truncated, making it impossible for the model to accurately associate the ambiguous spatiotemporal pronouns in the user's instructions with the specific interface objects generated across applications. On the other hand, the method adopts a one-way open-loop paradigm, and does not perform result status verification after the action is executed. Once an interface abnormality occurs, the model is prone to continue to deduce subsequent steps under incorrect assumptions, resulting in repeated invalid operations and insufficient system robustness. To address these issues, we propose an intelligent application interaction method based on natural language processing. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent application interaction method based on natural language processing.

[0006] The technical problems to be solved by this invention include the following two aspects: First, in intelligent interaction scenarios with high-frequency switching between multiple applications, existing methods rely on full-text interfaces or full-screen screenshots as model input. As the number of interaction steps increases, the amount of input text expands rapidly, which can easily cause the model context window to be truncated. This results in the model being unable to accurately associate ambiguous spatiotemporal pronouns in the user's statement with specific interface objects generated across applications, leading to spatiotemporal reference failure. At the same time, this method consumes a lot of computing power and is difficult to deploy in real time on low-computing-power devices. Second, existing natural language-driven graphical user interface interaction methods generally adopt an open-loop paradigm of perception, decision-making, and execution. After generating action instructions, the model cannot perceive whether the action has taken effect in the real interface. Once encountering abnormal situations such as network latency, interface lag, or pop-up window obstruction, the model is prone to continue to extrapolate subsequent steps under the incorrect assumption that the action has been successfully executed, eventually falling into a loop of repeated clicks or long waiting times. It lacks self-verification and abnormal recovery capabilities, resulting in insufficient system robustness.

[0007] To address the aforementioned technical problems, this invention provides an intelligent application interaction method based on natural language processing, comprising the following technical solutions:

[0008] The system continuously monitors interface change events of the currently active window on the terminal device through the accessibility service interface provided by the operating system. When an interface change is detected, it extracts the set of user interface nodes that have changed compared to the previous moment, forming an incremental text stream. This incremental text stream is then written into a dynamic semantic graph using a three-dimensional indexing method of intent index, time index, and spatial index. The system receives user natural language commands, extracts spatiotemporal pronouns from these commands, and combines them with candidate user interface nodes in the dynamic semantic graph. Through bidirectional referencing resolution and cross-attention alignment, it determines the specific interface object corresponding to the spatiotemporal pronoun, completing object binding. Based on the object binding result, historical interaction commands, and the current information in the dynamic semantic graph, it generates the interaction action text for the current interaction step in parallel. This involves taking the expected result semantic description text of the interface state after the interactive action text is executed, performing the operation corresponding to the interactive action text, re-collecting the incremental text stream of the current interface after execution, calculating the semantic similarity between the re-collected incremental text stream and the expected result semantic description text, determining whether the semantic similarity reaches a preset semantic deviation threshold, determining that the action execution result is consistent with the expected result when the semantic deviation threshold is reached, and returning to the process of continuously listening for interface change events, and determining that an anomaly has occurred when the semantic deviation threshold is not reached, generating a semantic anomaly summary, feeding the semantic anomaly summary back to the parallel generation of interactive action text and expected result semantic description text, and regenerating the interactive action text and expected result semantic description text.

[0009] Compared with the prior art, the beneficial effects of the present invention by adopting the above technical solution are as follows:

[0010] 1. This invention replaces the full interface text or full-screen screenshot as model input with incremental structured text stream, which significantly reduces the amount of data input to the model and the consumption of computing resources, enabling this method to be deployed on low-computing-power devices on the edge.

[0011] 2. This invention uses a dynamic semantic graph that is continuously updated according to the three-dimensional index of intent, time and space, and combines bidirectional referential resolution and cross-attention alignment processing. It can accurately associate historical interface objects generated across applications with only incremental text, effectively avoiding the binding error of ambiguous spatiotemporal referentials caused by context window truncation, and improving the object binding accuracy.

[0012] 3. This invention enables the method to have self-verification capabilities during execution by generating interactive action text and expected result semantic description text in parallel, and by combining semantic similarity comparison and anomaly feedback replanning mechanism after execution. This allows the method to detect abnormal situations such as network latency, interface lag, and pop-up window obstruction in a timely manner and trigger replanning, avoiding getting stuck in a loop of repeated invalid operations, and improving the overall success rate and system robustness of complex cross-application tasks. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the overall main interaction process in an embodiment of the present invention;

[0014] Figure 2 This is a schematic diagram of the dynamic semantic graph construction process in an embodiment of the present invention;

[0015] Figure 3 This is a schematic diagram of the bidirectional semantic feedback closed-loop verification process in an embodiment of the present invention. Detailed Implementation

[0016] The following is in conjunction with the appendix Figure 1-3 The specific embodiments of the present invention will be further described below. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0017] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0018] I. System Overall Architecture:

[0019] The intelligent application interaction method provided in this embodiment runs on an overall system architecture that includes an operating system accessibility service interface, a dynamic semantic graph construction module, a bidirectional referential resolution module, a large language model interaction decision module, an action execution module, an expected state generation module, a semantic feedback verification module, and an abnormal interruption replanning module. The modules are interconnected by data flow and control flow to form a closed loop.

[0020] The operating system accessibility service interface is used to read the control tree of the currently active window of the terminal device without relying on the application source code. This includes structured information such as control type, text content, coordinate position, and hierarchical relationship. The dynamic semantic graph construction module receives the above structured information incrementally and continuously maintains a cross-application and cross-time semantic relationship network according to the three-dimensional index of intent, time, and space. The bidirectional referencing resolution module receives the user's natural language instructions and candidate nodes in the dynamic semantic graph and outputs the object binding result. The large language model interaction decision module receives the object binding result and historical interaction instructions and outputs two signals in parallel: the interaction action text and the expected result semantic description text. The action execution module receives the interaction action text and applies it to the terminal device. The semantic feedback verification module receives the incremental text and the expected result semantic description text that are re-collected after execution and outputs a consistent or abnormal judgment signal. The abnormal interruption replanning module is triggered when the verification result is abnormal. Its output is fed back to the large language model interaction decision module, forming a closed loop connection.

[0021] II. Overall Method Flow:

[0022] The intelligent application interaction method provided in this embodiment generally includes the following processing steps:

[0023] The terminal device continuously monitors interface change events of the currently active window through the operating system's accessibility service interface. When an interface change is detected, including in-application refresh or cross-application switching, it extracts the set of user interface nodes that have changed compared to the previous moment, forming an incremental text stream. This stream is then written into a dynamic semantic graph using a three-dimensional indexing method based on intent, time, and space. The device receives user natural language commands, extracts spatiotemporal pronouns from the commands, and combines them with candidate user interface nodes in the dynamic semantic graph. Through bidirectional referencing resolution and cross-attention alignment algorithms, it determines the specific interface object corresponding to the pronoun, completing object binding. Based on the object binding results, historical interaction commands, and current semantic graph information, the large language model interaction decision module generates in parallel the text of the interaction action to be performed in the current interaction step and the execution of that action. The expected result semantic description text is then processed by the action execution module, which parses and executes the specific operations corresponding to the interactive action text, including click, input, and swipe operations. After the action is executed, the incremental text stream of the current interface is re-collected, and the semantic similarity between it and the expected result semantic description text is calculated by the semantic feedback verification module. It is then determined whether the semantic similarity reaches the preset semantic deviation threshold. If the threshold is reached, the action execution result is determined to be consistent with the expectation, and the process returns to continuously listen for interface change events to continue processing subsequent tasks. If the threshold is not reached, an anomaly is determined to have occurred, a semantic anomaly summary is generated, the interruption protection mechanism is triggered, and the anomaly information is fed back to the large language model interaction decision module to replan the subsequent interaction path and return to the parallel generation of interactive action text and expected result semantic description text.

[0024] III. Cross-application dynamic semantic graph construction and incremental update:

[0025] The dynamic semantic graph in this embodiment is organized according to a three-dimensional index of intent, time and space: the intent dimension records the current task objective and the historical instruction sequence, the time dimension records the order of appearance of each user interface node in the form of a timestamp linked list, and the space dimension records the coordinate position and hierarchical relationship of the user interface node in its window. Each node in the graph contains attribute fields such as node text, control type, coordinates, application identifier, timestamp and parent-child hierarchy pointer.

[0026] The specific implementation process of cross-application dynamic semantic graph construction and incremental update is as follows: Listen for window focus change events and interface refresh events on the terminal device. When an application switch or interface content change is detected, this process is triggered. The operating system's accessibility service interface is called to obtain the complete user interface control tree of the currently active window, which is recorded as the current user interface control tree. The current user interface control tree is compared node-by-node with the user interface control tree recorded in the semantic graph at the previous moment, identifying newly added nodes, nodes whose content has been modified, and nodes that have disappeared. These three types of nodes together constitute the incremental node set. For each node in the incremental node set, its text content and control type are extracted. The system generates a structured incremental text stream by analyzing the type, screen coordinates, application identifier, and parent-child hierarchy. This incremental text stream is then written into the semantic graph database using a three-dimensional indexing method based on intent, time, and space. The current task intent is used as the first-level index, the timestamp generated by the node is used as the second-level index, and the application and coordinate hierarchy of the node are used as the third-level index. The relationships between these three are established, and the timestamp linked list in the semantic graph is updated. The incremental node is then inserted into the linked list in chronological order for subsequent reference resolution phases to retrieve candidate nodes by time window. The system determines whether the task is finished. If it is not finished, it returns to the process of listening for the next interface change. If the task is finished, the current semantic graph is cleared or archived.

[0027] Through the incremental comparison and three-dimensional index writing method described above, this embodiment only needs to process the interface text that has changed compared to the previous moment, instead of re-collecting and inputting the full amount of interface text in each interaction step. This significantly reduces the amount of data input to the model while preserving cross-application historical object information.

[0028] IV. Two-way referential resolution and cross-attention alignment:

[0029] The specific implementation process of bidirectional referential resolution and cross-attention alignment is as follows: The natural language command input by the user is segmented and labeled with part of speech to identify the spatiotemporal referents and their contextual words. The spatiotemporal referents include fuzzy words used to refer to the preceding operation object or interface element. According to the timestamp list in the semantic graph, the set of candidate user interface nodes generated in the most recent preset time window is selected as the set of candidate objects to be aligned. The lightweight bidirectional encoder is used to perform vector encoding on the contextual text of the spatiotemporal referents and the text of each node in the set of candidate user interface nodes to obtain the context vector of the referent and the semantic vector of each candidate node.

[0030] Based on the encoded vectors, the cross-attention alignment score between the pronoun context vector and the semantic vector of each candidate node is calculated as follows:

[0031] ;

[0032] ;

[0033] In the above formula, Indicates the first The contextual vector of a spatiotemporal pronoun. Indicates the first The semantic vectors of the candidate user interface nodes. This represents a learnable query projection matrix used for linear transformations of context vectors. This represents the learnable key projection matrix, used for linear transformations of semantic vectors. This represents the vector dimension, used to scale the dot product result to avoid making the value too large. Indicates the first The individual pronoun and the first The original alignment score between candidate nodes This represents the total number of nodes in the current candidate user interface node set. This indicates the number of digits after normalization. The individual pronoun and the first The final alignment score between candidate nodes ranges from 0 to 1, and for a fixed number of nodes... ,all corresponding The sum is 1;

[0034] For each spatiotemporal pronoun, the alignment score is calculated based on its corresponding candidate node. In the process, the candidate user interface node whose value exceeds the alignment score of all other candidate nodes is selected as the object binding result of the pronoun. The object binding result is output together with the remaining semantic information in the original natural language instruction for the large language model interaction decision module to use for subsequent action text generation.

[0035] Taking the user's instruction to send the previous text to the group as an example, both the previous text and the group are spatiotemporal pronouns. The system filters out candidate user interface nodes that have appeared in the most recent preset time window based on the timestamp linked list. These include text nodes that were previously copied in the browser application and chat group nodes that were previously opened. Through the cross-attention alignment process mentioned above, the alignment score between the previous text and the candidate text node, and the alignment score between the group and the candidate chat group node are calculated respectively. This determines the specific interface object corresponding to each pronoun and completes the cross-application object binding.

[0036] V. Two-way semantic feedback closed-loop verification mechanism:

[0037] The specific implementation process of the two-way semantic feedback closed-loop verification mechanism is as follows: While generating the interactive action text of the current interaction step, the large language model interaction decision module generates the expected result semantic description text of the interface state after the action is executed, based on the current task intent and interface state. The action execution module parses and executes the specific operation corresponding to the interactive action text. After the action is executed, after a preset short delay, the incremental user interface text of the currently active window is re-collected through the accessibility service interface of the operating system to form the current actual interface state description text.

[0038] Using a pre-trained semantic embedding model, the current actual interface state description text and the expected result semantic description text are vectorized to obtain corresponding semantic vectors. Then, the cosine similarity between the two semantic vectors is calculated as follows:

[0039] ;

[0040] In the above formula, This represents the semantic vector obtained after the current actual interface state description text is encoded by a semantic embedding model. This represents the semantic vector obtained after the semantic description text of the expected result is encoded by a semantic embedding model. Represents the dot product of two vectors. and Let represent the Euclidean norms of the two vectors, respectively. This represents the calculated cosine similarity value, which ranges from -1 to 1. The larger the value, the closer the two texts are semantically.

[0041] The above similarity values ​​are compared with a preset semantic deviation threshold. Comparison: In Not less than In this case, the judgment flag is set to 0, indicating that the actual interface state is consistent with the expected result, and this step was executed normally. Less than In this case, the judgment flag is set to 1, indicating that the actual interface state deviates from the expected result and the action in this step is abnormal.

[0042] When the judgment flag is set to 0, the system proceeds to the next step of interaction task planning and returns to the continuous monitoring of interface change events. When the judgment flag is set to 1, the system automatically generates a semantic anomaly summary. The summary includes the main differences between the current actual interface state description and the expected result semantic description, which explains the reasons for the interface deviating from expectations, such as pop-up obstruction, loading failure, or network anomaly. It also triggers the interruption protection mechanism, pausing the continued execution of the current action sequence. Subsequently, the generated semantic anomaly summary is fed back to the large language model interaction decision module. Based on the anomaly summary and the current semantic graph information, the model replans the subsequent interaction path, regenerates the interaction action text and the expected result semantic description text, and returns to the continuous monitoring of interface change events, thus forming a complete two-way semantic feedback loop.

[0043] VI. Alternative Implementation Methods:

[0044] In terminal environments where accessibility service permissions cannot be obtained from the operating system, a lightweight approach combining partial screenshots and optical character recognition can be used to replace the accessibility service interface. Specifically, only the local area of ​​the interface that has changed is screenshotted and processed by optical character recognition to extract the text content and coordinate information of the local area. This allows the construction of an incremental text stream with the same structure as the aforementioned embodiment, thereby completing the construction of a dynamic semantic graph and subsequent referencing resolution and closed-loop verification. This alternative implementation method, while sacrificing some text extraction accuracy, is applicable to application scenarios with limited accessibility permissions.

[0045] In the closed-loop verification stage, instead of deploying a lightweight semantic embedding model separately for cosine similarity calculation, the large language model can directly use natural language judgment to input the current actual interface state description text and the expected result semantic description text as prompt content into the model. The model will directly output the judgment result of semantic consistency or semantic inconsistency and the corresponding reason, which replaces the aforementioned processing process based on vectorized similarity calculation. This alternative implementation method is suitable for deployment scenarios with sufficient cloud computing power and the ability to withstand high inference latency.

[0046] VII. Technical Effects Description:

[0047] Using the method provided in this embodiment, since each interaction step only needs to process the incremental interface text that has changed compared to the previous moment, without needing to re-collect and input the full interface text or full-screen screenshot, the amount of text input by the model will not continuously accumulate and expand with the increase of interaction steps. This avoids the loss of historical cross-application object information due to context window truncation. Combined with the historical object records organized by the three-dimensional index of intent, time and space in the dynamic semantic graph, as well as bidirectional referential resolution and cross-attention alignment processing, the system can accurately bind the ambiguous spatiotemporal pronouns in the user's instructions to the actual interface objects they refer to in cross-application and cross-time window scenarios. At the same time, since the large language model generates the expected result semantic description text in parallel while generating the interactive action text, and performs result verification by semantic similarity comparison after execution, the system can promptly identify deviations when abnormal situations such as network latency, interface lag or pop-up occlusion occur, generate semantic abnormality summaries and trigger replanning, thereby avoiding the cycle of repeated invalid operations due to lack of result feedback, and improving the overall execution success rate and system robustness of complex cross-application interaction tasks.

[0048] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for intelligent application interaction based on natural language processing, characterized in that, include: The system continuously monitors interface change events of the currently active window of the terminal device through the accessibility service interface provided by the operating system. When an interface change is detected, the system extracts the set of user interface nodes that have changed compared to the previous moment, forms an incremental text stream, and writes the incremental text stream into a dynamic semantic graph according to the three-dimensional indexing method of intent index, time index and spatial index. Receive user natural language instructions, extract spatiotemporal pronouns from the user natural language instructions, combine them with candidate user interface nodes in the dynamic semantic graph, and determine the specific interface object corresponding to the spatiotemporal pronouns through bidirectional pronoun resolution and cross-attention alignment processing, and complete object binding; Based on the object binding result, historical interaction instructions and the information of the current dynamic semantic graph, the interaction action text of the current interaction step and the expected result semantic description text of the interface state after the interaction action text is executed are generated in parallel. Execute the operation corresponding to the interactive action text, and after the execution is completed, re-collect the incremental text stream of the current interface, and calculate the semantic similarity between the current actual interface state description text corresponding to the re-collected incremental text stream and the expected result semantic description text. Determine whether the semantic similarity reaches a preset semantic similarity threshold. If the semantic similarity threshold is reached, determine that the action execution result is consistent with the expected result, and return to the process of continuously monitoring the interface change event of the currently active window of the terminal device. If the semantic similarity threshold is not reached, determine that an anomaly has occurred, generate a semantic anomaly summary, and feed the semantic anomaly summary back to the parallel generation of interactive action text and expected result semantic description text, and regenerate the interactive action text and expected result semantic description text.

2. The intelligent application interaction method based on natural language processing according to claim 1, characterized in that, The extraction of the set of user interface nodes that have changed compared to the previous moment, forming an incremental text stream, and writing the incremental text stream into a dynamic semantic graph according to a three-dimensional indexing method of intent index, time index, and spatial index, includes: The user interface control tree of the currently active window is compared with the user interface control tree of the previous moment recorded in the dynamic semantic graph. The text and coordinates of each node are compared to identify newly added nodes, nodes whose content has been modified, and nodes that have disappeared. The three types of nodes constitute the incremental node set. For each node in the incremental node set, extract the node's text content, control type, screen coordinates, application identifier, and parent-child hierarchy to form a structured incremental text stream; The current task intent is used as the first-level index, the timestamp generated by the node is used as the second-level index, and the application identifier and coordinate level to which the node belongs are used as the third-level index. The association between the first-level index, the second-level index and the third-level index is established, and the association is written into the dynamic semantic graph.

3. The intelligent application interaction method based on natural language processing according to claim 2, characterized in that, Also includes: Update the timestamp linked list in the dynamic semantic graph, and insert the current incremental node set into the timestamp linked list in chronological order; The user interface nodes generated within a preset time window are selected from the timestamp linked list as candidate user interface nodes.

4. The intelligent application interaction method based on natural language processing according to claim 1, characterized in that, The process of determining the specific interface object corresponding to the spatiotemporal pronoun through bidirectional referential resolution and cross-attention alignment includes: A bidirectional encoder is used to perform vector encoding on the contextual text of the spatiotemporal pronoun and the text of the candidate user interface node, respectively, to obtain the contextual vector of the pronoun and the semantic vector of the candidate node. The query projection matrix is ​​used to perform a linear transformation on the pronoun context vector, and the key projection matrix is ​​used to perform a linear transformation on the candidate node semantic vector. The dot product between the two vectors after the linear transformation is calculated, and the dot product is scaled according to the square root of the vector dimension to obtain the original alignment score. The original alignment scores of all candidate user interface nodes corresponding to the same spatiotemporal pronoun are normalized to obtain the alignment score. The candidate user interface node whose alignment score exceeds that of all other candidate user interface nodes is selected as the object binding result for that spatiotemporal pronoun.

5. The intelligent application interaction method based on natural language processing according to claim 1, characterized in that, The calculation of the semantic similarity between the current actual interface state description text corresponding to the re-acquired incremental text stream and the expected result semantic description text includes: The current actual interface state description text and the expected result semantic description text are vector-encoded using a pre-trained semantic embedding model to obtain the current semantic vector and the expected semantic vector. Calculate the cosine similarity between the current semantic vector and the expected semantic vector, and use the cosine similarity as the semantic similarity.

6. The intelligent application interaction method based on natural language processing according to claim 1, characterized in that, The semantic anomaly summary includes a description of the differences between the current actual interface state description text and the expected result semantic description text, and the description of the differences is used to characterize the cause of the interface anomaly.

7. The intelligent application interaction method based on natural language processing according to claim 1, characterized in that, When the terminal device does not obtain accessibility service permissions from the operating system, a screenshot is taken of the local area in the currently active window that has changed, and optical character recognition processing is performed on the screenshot image to obtain the text content and coordinate information of the local area. The incremental text stream is formed by the text content and coordinate information.

8. The intelligent application interaction method based on natural language processing according to claim 1, characterized in that, The current actual interface state description text and the expected result semantic description text are input into the large language model to obtain the semantic consistency judgment result output by the large language model. The semantic consistency judgment result is used to replace the semantic similarity to determine whether the action execution result is consistent with the expected result.

9. An intelligent application interaction device based on natural language processing, characterized in that, include: The incremental graph construction module is used to continuously monitor the interface change events of the currently active window of the terminal device through the accessibility service interface provided by the operating system. When the interface change is detected, the set of user interface nodes that have changed compared to the previous moment is extracted to form an incremental text stream, and the incremental text stream is written into the dynamic semantic graph according to the three-dimensional indexing method of intent index, time index and spatial index. The pronoun resolution module is used to receive user natural language commands, extract spatiotemporal pronouns from the user natural language commands, combine them with candidate user interface nodes in the dynamic semantic graph, and determine the specific interface object corresponding to the spatiotemporal pronouns through bidirectional pronoun resolution and cross-attention alignment processing, and complete object binding. The interaction decision module is used to generate, in parallel, the interaction action text of the current interaction step and the expected result semantic description text of the interface state after the interaction action text is executed, based on the object binding result, historical interaction instructions and the information of the current dynamic semantic graph. The action execution module is used to execute the operation corresponding to the interactive action text; The semantic feedback verification module is used to re-collect the incremental text stream of the current interface after the action execution module has finished executing, calculate the semantic similarity between the current actual interface state description text corresponding to the re-collected incremental text stream and the expected result semantic description text, and determine whether the semantic similarity reaches the preset semantic deviation threshold. The abnormal interruption replanning module is used to determine that an abnormality has occurred when the semantic similarity does not reach the semantic deviation threshold, generate a semantic abnormality summary, and feed the semantic abnormality summary back to the interaction decision module, triggering the interaction decision module to regenerate the interaction action text and the expected result semantic description text. The semantic feedback verification module is also used to determine that the action execution result is consistent with the expected result when the semantic similarity reaches the semantic deviation threshold, and to trigger the incremental graph construction module to continue to monitor interface change events.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the intelligent application interaction method based on natural language processing as described in any one of claims 1 to 8.