A task response execution method, a computer device and a storage medium

Abstract

The application discloses a task response execution method, a computer device and a storage medium, and relates to the technical field of task response execution. The method comprises the following steps: in response to a user initiating a voice interaction request in a target interactive environment, obtaining voice interaction information, and generating environment interface time sequence information according to interface state information in a request window; analyzing the voice interaction information to obtain a target interactive operation and target description information, combining the environment interface time sequence information to determine a target interface and a candidate interface object, and generating a candidate task; adding the candidate task to a task response execution sequence, if there is an associated task that is prior and has not been completed, locking subsequent response execution, determining a task replacement relationship and updating the task response execution sequence according to the task replacement relationship; and determining and executing a target task in an executable state. The application can improve the matching accuracy of voice requests and interface objects in a dynamic interactive environment, and reduce response conflicts or execution disorder of corresponding tasks of continuous voice requests.

Description

Technical Field

[0001] This application relates to the field of task response execution technology, and in particular to a task response execution method, computer device and storage medium. Background Technology

[0002] With the development of speech recognition, natural language processing, and smart terminal technologies, voice interaction has been widely applied in scenarios such as smart reading, online courses, and multimedia playback. Users can use voice commands to select, open, and play books, courses, or media resources displayed on the current interface.

[0003] Existing voice interaction systems typically determine the target of an operation based on speech recognition results and the interface state during the voice request processing phase. However, speech acquisition, speech recognition, semantic parsing, and task execution all involve a certain processing delay. During this period, the application interface may change due to user actions, content updates, or page navigation, resulting in inconsistencies between the interface state at the time the voice request is initiated and the state during voice request processing. For voice requests containing indicative or relative positional semantics, the system is prone to incorrectly determining the target of the operation based on the changed interface, thereby reducing the accuracy of voice interaction operations.

[0004] Furthermore, during continuous voice interaction, multiple voice requests may be generated successively before the previous task has been completed. There may be succession, substitution, or other semantic relationships between the operation objects and content corresponding to different tasks. Existing technologies typically execute tasks sequentially according to their generation order, or directly overwrite earlier tasks with newly generated ones. This makes it difficult to coordinate subsequent execution processes based on the relationship between tasks, easily leading to problems such as unreasonable execution order of related operations, continued execution of invalid operations, erroneous cancellation of valid operations, or conflicts between multiple operations. This, in turn, affects the consistency between the task execution result and the user's interaction intent. Summary of the Invention

[0005] Based on this, it is necessary to provide a task response execution method, computer device, and storage medium that can improve the matching accuracy between voice requests and interface objects in a dynamic interactive environment and coordinate the response execution of tasks corresponding to continuous voice requests, so as to at least solve the technical problems in the related technologies where voice requests and interface objects are prone to mismatch, and where response conflicts or execution disorder are prone to occur between multiple asynchronous tasks.

[0006] On the one hand, a task response execution method is provided, the method comprising: In response to a target user initiating a voice interaction request in the target interaction environment, the corresponding voice interaction information is obtained, and environmental interface timing information is generated based on the interface state information of the target interaction environment in the corresponding request window. The voice interaction information is parsed to obtain the target interaction operation and target description information. Based on the target description information and the environmental interface timing information, the target interface and the corresponding candidate interface object that match the voice interaction request in the target interaction environment are determined. Candidate tasks are generated based on the target interaction operation and the candidate interface object. The candidate task is added to the task response execution sequence corresponding to the target interaction environment, and it is determined whether there is an associated task in the task response execution sequence that was generated earlier than the candidate task and has not yet completed the response execution. If they exist, the execution of the subsequent response of the associated task is locked, and the task iteration relationship between the candidate task and the associated task is determined according to the candidate interface objects and interactive operations corresponding to the candidate task and the associated task, respectively. Based on the task iteration relationship, adjust the candidate tasks and / or the associated tasks to update the task response execution sequence; Based on the updated task response execution sequence, the target task in the executable state is determined and executed.

[0007] On the other hand, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described task response execution method.

[0008] In another aspect, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described task response execution method.

[0009] The aforementioned task response execution method, apparatus, computer device, and storage medium associate voice interaction requests with changes in the interface state within the corresponding request window. This allows the interaction intent represented in the voice information to be parsed in conjunction with the interface context during the voice request's occurrence, thereby reducing mismatches of interaction objects caused by interface refreshes, switching, or object position changes during voice request processing and improving the matching accuracy between voice requests and interface objects in dynamic interactive environments. Furthermore, the parsed interaction operations and their corresponding interface objects are organized into schedulable tasks. After a new task is added to the execution process, incomplete associated tasks are identified. The iteration relationship between old and new tasks is determined while restricting the continued execution of associated tasks. Based on this, the relevant tasks and their execution order are adjusted, enabling subsequently generated interaction intents to participate in the coordinated processing of previously incomplete responses. This avoids continuing to execute responses that are no longer compatible with the current interaction intent before the task relationship is determined, and reduces the probability of repetition, conflict, or execution disorder between interaction operations corresponding to consecutive voice requests. Therefore, it can balance the accuracy of determining the interactive object under dynamic interface conditions and the consistency of response execution between continuous asynchronous tasks, so that the actual interactive response matches the user's interaction intention, thereby improving the accuracy and reliability of task response execution in voice interaction scenarios. Attached Figure Description

[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is an application environment diagram of the task response execution method in one embodiment; Figure 2 This is a flowchart illustrating a task response execution method in one embodiment; Figure 3 This is a structural block diagram of a task response execution device in one embodiment; Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0013] This application provides a task response execution method that can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A target user can initiate a voice interaction request within the target interactive environment provided by terminal 102. Terminal 102 collects the corresponding voice interaction information and the interface state information of the target interactive environment within the request window, and sends the collected information to server 104. Server 104 generates environmental interface timing information based on the interface state information, and obtains the target interaction operation and target description information by parsing the voice interaction information. Based on the target description information and environmental interface timing information, it determines the target interface matching the voice interaction request and the corresponding candidate interface object, and then generates candidate tasks based on the target interaction operation and candidate interface object. Server 104 adds the candidate tasks to the task response execution sequence corresponding to the target interactive environment, and determines whether there are any related tasks whose generation time is earlier than the candidate tasks and whose response execution has not yet been completed. If so, it locks the subsequent response execution of the related tasks, determines the task iteration relationship between the candidate tasks and related tasks based on the candidate interface objects and interaction operations corresponding to them respectively, and adjusts the candidate tasks and / or related tasks based on the task iteration relationship to update the task response execution sequence. If not, it can maintain the task order of the candidate tasks in the task response execution sequence. Server 104 further determines the target task in an executable state based on the updated task response execution sequence and sends the corresponding task execution instruction to terminal 102, so that terminal 102 performs corresponding interactive operations on the target interface object in the target interface; terminal 102 can also return the execution result of the interactive operation to server 104, so that server 104 can update the response execution status of the corresponding task. The aforementioned information parsing, task generation, task scheduling, and task execution processes can be performed by either terminal 102 or server 104, or by both, depending on actual deployment needs. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, and server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0014] In one specific embodiment, the target interactive environment can be a smart reading application running on terminal 102. The smart reading application can display at least one interface of book list, book details, course list, audio playback, and reading content to the target user, and allow the target user to request interactive operations such as opening, playing, pausing, selecting, displaying, or switching the books, courses, audio, or other interactive objects in the corresponding interface through voice interaction.

[0015] For example, when a target user browses books in the book list interface of a smart reading application, the current interface displays multiple books in a preset order, with each book corresponding to an interactive interface object. When the target user initiates a voice interaction request to "open the second book" while the book list interface is displayed, the terminal 102 collects the voice data input by the target user and records the start and end times of the voice interaction request. Simultaneously, the terminal 102 collects the interface state information of the smart reading application within the corresponding request window. The interface state information may include at least one of the following: the interface identifier of the book list interface, the interface object identifier corresponding to each book, the resource type identifier, the object sequence index, the object location area, the display status, and the time of the corresponding status change.

[0016] For example, when the target user says "Open the second book," the user can also perform a swipe operation on the book list, moving the book that was originally in the second position out of the current display area and moving other books into the current display area. Terminal 102 sends the voice interaction information and the corresponding interface state information to server 104. Server 104 generates environmental interface timing information based on the time of state changes and the duration of the states of each interface and interface object in the request window, thereby preserving the interface state of the book list interface before and after the swipe during the time the target user initiates the voice interaction request, rather than only using the interface state currently displayed by terminal 102 when server 104 processes the voice interaction request.

[0017] Server 104 parses the voice interaction information to obtain the target interaction operation of "open," and retrieves target description information from "the second book" to define the book resource type and object arrangement order. Server 104 matches the target description information with the environmental interface timing information to determine the target interface corresponding to the user's description. Within the target interface, it identifies candidate interface objects matching "the second book" based on resource type, object order, and the duration of the corresponding interface object's state. Therefore, even if the intelligent reading application displays other books due to user swiping during voice recognition or subsequent processing, Server 104 can determine the book actually referred to by the user based on the interface state corresponding to the stage of the voice interaction request, instead of directly using the second book currently displayed on the interface as the target object.

[0018] Server 104 further generates candidate tasks based on the target interactive operation corresponding to "open" and the determined candidate interface objects, and adds the candidate tasks to the task response execution sequence corresponding to the intelligent reading application. When there are no related tasks in the task response execution sequence that were generated earlier than the candidate tasks and have not yet completed their response execution, the task order of the candidate tasks in the task response execution sequence can be maintained, and the status of the candidate tasks can be determined according to the corresponding scheduling rules.

[0019] In another scenario, when the aforementioned candidate tasks are added to the task response execution sequence, the sequence may also contain related tasks generated by a prior voice interaction request from the target user that have not yet been executed. For example, the target user first initiates a voice interaction request to "introduce this book" for a target book in the book list. Before the corresponding introduction task has been completed, the user then initiates a voice interaction request to "don't introduce it, just open this book." Server 104 can add the candidate task generated by the latter voice interaction request to the task response execution sequence and determine that both tasks apply to the same book based on the candidate interface objects corresponding to the two tasks. Based on this, it locks the subsequent execution of the related tasks that have not yet been completed and compares the interaction operations corresponding to the related tasks and the candidate tasks to determine the task iteration relationship between them.

[0020] For example, when an incomplete interactive operation in a related task is used to continue displaying the introduction of the target book, while the target interactive operation of the candidate task is used to directly open the same target book, the task sequence of the incomplete interactive operation in the related task and the candidate task can be adjusted according to the cancellation or modification intention expressed by the target user in the next voice interaction request. This will prevent subsequent responses that are no longer compatible with the current voice interaction intention from being executed, and allow the interactive operation corresponding to opening the target book to enter the updated task response execution sequence.

[0021] Server 104 determines the target task in an executable state based on the updated task response execution sequence. Before executing the target task, it can verify the execution status of the target interface, candidate interface objects, and pre-interaction operations corresponding to the target task. When the corresponding execution conditions are met, server 104 sends task execution information corresponding to the target task to terminal 102, so that the smart reading application can perform interactive operations such as opening the corresponding candidate interface object. Terminal 102 can also return the execution results of the corresponding interactive operations to server 104 so that server 104 can update the response execution status of the target task.

[0022] Therefore, in intelligent reading applications, even if the target user continues to swipe, refresh, or switch interfaces during voice input, voice recognition, or task response execution, the interface object actually referred to by the target user can be determined by utilizing the environmental interface timing information corresponding to the stage in which the voice interaction request occurs. Furthermore, when the target user continuously initiates multiple voice interaction requests with modification, cancellation, or succession relationships, the corresponding task response execution sequence can be adjusted by identifying related tasks and task iteration relationships, thereby reducing the misbinding of book objects caused by changes in interface state, as well as the repeated responses, conflicting responses, or execution disorder caused by the continuous execution of multiple asynchronous voice tasks.

[0023] In one embodiment, such as Figure 2 As shown, a task response execution method is provided, which is applied to... Figure 1 Taking server 104 as an example, the following steps are included: Step 201: In response to the target user initiating a voice interaction request in the target interaction environment, the corresponding voice interaction information is obtained, and the environment interface timing information is generated based on the interface state information of the target interaction environment in the corresponding request window. Step 202: Parse the voice interaction information to obtain the target interaction operation and target description information, so as to determine the target interface and corresponding candidate interface objects that match the voice interaction request in the target interaction environment based on the target description information and the environmental interface timing information, and generate candidate tasks based on the target interaction operation and candidate interface objects. Step 203: Add the candidate task to the task response execution sequence corresponding to the target interaction environment, and determine whether there are any related tasks in the task response execution sequence that were generated earlier than the candidate task and have not yet completed their response execution. Step 204: If they exist, lock the execution of the subsequent response of the associated task, and determine the task iteration relationship between the candidate task and the associated task based on the candidate interface objects and interactive operations corresponding to the candidate task and the associated task, respectively. Step 205: Based on the task iteration relationship, adjust the candidate tasks and / or related tasks to update the task response execution sequence; Step 206: Based on the updated task response execution sequence, determine the target task in the executable state and execute it.

[0024] It is worth noting that if there is no associated task in the task response execution sequence that was generated earlier than the candidate task and has not yet completed its response execution, then there is no need to perform associated task locking and task iteration relationship determination. The task order of the candidate task in the task response execution sequence is maintained, and the task response execution sequence after adding the candidate task is used as the updated task response execution sequence. The target task in the executable state is determined and executed based on the updated task response execution sequence.

[0025] In the aforementioned task response execution method, by associating voice interaction requests with changes in the interface state within the corresponding request window, the interaction intent represented in the voice information can be parsed in conjunction with the interface context at the stage of the voice request. This reduces mismatches of interaction objects caused by interface refreshes, switching, or object position changes during voice request processing, and improves the matching accuracy between voice requests and interface objects in dynamic interaction environments. Based on this, the parsed interaction operations and their corresponding interface objects are organized into schedulable tasks. After a new task is added to the execution process, incomplete associated tasks are identified, and the iteration relationship between old and new tasks is determined under the condition that associated tasks are restricted from continuing execution. Accordingly, related tasks and their execution order are adjusted so that subsequently generated interaction intents can participate in the coordinated processing of previously incomplete responses. This avoids continuing to execute responses that are no longer suitable for the current interaction intent before the task relationship is determined, and reduces the probability of duplication, conflict, or execution disorder between interaction operations corresponding to consecutive voice requests. Therefore, it can balance the accuracy of interaction object determination under dynamic interface conditions with the consistency of response execution between consecutive asynchronous tasks, ensuring that the actual executed interaction response matches the user's interaction intent, and improving the accuracy and reliability of task response execution in voice interaction scenarios.

[0026] In one embodiment, the corresponding voice interaction information is obtained, and environmental interface timing information is generated based on the interface state information of the target interaction environment within the corresponding request window, including: Record the voice data generated when the target user initiates a voice interaction request, and perform voice recognition processing on the voice data to obtain the text information corresponding to the voice interaction request as the voice interaction information; The voice data can be collected by the microphone of the terminal where the target interaction environment is located or by a voice acquisition device that is connected to the terminal; the voice recognition processing can use a voice recognition model pre-configured on the terminal side, or the voice data can be sent to the server for voice recognition; the voice interaction information includes at least the text content converted from the voice data, and may also include the recognition timestamps and recognition confidence of each word in the text.

[0027] Record the start and end times of the voice interaction request respectively, and determine the request window based on the start and end times; The start time can be the moment when the target user begins to input valid voice data, and the end time can be the moment when the input of valid voice data ends. For example, the start and end positions of valid voice segments can be determined by voice activity detection, and the start and end times can be determined based on the system timestamps corresponding to the start and end positions of the valid voice segments; the request window is the time interval defined from the start time to the end time. Based on the preset pre-time tolerance and post-time tolerance, the start and end boundaries of the request window are expanded respectively to obtain the interface observation window; The pre-time tolerance is used to compensate for the time difference between the user's observation of the interface and the start of voice input, while the post-time tolerance is used to compensate for time synchronization errors in voice end detection, interface state acquisition, and different processing modules. The pre-time tolerance and post-time tolerance can be preset according to the terminal's interface refresh rate, voice activity detection delay, and system processing delay. They can be the same or different; for example, the pre-time tolerance can be set to 0.5 to 2 seconds, and the post-time tolerance to 0.2 to 1 second. In one example, the pre-time tolerance is set to 1 second, and the post-time tolerance is set to 0.5 seconds. The above values ​​are only for illustrating the parameter setting method and do not constitute a limitation. Acquire interface state information when at least one interface state change occurs in the target interactive environment within the interface observation window, such as the display, disappearance, position change, or interface switching of interface objects, and record the time of state change corresponding to each interface state information. The interface state information is used to characterize the display state of the interface and / or interface objects in the target interactive environment at a given time. It can be collected by the client corresponding to the target interactive environment when the interface is rendered, the interface is switched, or the attributes of the interface objects change. The interface state information may include at least one of the following: interface identifier, interface object identifier, resource type identifier, object sequence index, object location area, display state, and the time of corresponding state change. The interface identifier is used to distinguish different interfaces, and the interface object identifier is used to distinguish different interactive objects in the corresponding interface. Preferably, the interface identifier and the interface object identifier can be assigned by the client when the interface or object is created, or existing identifiers such as page route identifier, resource identifier, or control identifier can be used. Preferably, the display of an interface object can refer to the interface object changing from a non-displayed state to a displayed state; the disappearance of an interface object can refer to the interface object changing from a displayed state to a non-displayed state; the change in position can refer to the change in the position area of ​​the same interface object in the interface coordinate system; the interface switching can refer to the current displayed interface changing from the first interface to the second interface; for cases where scrolling causes the original interface object to move out of the display area and other interface objects to enter the display area, the exit state and entry state of the corresponding objects can also be recorded separately. The state information of multiple interfaces is sorted in time according to the time of state change. For the same interface and / or the same interface object, the interface state information representing the entry into the corresponding interface state is matched in time with the interface state information representing the exit from the corresponding interface state to determine the duration of the state corresponding to the interface and / or interface object. Based on the duration of the states corresponding to multiple interfaces and / or interface objects and their corresponding temporal relationships, generate environmental interface timing information. The environmental interface timing information can be stored using a time-arranged set of state records, a timing table, or a timing diagram structure. This information is used to characterize when different interfaces and their interface objects appear and last until what duration, as well as the sequential relationship between different interface states within the interface observation window. Each state record in the environmental interface timing information can be associated with an interface identifier, an interface object identifier, a state duration period, and the corresponding interface state information, so that the interface context at the stage of the voice request can be traced back based on the voice interaction information.

[0028] Specifically, for the same interface or the same interface object, the corresponding interface state information can be traversed from earliest to latest according to the state change time. The state change time of entering the corresponding interface state is taken as the start time of the state duration, and the state change time of the first exit from the interface state is taken as the corresponding end time, thus obtaining the corresponding state duration. For interfaces or interface objects that are already in the display state when the interface observation window starts, the start time of the interface observation window can be taken as the start time of the corresponding state duration; for interfaces or interface objects that have not exited the corresponding state by the end of the interface observation window, the end time of the interface observation window can be taken as the end time of the corresponding state duration. In this way, it is possible to avoid the inability to form a complete state duration due to state entry events or state exit events being outside the boundaries of the interface observation window.

[0029] Specifically, in this embodiment, the request window is defined by the actual start and end times of the voice interaction request, and the interface observation range is expanded using pre-time tolerance and post-time tolerance. This compensates for the time deviation between voice acquisition, voice recognition, and interface state acquisition. Simultaneously, by recording the state changes and corresponding change times of the interface and interface objects, and performing time-series matching of entry and exit states, the actual duration of each interface and interface object can be determined, thereby reconstructing the interface evolution process of the target interaction environment before and after the voice interaction request occurs. Therefore, subsequent processing can determine the object pointed to by the user based on the interface context at the time of the voice request, rather than being limited to the current interface at the processing time. This improves the time alignment accuracy between voice interaction information and interface state, and reduces object misbinding caused by interface switching, refreshing, or object position changes.

[0030] In one specific embodiment, when obtaining the interface state information corresponding to the target interactive environment within the interface observation window, at least one of the following can also be obtained: time base identifier, interface instance identifier, interface state generation, and object set version information corresponding to the interface state information. The time reference identifier is used to characterize the time recording reference used for voice data and interface state information. Preferably, the start and end times of voice interaction requests and the state change times corresponding to interface state information are recorded using the same monotonic time reference to avoid the impact of terminal system time adjustments, network time synchronization, or time zone changes on the chronological order of events. If the voice data and interface state information are generated by processing modules using different time references, the time offset can be determined based on the synchronous events recorded by the two processing modules, and the corresponding timestamps can be mapped to a unified time reference based on the time offset. When the voice recognition result has word-level or sentence-level time information, the word-level or sentence-level time information can be converted into timestamps under a unified time reference based on the start time of voice data acquisition to determine the time period of the description corresponding to the target description information. When the voice recognition result does not have corresponding time information, the request window or interface observation window can be used as the degradation time range corresponding to the target description information, and the matching conditions used for subsequent interface object matching can be appropriately increased. The interface instance identifier is used to distinguish interface instances of the same interface type formed in different display lifecycles; the interface state generation is used to distinguish different interface states of the same interface instance before and after content refresh, object rearrangement, page turning, or resource set replacement; for example, when it is detected that the target interface performs refresh, reload, page turning, or the resource set it displays changes, the corresponding interface state generation can be incremented, and the changed interface state information can be associated with the new interface state generation.

[0031] In one specific embodiment, for a target interface containing multiple interface objects with a defined arrangement relationship, an object set identifier and an object set version can be configured for the object sets corresponding to the multiple interface objects, and the order index of each object can be associated with the corresponding object set identifier and object set version. Therefore, when determining candidate interface objects based on object order constraints such as "first," "second," and "last," the object order index is compared only among interface objects whose object set identifier and object set version both match.

[0032] If the same target interface is refreshed or rearranged during a voice interaction request, causing a change in the object set version, the order indices of the same objects before and after the refresh or rearrangement belong to different object set versions. The changed positions of the same objects are not directly replaced with the interface objects the user was referring to before the change. This avoids silent mismatches that occur when the interface identifiers and object order remain similar after a refresh or resource rearrangement, even if the object being referred to by the user has changed.

[0033] In one specific embodiment, the voice interaction information can also be associated with the corresponding voice recognition version information; before the candidate task has completed the response execution, if an updated voice recognition result for the same voice interaction request is obtained, the updated voice recognition result is compared with the voice interaction information used by the current candidate task. In response to the change in at least one of the target interactive operation, target description information, temporal reference information or object constraint information caused by the update of the speech recognition result, a new version of the speech interaction information corresponding to the updated speech recognition result is generated, and candidate tasks formed based on the original version of the speech interaction information that have not yet been executed are locked. Based on the new version of voice interaction information, the target interaction operation, target description information, target interface and candidate interface objects are redefined, and the task iteration relationship between the relevant listed tasks in the task response execution sequence is redefined. For interactive operations where the original candidate task has already completed the response execution, the operation results will not be directly rolled back due to the correction of the speech recognition result; for interactive operations where the original candidate task has not yet completed the response execution, the decision to retain, delete or replace will be made according to the redefined task iteration relationship. If there is no substantial change in the target interaction operation and object constraint information between the updated speech recognition result and the current speech interaction information, the existing candidate task and task response execution sequence can be maintained without re-executing the interface object matching process.

[0034] In one embodiment, based on target description information and environmental interface timing information, the target interface matching the voice interaction request in the target interactive environment and the corresponding candidate interface object are determined, including: Parse the target description information to obtain temporal reference information and object constraint information. The temporal reference information is used to characterize the temporal range of the interface corresponding to the referenced object, and the object constraint information is used to characterize at least one of the resource type, object order and object spatial location of the referenced object. The temporal reference information can include at least one of current reference, historical reference, and relative temporal reference. For example, "this book" and "this one" can be resolved as temporal reference information pointing to the current interface at the stage where the voice interaction request occurs; "the page before" and "the previous page" can be resolved as temporal reference information pointing to the historical interface before the current interface; the object constraint information can be obtained from semantic components such as nouns, quantifiers, ordinal numbers, and locative words in the target description information. For example, "the second book" can obtain object constraint information that the resource type is books and the object order is second; "the course on the left" can obtain object constraint information that the resource type is courses and the object spatial location is the left area. Preferably, the target description information can be parsed using preset semantic rules, natural language processing models, or a combination of both; for target description information that does not contain explicit temporal pronouns, the temporal range corresponding to the interface that is in the display state when the voice interaction request occurs can be used as the default interface temporal range. Based on the matching results of the timing reference information and the duration of multiple interfaces in the timing information of the environment interface, as well as the timing relationship between multiple interfaces, at least one interface to be matched is selected from multiple interfaces. Specifically, the target temporal range or position relative to the request window can be determined first based on the temporal reference information. Then, it can be determined whether the duration of each interface's state overlaps with the target temporal range, or whether the relative position of the corresponding interface in the interface temporal sequence satisfies the temporal reference information. For the current reference, interfaces whose state duration overlaps with the start time, end time, or a preset reference time between the two of the voice interaction request can be filtered. For historical reference, one or more historical interfaces can be determined from the nearest to the furthest exit time of each interface. If multiple interfaces satisfy the corresponding temporal constraints, they are all retained as interfaces to be matched, and further disambiguation is achieved through subsequent object matching. If no interface satisfies the temporal constraints, the interface with the closest time distance to the target temporal range can be used as the interface to be matched, or the current voice interaction request can be marked as a state to be re-parsed. Extract at least one of the resource type identifier, object sequence index, and object location region corresponding to the interface object contained in at least one interface to be matched, and generate interface object feature information; The resource type identifier is used to represent the resource category corresponding to the interface object, such as books, courses, chapters, audio, or video; the object sequence index is used to represent the arrangement position of the interface object in the corresponding interface or object collection; the object position area is used to represent the spatial position of the interface object in the interface coordinate system, which can be represented by the bounding box coordinates, center point coordinates, or pre-divided position areas such as left, right, top, bottom, and middle of the interface object; the above information can be obtained from the attribute information, layout information, or business resource information of the interface object during the interface rendering process; The interface object feature information is compared with the object constraint information to determine the resource type sub-matching degree based on the consistency of the resource type identifier, the object order sub-matching degree based on the difference between the object order indices, and the spatial location sub-matching degree based on the positional distance or region overlap between the object location region and the constraint location region represented by the object constraint information. Based on the constraint terms contained in the object constraint information, determine the fusion weights corresponding to the resource type sub-matching degree, object order sub-matching degree, and spatial location sub-matching degree, and perform weighted fusion on the corresponding sub-matching degrees to obtain the object matching degree corresponding to the interface object; In this process, only the sub-matching degrees corresponding to the constraints actually included in the target description information participate in the fusion. Each fusion weight can be pre-configured or dynamically determined according to the recognition confidence of different constraints. Preferably, the sum of the fusion weights participating in the fusion is 1. For example, when the object constraint information includes resource type, object order, and object spatial location, fusion weights of 0.3, 0.4, and 0.3 can be set respectively; when only resource type and object order are included, fusion weights of 0.4 and 0.6 can be set respectively. The above weights are just examples. Specifically, the object matching degree can be obtained by summing the products of each sub-matching degree and the corresponding fusion weight, that is, so that the interface object with a higher comprehensive constraint compliance degree obtains a higher object matching degree. The interface matching degree of the interface to be matched is determined based on the object matching degree of the interface objects within the same interface to be matched, and the target interface is determined from at least one interface to be matched based on the interface matching degree. The interface matching degree is used to characterize the degree to which the corresponding interface to be matched as a whole satisfies the target description information. Preferably, the highest object matching degree within the same interface to be matched can be used as the interface matching degree of that interface. When the target description information involves multiple objects, the matching degrees of the top-ranked preset number of objects can be summed or averaged to obtain the interface matching degree. The interface to be matched with the highest interface matching degree is determined as the target interface. When the interface matching degrees of multiple interfaces to be matched are the same or the difference is less than a preset difference threshold, the selection can be further based on the time distance between the corresponding interface and the time series reference range, and the interface to be matched with the smaller time distance is preferentially determined as the target interface. Interface objects in the target interface whose object matching degree meets the preset matching conditions are identified as candidate interface objects; The preset matching conditions can be that the object matching degree is not lower than the preset matching threshold, or that the object matching degree is ranked from high to low and is among the first preset number. For example, when the object matching degree uses a normalized value of 0 to 1, the preset matching threshold can be set to 0.6, 0.7, or adjusted according to historical matching results. When there is no interface object that meets the preset matching conditions, no candidate interface object can be generated, and the corresponding voice interaction request can be set to a matching failure state or trigger re-parsing processing.

[0035] Specifically, each sub-matching degree can be normalized to a numerical range of 0 to 1. For resource type sub-matching degree, it can be set to 1 when the resource type identifier of the interface object matches the resource type defined by the object constraint information, and set to 0 when they do not match. When there is a resource type hierarchy or similarity relationship, the value between 0 and 1 can also be determined according to the preset type association degree between the two resource types. Specifically, for object order sub-matching degree, it can be determined according to the difference between the object order index of the interface object and the target order index defined by the object constraint information. For example, a function that monotonically decreases as the absolute value of the index difference increases can be used to calculate the object order sub-matching degree, so that the sub-matching degree is 1 when the indices are completely consistent, and the interface object with a larger index deviation gets a lower sub-matching degree. For spatial location sub-matching degree, the normalized distance between the center position of the interface object and the center position of the constraint position area can be calculated, and the spatial location sub-matching degree can be reduced as the distance increases; or the ratio of the intersection area of ​​the position area of ​​the interface object and the constraint position area to the union area of ​​the two can be calculated, and this ratio or its monotonic mapping result can be used as the spatial location sub-matching degree. Therefore, those skilled in the art can choose between distance matching or region overlap matching based on the actual interface layout.

[0036] Specifically, in this embodiment, by decomposing the target description information into temporal reference information for limiting the time period of the interface appearance and object constraint information for limiting object attributes, the range of interfaces to be matched can be narrowed down first based on the duration of the interface and the temporal relationship between interfaces. Then, the interface objects within this range are matched from dimensions such as resource type, arrangement order, and spatial location. Furthermore, by configuring the fusion weight of the corresponding sub-matching degree according to the constraint items actually included in the target description information, it is possible to avoid undue interference from object features not specified by the user on the matching results. On this basis, the object matching degree is used to further evaluate the degree of interface matching and determine the target interface and candidate interface objects. This can achieve hierarchical disambiguation of the temporal range of the interface and the object attribute constraints, thereby accurately distinguishing the current interface object from the historical interface object and improving the recognition accuracy of the target interface and target object under complex reference expressions.

[0037] In one specific embodiment, after determining the candidate interface object, resource association information corresponding to the candidate interface object can also be obtained, and resource verification status information can be formed based on the resource association information.

[0038] Specifically, the process involves obtaining the object identifier corresponding to the candidate interface object and the resource identifier or temporary resource identifier provided by the target interaction environment; querying pre-defined authoritative resource data based on the resource identifier or temporary resource identifier to determine the standard resource identifier actually corresponding to the candidate interface object; and associating and saving the standard resource identifier with at least one of the following: the candidate interface object's object identifier, the target interface, the object set identifier, the object set version, and the object sequence index. The standard resource identifier, after verification by the authoritative resource data corresponding to the target interaction environment, uniquely represents a business resource that can be actually used by the target interaction operation. The resource identifier carried by the candidate interface object itself can be used as the basis for obtaining the standard resource identifier, but resource validity verification can also be performed on it before the target interaction operation is actually executed.

[0039] In one specific embodiment, when the standard resource identifier is determined for the first time, the resource version, resource availability status, permission verification result and verification time can be recorded to form a corresponding resource status snapshot. After the resource status snapshot is formed, the historical verification results it records remain unchanged. When the availability status or permission status of the corresponding resource changes, the current resource availability status information is formed separately, instead of using the changed status to overwrite the original resource status snapshot. Before executing the target task according to the updated task response execution sequence, the current resource availability status corresponding to the canonical resource identifier is retrieved again and compared with the operation type and operation parameters corresponding to the target task. If the canonical resource identifier still meets the execution conditions of the target interactive operation, the target task is allowed to proceed to subsequent execution. If the canonical resource identifier does not meet the execution conditions of the target interactive operation, the corresponding interactive operation is prevented from continuing to execute. When the target description information explicitly points to a specific interface object or a specific object sequence position, if the resource corresponding to the original candidate interface object is currently unavailable, other resources in the same object sequence position after refreshing or rearranging will not be directly replaced with the original candidate interface object; when the target description information only limits the resource category but allows the use of other objects that meet the conditions, the candidate interface object determination or resource filtering process can be re-executed.

[0040] Specifically, this embodiment records and judges "which interface object the user is referring to during the voice request stage" and "whether the resource corresponding to the interface object can still be operated during execution" respectively. This preserves the identity of the target user's original referent and can also cope with situations such as resource removal, permission changes, or resource status changes during task waiting, avoiding the concealment of resource status changes by modifying historical object binding results.

[0041] In another embodiment, the object referred to by the target description information may also be an expected interface object that has not yet been formed in the target interaction environment but needs to be generated by the corresponding pre-interaction operation; Specifically, when parsing the target description information to obtain the description content used to characterize "performing the next operation on the object in its result after completing the previous operation", it is determined that the target description information contains the expected object reference, and the preceding interactive operation required to generate the corresponding expected object and the subsequent target interactive operation acting on the expected object are determined. Generate a pre-interaction task to perform the pre-interaction operation, and establish a result dependency relationship between the subsequent target interaction operation and the pre-interaction task; when the pre-interaction task has not yet generated a corresponding set of objects, set the subsequent target interaction operation to a dependent waiting state, instead of inferring candidate interface objects from similar interface objects in the current display interface or the history interface. In response to the completion of the preceding task and the generation of a result object set with a defined object order, the corresponding candidate interface objects are determined based on the object set identifier, object set version, and object order index of each result object, combined with the object constraint information in the target description information, and the task generation and response execution of subsequent target interaction operations are resumed. If the preceding task is canceled, fails to execute, or fails to form a set of corresponding result objects after a preset waiting time, the subsequent target interaction operation applied to the expected object will not be executed, and confirmation information will be output or the corresponding task will be terminated according to the corresponding abnormal status.

[0042] For example, in a voice interaction request to "recommend three management books, and open the first one immediately after the recommendation," the interface object corresponding to the "first book" has not yet been actually generated when the request is initiated. In this case, the management book recommendation operation can be performed first. After the recommendation results form a set of objects with definite versions and order, the corresponding candidate interface object can be determined according to the object order constraint of the "first book," instead of using the first book on the current page at the time the request is initiated as the target object.

[0043] In one embodiment, generating candidate tasks based on the target interactive operation and candidate interface objects includes: Determine the corresponding operation type and operation parameters based on the target interactive operation; The operation type is used to characterize the type of interactive behavior that needs to be performed on the interface object, including but not limited to opening, closing, playing, pausing, switching, selecting, displaying, and canceling display; the operation parameters are used to limit the specific execution content of the corresponding interactive behavior, which may include at least one of resource identifier, playback position, display method, target state, and action parameters. For example, for "play the third chapter of this book", the operation type can be playback, and the operation parameters may include the corresponding book resource identifier and the chapter identifier corresponding to the third chapter. Based on the object matching degree of the candidate interface objects, the candidate interface objects are filtered and sorted to obtain a candidate object sequence; The system can sort objects from highest to lowest matching degree and filter them based on a preset quantity threshold or a preset matching degree threshold. For example, it can retain the top 3 candidate interface objects with the highest matching degree, or retain candidate interface objects with a matching degree of not less than 0.6. The specific threshold can be predetermined or dynamically adjusted based on the object matching accuracy of historical voice interaction samples. Based on the interface state information corresponding to the target interface, the executability of the candidate object sequence is determined, and the candidate interface objects in the executable state are retained; The executable state is used to characterize whether the corresponding candidate interface object can accept the target interactive operation in the corresponding state of the target interface. Specifically, it can be determined based on at least one of the following in the interface state information: object display state, object interactive attributes, resource validity state, and execution conditions corresponding to the operation type. For example, when the target interactive operation is an open operation, it can be determined whether the candidate interface object is in the display state and whether it is associated with a valid openable resource; when the target interactive operation is a play operation, it can be determined whether the candidate interface object is associated with a playable resource. If the corresponding operation execution conditions are met, the candidate interface object is retained; otherwise, it is excluded from the candidate object sequence. The retained candidate interface objects are associated and combined with the target interface, operation type and operation parameters to generate multiple candidate tasks. The candidate tasks are used to instruct the corresponding candidate interface objects to perform the target interactive operation on the target interface. Candidate tasks can be stored using a task data structure. This task data structure includes at least a task identifier, a target interface identifier, a candidate interface object identifier, an operation type, operation parameters, and a task execution status. It may also include information such as the task generation time, object matching degree, and prior operation identifier to support subsequent task association judgment and task scheduling. The order of multiple candidate tasks is determined based on object matching degree.

[0044] Specifically, multiple candidate tasks can be ranked from highest to lowest based on the object matching degree of the candidate interface objects associated with each candidate task, so that candidate tasks with higher object matching degrees have higher processing priority. When only one candidate interface object is retained, the corresponding single candidate task can be generated directly. If there is no candidate interface object in the candidate object sequence that is in an executable state, no candidate task can be generated, and the current voice interaction request can be set to a temporarily unexecutable or matching failure state to avoid generating invalid tasks for unexecutable objects.

[0045] Specifically, in this embodiment, by extracting the operation type and operation parameters from the target interaction, and filtering and sorting candidate interface objects based on object matching degree, interface objects that are more consistent with the user description can be given higher task processing priority. Furthermore, by combining the actual state of the target interface to determine the executability of candidate interface objects, objects that cannot accept the corresponding operation in the current state can be excluded during the task generation stage. Based on this, the executable candidate interface objects are associated with the target interface and corresponding operation content to form an ordered candidate task. This can convert voice requests with certain ambiguities into schedulable tasks with clear target interfaces, objects, and operation content, thereby reducing the generation of invalid tasks and erroneous operations, and improving the accuracy and efficiency of subsequent task scheduling and response execution.

[0046] In another embodiment, after obtaining the object matching degree corresponding to each interface object in the target interface, a uniqueness judgment can be performed on the candidate interface objects before generating candidate tasks; specifically, the interface objects in the target interface are arranged from high to low according to the object matching degree to determine the first object matching degree and the second object matching degree; it is determined whether the first object matching degree reaches the preset acceptance threshold, and whether the matching degree difference between the first object matching degree and the second object matching degree reaches the preset distinction threshold. In response to the first object matching degree reaching the preset acceptance threshold and the matching degree difference reaching the preset differentiation threshold, the interface object corresponding to the first object matching degree is determined as the unique candidate interface object, and a candidate task is generated based on the unique candidate interface object. If the first object matching degree does not reach the preset acceptance threshold, or the matching degree difference does not reach the preset differentiation threshold, it is determined that the current target description information is insufficient to uniquely determine the candidate interface object. No executable candidate task is generated for the time being, and object confirmation information used to distinguish at least two highly matched interface objects is output. After obtaining the confirmation information returned by the target user, the candidate interface object is re-determined based on the confirmation information. The preset acceptance threshold and preset discrimination threshold can be statistically set based on the correct matching samples corresponding to historical voice interaction requests. For example, when the object matching degree is normalized to 0 to 1, the preset acceptance threshold can be set to 0.7 to 0.8 and the preset discrimination threshold can be set to 0.1 to 0.2. The above values ​​are only illustrative examples.

[0047] Specifically, in this embodiment, by simultaneously limiting the absolute reliability of the highest object matching degree and its distinguishability relative to the second highest object matching degree, it is possible to avoid directly forming an executable task simply because one of the interface objects has a slightly higher matching degree when multiple interface objects can satisfy the user's partial description. This further reduces erroneous operations caused by ambiguous ordinal references, spatial references, or historical interface references.

[0048] In one embodiment, determining whether there exists an associated task in the task response execution sequence that was generated earlier than the candidate task but has not yet completed its response execution includes: Obtain the response execution status of multiple tasks in the task response execution sequence, excluding candidate tasks, and filter out the tasks in the queue that have not yet completed response execution from the multiple tasks in the queue; The response execution status can include pending execution, in-process execution, paused, and completed status; the listed tasks that have not yet completed response execution can include tasks in the pending execution, in-process execution, or paused status; each task in the task response execution sequence can record the task generation time and current response execution status, and update them according to the actual execution progress during task execution; Retrieve candidate tasks and the target interface and candidate interface objects corresponding to each of the filtered tasks in the list; Compare the target interface and candidate interface object corresponding to the candidate task with the target interface and candidate interface object corresponding to each task in the list, respectively. Among them, target interfaces can be compared through interface identifiers, and candidate interface objects can be compared through object identifiers or resource identifiers associated with the corresponding objects. When the interface identifiers are the same, it can be determined that the corresponding tasks are applied to the same target interface. When the object identifiers are the same, or the object identifiers are different but correspond to the same business resources, it can be determined that the corresponding tasks are applied to the same candidate interface objects. Based on the comparison results, identify the tasks to be associated with the candidate tasks that act on the same target interface and / or the same candidate interface object from the selected tasks in the list. Among them, the association conditions for "same target interface and / or same candidate interface object" can be set according to specific application scenarios. For example, for page-level operations, tasks that act on the same target interface can be identified as tasks to be associated. For object-level operations, tasks that act on the same candidate interface object can be identified as tasks to be associated. If the task in the list and the candidate task do not act on the same target interface or the same candidate interface object, then the task in the list will not be identified as a task to be associated, and it will be allowed to continue to execute according to the original task response execution sequence. Obtain the generation time of each task to be associated, and sort the multiple tasks to be associated in chronological order according to their generation time; Based on the time sequence sorting results, determine the task with the latest generation time from multiple tasks to be associated, and / or determine the task to be associated whose generation time interval with the task with the latest generation time meets the preset duration condition, and use it as the associated task. The preset duration condition is used to limit the time range of continuous interaction context. It can be preset according to the statistical time interval of continuous voice requests in the target interaction environment. For example, the preset duration can be set to 3 seconds, 5 seconds, or 10 seconds, and tasks to be associated that have a generation time interval of no more than the preset duration with the task to be associated with the latest generation time can be identified as associated tasks. In one implementation, the task to be associated with the latest generation time can always be identified as an associated task, and other tasks to be associated that have a generation time interval of no more than 5 seconds with that task can also be identified as associated tasks. The above values ​​are only examples and do not constitute a limitation. When there are no tasks to be associated, or when none of the tasks to be associated meet the preset association conditions, it can be determined that there are no associated tasks corresponding to the candidate tasks in the task response execution sequence. At this time, there is no need to perform pause and task iteration processing for associated tasks. The candidate tasks can enter the subsequent execution process according to the scheduling rules corresponding to the task response execution sequence. Pause the subsequent response execution of the associated task, and maintain the current response execution state of the associated task in the task response execution sequence; Specifically, pausing the subsequent response execution of associated tasks means preventing associated tasks from initiating interactive operations that have not yet started, while not performing rollbacks for interactive operations that have already been completed; for atomic interactive operations that are being executed and cannot be interrupted immediately, it is permissible to enter the paused state after the atomic interactive operation has finished executing; maintaining the current response execution state means saving the interactive operations that have been completed, the interactive operations that have not yet been completed, the current operation position, and the operation results that have been generated by the associated tasks, so as to determine the task iteration relationship and adjust the task response execution sequence later.

[0049] Specifically, in this embodiment, by filtering incomplete tasks from the task response execution sequence and determining tasks with a common scope of influence with the candidate tasks based on the consistency between the target interface and the candidate interface objects, the task iteration judgment can be limited to tasks that may influence each other, avoiding interference from irrelevant tasks. Furthermore, by combining the task generation sequence and generation time interval to select associated tasks, the determined associated tasks can be concentrated on prior tasks with strong contextual continuity with the current voice request. By pausing the subsequent response execution of associated tasks and maintaining their current execution state, it is also possible to prevent associated tasks from continuing to change related interfaces or objects before new task relationships are determined, while preserving their existing execution progress. This reduces operational conflicts caused by the parallel advancement of asynchronous tasks and provides a stable state basis for the determination and sequence adjustment of subsequent task iteration relationships.

[0050] In one embodiment, determining the task iteration relationship between candidate tasks and associated tasks based on the candidate interface objects and interaction operations corresponding to the candidate tasks and associated tasks respectively includes: Get at least one pending interactive operation in the associated tasks that has not yet completed the response execution, and the target interactive operation corresponding to the candidate task; A task may include one or more interactive operations that are executed in a predetermined order; an interactive operation to be executed refers to an interactive operation in an associated task that is after the current execution progress and has not yet been actually completed. It may include interactive operations that have not yet started and interactive operations that have stopped executing due to the suspension of the associated task. Compare the target interactive operation with each interactive operation to be executed; If the target interactive operation and at least one interactive operation to be executed correspond to the same candidate interface object, and the two have the same operation type but different operation parameters, then the corresponding interactive operation to be executed is determined as the first operation to be iterated. Among them, the same operation type but different operation parameters are used to characterize the parameter-level update of the operation purpose for the same interface object. For example, when the interaction operation to be executed in the associated task is to play the target resource according to the first playback position, and the target interaction operation in the candidate task is to play the same target resource according to the second playback position, the aforementioned interaction operation to be executed can be determined as the first operation to be iterated. If the target interaction operation and at least one interaction operation to be executed correspond to the same candidate interface object, and the target interaction operation and the interaction operation to be executed have a mutually exclusive execution relationship, then the corresponding interaction operation to be executed is determined as the second operation to be iterated. Mutually exclusive execution relationship refers to the relationship that two interactive operations on the same candidate interface object should not be retained or implemented continuously in the same execution phase. Preferably, a mutual exclusion relationship table between operation types can be established in advance. For example, operation types such as "play" and "stop playing", "open" and "close", "display" and "cancel display" can be set as mutually exclusive operation types, and the mutual exclusion relationship table can be queried to determine whether two interactive operations have a mutually exclusive execution relationship. The mutual exclusion relationship table can be pre-configured according to the operation types and business rules supported by the target interaction environment. In response to the existence of a first operation to be replaced and / or a second operation to be replaced, it is determined that there is a task substitution relationship between the candidate task and the associated task; Accordingly, when there is no first operation to be replaced and no second operation to be replaced, it can be determined that no task substitution relationship between the candidate task and the associated task has been identified based on the above operation comparison results. In this case, no deletion processing based on the task substitution relationship is performed on the interaction operations to be executed in the associated task. Obtain the interface state information required for the target interactive operation corresponding to the candidate task, and determine whether the interface state information contains the operation result generated by the interactive operation that has been completed by the associated task. The interface state information required for the target interactive operation can be determined based on the execution interface or operation template corresponding to the operation type. For example, when the target interactive operation requires the use of state information such as the current resource list, target resource identifier, current playback object, or opened page, the corresponding state information can be determined as the interface state information to be called. The operation result generated by the completed interactive operation can include at least one of the following: interface jump result, resource loading result, object display result, list generation result, and object state change result, and can be associated with and saved with the operation identifier of the interactive operation after the corresponding interactive operation is completed. If included, the completed interactive operation that produces the operation result is identified as the preceding interactive operation; the operation parameters corresponding to the target interactive operation are matched with the operation result produced by the preceding interactive operation, and when the operation result satisfies the operation parameters corresponding to the target interactive operation, it is determined that there is a task dependency relationship between the candidate task and the associated task. Specifically, resource identifiers, interface identifiers, object identifiers, or state parameters can be extracted from the operation results and compared with the corresponding operation parameters required when the target interactive operation is executed. When the resources, objects, or states required by the target interactive operation can be provided by the operation results of the preceding interactive operation, it can be determined that the operation results satisfy the operation parameters corresponding to the target interactive operation. For example, if the associated task has completed the resource list loading operation, and the target interactive operation needs to perform an open operation on the target object in the resource list, then when the resource list contains the object specified by the target interactive operation, it can be determined that there is a task dependency relationship between the two. If the interface status information does not contain the operation results generated by the completed interaction operation of the associated task, or although it contains the corresponding operation results, the operation results do not meet the operation parameters corresponding to the target interaction operation, then it can be determined that the task dependency relationship between the candidate task and the associated task has not been identified based on the operation results. Based on the results of determining the task substitution relationship and task dependency relationship, determine the task iteration relationship between the candidate task and the related task; Among them, the task iteration relationship is used to characterize the adjustment relationship between the interaction operation of the candidate task and the existing interaction operation of the associated task after the candidate task arrives. It can include at least the existence of only task substitution relationship, only task dependency relationship, both task substitution relationship and task dependency relationship, and neither of the above two relationships. In the case where there is no task substitution relationship and no task dependency relationship, the pending operations of the candidate task and the associated task can be maintained, and the two can be not iterated and merged. The tasks can continue to be executed according to the original scheduling order or the preset task priority.

[0051] Specifically, in this embodiment, by comparing the target interaction operation of the candidate task with the unexecuted interaction operations in the associated tasks one by one, operations that act on the same interface object but whose operation parameters have changed, as well as operations that are mutually exclusive with the target interaction operation, can be identified. This determines the substitution relationship between the candidate task and the unexecuted part of the associated task. Simultaneously, by determining whether the interface state required by the target interaction operation includes the result of the operation already completed in the associated task, and further verifying the compatibility of this result with the target interaction operation parameters, it can be determined whether the candidate task needs to use the operation result already formed by the associated task. Therefore, it is possible to distinguish different iteration scenarios such as modification, negation, and acceptance of prior requests by new requests, retaining valid execution results that can be used by subsequent tasks while replacing conflicting operations. This avoids the loss of valid preceding results due to simple overwriting, or the repeated implementation of mutually exclusive operations due to simple sequential execution, thus improving the accuracy of task relationship recognition in continuous voice interaction scenarios.

[0052] In another embodiment, after determining the task iteration relationship between candidate tasks and associated tasks, the method further includes: Retrieve response artifacts from related tasks that have not yet been completed and have been executed as intermediate results; Based on the cancellation, negation, or modification semantics represented by the target description information, and the data required for the execution of the candidate task, determine whether the candidate task needs to prevent the associated task from continuing to output the corresponding response product, and whether the candidate task needs to use the intermediate execution results already formed by the associated task. In response to the need for a candidate task to prevent associated tasks from continuing to output corresponding response products, the corresponding response products in the associated tasks that have not yet completed their response output are set to a prohibited response state. In response to the need for candidate tasks to use the intermediate execution results already formed by associated tasks, further compare the compatibility of the target interface, candidate interface objects, resource types, and object constraints corresponding to the intermediate execution results with the corresponding conditions of the candidate tasks. If the intermediate execution result is compatible with the corresponding conditions of the candidate task, the intermediate execution result is retained, and a result reuse association is established between the intermediate execution result and the candidate task, so that the candidate task can use the corresponding intermediate execution result. If the intermediate execution result is incompatible with the corresponding conditions of the candidate task, the candidate task is prohibited from using the corresponding intermediate execution result, and the data required for execution is re-obtained according to the candidate task. If a candidate task needs to prevent associated tasks from continuing to output corresponding response products, but also needs to use the intermediate execution results that have been formed by the associated tasks and meet the compatibility conditions, then while keeping the response products in a state of prohibiting further response, the intermediate execution results are retained and used only by the candidate task or subsequent tasks that have a clear dependency on the candidate task. Among them, the response product refers to at least one of the text information, resource display information and interface interaction operations used to provide to the target user during the task response execution process; the intermediate execution result refers to the data that the associated task has been formed through response execution and can be used as input for subsequent task execution, including at least one of the following: a determined set of interface objects, a resource list, an object arrangement result, an interface state change result, and an interaction operation execution result. Preferably, the task identifier and applicable scope of the intermediate execution result can be recorded for reuse. Before actually reading the corresponding intermediate execution result, the candidate task verifies again whether its own task identifier belongs to the scope of reuse and whether the target interface, resource type and object constraints corresponding to the intermediate execution result still match the candidate task.

[0053] In one specific embodiment, when the intermediate execution result required by a candidate task has not yet been formed, the interactive operations required to generate the intermediate execution result in the associated task can continue to be executed, and the candidate task can be set to a dependent waiting state, instead of inferring the target object based on similar objects in the current interface. Once the intermediate execution result is formed and the corresponding compatibility conditions are met, the subsequent response execution of the candidate task is resumed; if the intermediate execution result is not obtained after a preset dependent waiting time, the candidate interface object can be re-determined, a confirmation message can be output, or the corresponding candidate task can be terminated. Specifically, in this embodiment, by separately determining whether the user-perceptible response of the incomplete associated task is still applicable to the current user intent, and whether the intermediate execution result of the associated task can still provide valid input for the candidate task, the "continue to respond to the user" and "whether the existing execution result is still usable" of the associated task no longer use the same failure rules. Therefore, when a subsequent voice request cancels or modifies the prior response form, it can prevent the continued output of a delayed response that no longer conforms to the current user intent, while retaining the valid execution result that subsequent tasks still clearly depend on, thereby reducing the repeated acquisition of interface objects or resource data and avoiding the loss of valid results required by subsequent tasks due to directly canceling the associated task.

[0054] In one embodiment, adjusting the candidate tasks and / or the associated tasks to update the task response execution sequence based on the task iteration relationship includes: Based on the interactive operations that have completed their responses and the interactive operations that have not yet completed their responses in the associated tasks, determine the current execution boundary of the associated tasks, and determine the sequence of interactive operations to be executed based on the current execution boundary; The current execution boundary is used to distinguish between interactive operations in related tasks that have already produced execution results and will no longer be modified, and interactive operations that are not yet completed but can be adjusted later. Specifically, the position after the last interactive operation in the related task that has completed its response can be determined as the current execution boundary; if there are no interactive operations in the related task that have completed their response, the position before the first interactive operation in the related task can be determined as the current execution boundary. By extracting the interactive operations to be executed after the current execution boundary according to the original execution order, the sequence of interactive operations to be executed can be obtained. In response to the existence of a task substitution relationship between candidate tasks and associated tasks, the first to be iterated operation and / or the second to be iterated operation are deleted from the sequence of interactive operations to be executed. When there is no task dependency relationship between candidate tasks and associated tasks, the operation position corresponding to the earliest operation position before deletion of each to be iterated operation is determined as the target insertion position. In the case of deleting multiple operations to be iterated, the target insertion position is the earliest position of the original operation. This allows the target interactive operation to take over the original operation to be executed from the earliest position where it was iterated. At the same time, the interactive operations located after this position that have not been iterated continue to maintain their relative execution relationship. For example, if the original interactive operations to be executed are A, B, C, and D in sequence, and B and D are determined to be operations to be iterated, then after deleting B and D, the original operation position corresponding to B can be used as the target insertion position. In response to the existence of task dependencies between candidate tasks and related tasks, the first position to be executed after the current execution boundary is determined as the target insertion position; When candidate tasks have both task substitution and task dependency relationships, the corresponding first and / or second pending operations can be deleted first, and then the first pending execution position after the current execution boundary can be used as the target insertion position. Since the preceding interactive operation that produces the dependency result is already located before the current execution boundary and has completed the response execution, setting the target interactive operation after the current execution boundary can allow the target interactive operation to enter the subsequent execution process with priority while retaining the execution result of the preceding interactive operation. Insert the target interaction operation corresponding to the candidate task into the target insertion position, and maintain the original relative order between the non-deleted interaction operations in the sequence of interaction operations to be executed, to obtain the updated sequence of interaction operations to be executed; for example, when the sequence of interaction operations to be executed is A, B, C and the target insertion position is between A and B, the target interaction operation X can be inserted to obtain A, X, B, C; except for the operations deleted due to task substitution, the relative order of the other operations remains unchanged; Remove the target interaction operation that has been inserted at the target insertion position from the pending interaction operations corresponding to the candidate task, and remove the candidate task from the task response execution sequence if there are no other pending interaction operations for the candidate task; Removing the target interaction operation from the sequence of interaction operations to be executed in the associated task can prevent the same target interaction operation from being repeatedly scheduled as both a candidate task operation and an associated task operation. When the candidate task also includes other interaction operations to be executed that have not been inserted, the candidate task can be retained and its task content and execution status can be updated according to its remaining interaction operations to be executed. The updated sequence of interactive operations to be executed is appended to the interactive operations that have already been executed in the associated tasks, and the task response execution sequence is updated according to the adjusted associated tasks and candidate tasks; In cases where only task substitution exists, the update can be completed by deleting the operation to be replaced and inserting the target interaction operation at the corresponding replacement position. In cases where only task dependency exists, the existing operations to be executed in the associated tasks can be retained, and the target interaction operation can be inserted after the current execution boundary, allowing it to take over the completed preceding interaction operation. In cases where both task substitution and task dependency exist, the target interaction operation can be set after the current execution boundary after deleting the corresponding operation to be replaced. In cases where neither task substitution nor task dependency exists, the above insertion and deletion processing can be omitted from the sequence of interactions to be executed in the associated tasks; the candidate task continues to be retained as an independent task in the task response execution sequence and is executed according to the original scheduling rules of the task response execution sequence. Therefore, different determination results of task iteration relationships all have corresponding sequence processing methods.

[0055] Specifically, in this embodiment, by determining the current execution boundary based on the completed and pending operations of associated tasks, adjustments can be made only to the sequence of operations that have not yet been executed without revoking existing execution results. For operations that constitute a substitution relationship, by deleting the substituted or mutually exclusive pending operations and selecting the corresponding target insertion position based on the existence of task dependencies, the new target interaction operation can be placed in a position that conforms to the substitution semantics or dependency order. At the same time, by maintaining the original relative order of the remaining pending operations and synchronously adjusting the candidate tasks after the operation migration is completed, the same interaction operation can be avoided from being repeatedly retained in different tasks. Thus, the merging and rearrangement of candidate tasks and associated tasks can be completed incrementally, protecting the results of completed operations while maintaining the execution order and task dependencies of unaffected operations, thereby ensuring the consistency of the updated task response execution sequence and reducing duplicate execution, disordered order, and unnecessary task rollback.

[0056] In one embodiment, determining and executing a target task in an executable state based on the updated task response execution sequence includes: Get the interface identifier of the currently displayed interface and match it with the interface identifier of the target interface; Among them, the task to be scheduled can be determined according to the task order of the updated task response execution sequence, and the target interface is the target interface associated with the task to be scheduled. The interface identifier can be a page route identifier, a page instance identifier, or a unique identifier assigned by the target interaction environment to different interfaces. By comparing the interface identifier of the currently displayed interface with the interface identifier of the target interface, it can be determined whether the current actual interaction environment is still in the target interface required by the task after the task is generated. If the interface identifier of the currently displayed interface matches the interface identifier of the target interface, then the target interface object is queried in the current displayed interface based on the object identifier of the target interface object. The object identifier can be a UI control identifier, a business resource identifier, or a unique object identifier formed by a combination of UI and resource identifiers; preferably, the UI object that matches the object identifier can be queried through the object tree, control tree, or current UI object set of the currently displayed interface. In response to finding the target interface object in the current display interface, the execution status of the preceding interactive operations corresponding to the current task to be scheduled is obtained; if all the preceding interactive operations are in the completed state, the current task to be scheduled is determined as the target task in the executable state. Among them, the preceding interactive operations include completed interactive operations identified during the process of determining the task iteration relationship, whose operation results are depended on by the currently scheduled task, and may also include interactive operations that are located before the currently scheduled interactive operations according to the internal operation order of the task and must be completed first. If the interface identifier of the currently displayed interface does not match the interface identifier of the target interface, the currently scheduled task can be temporarily suspended and kept in a pending state or set to a waiting state for the interface to recover. If the target interface is detected to become the currently displayed interface again within the preset waiting time, the interface matching can be re-executed. If no match is found after the preset waiting time, the currently scheduled task can be set to an invalid state or the target interface can be determined again. The preset waiting time can be set according to the page switching time of the target interactive environment, for example, set to 2 seconds to 10 seconds.

[0057] If the current display interface matches the target interface but the target interface object is not found, it can be determined that the target interface object does not currently meet the execution conditions, and the current scheduled task will not be executed for the time being. The target interface object can be queried again based on the latest interface status information, or the current scheduled task can be set to an object invalid state after a preset number of queries. If the target interface object is found but at least one prerequisite interaction operation has not been completed, the pending execution state of the current scheduled task will be maintained, and priority will be given to waiting for or scheduling the corresponding prerequisite interaction operation to be completed, rather than executing the current scheduled task in advance. Based on the operation type and operation parameters corresponding to the interactive operation to be executed, execute the interactive operation to be executed on the target interface object in the currently displayed interface, and update the execution status of the target task according to the execution result of the interactive operation to be executed. Specifically, the corresponding operation interface pre-configured in the target interactive environment can be called according to the operation type, and the operation parameters and the object identifier of the target interface object can be used as the interface input. For example, when the operation type is to open, the object open interface is called, and when the operation type is to play, the media playback interface is called. When the operation is executed successfully, the corresponding interactive operation to be executed is set to the completed state. When the operation fails, it can be set to the execution failure state, and the task can be re-executed or the subsequent execution of the task can be stopped according to the preset number of retries. The preset number of retries can be set to 1 to 3 times, or it can be set separately according to different operation types. If all pending interactive operations corresponding to the target task have been completed, the target task is set to the completed state, and the target tasks in the executable state are determined according to the order of tasks following the target task in the task response execution sequence.

[0058] Accordingly, if the target task still has unfinished interactive operations to be executed, the target task remains in an executing state, and the system continues to determine whether the next interactive operation to be executed meets the corresponding execution conditions according to the order of interactive operations within the target task. If the corresponding interactive operation is temporarily unexecutable due to interface changes, object invalidation, or failure to meet preconditions, the execution of the target task can be paused, and its current execution progress can be retained until the corresponding execution conditions are met again. For tasks that have been determined to be invalid or have failed to execute and will not be retried, the corresponding task can be terminated according to the preset exception handling rules, and the system continues to determine whether the subsequent tasks in the task response execution sequence meet the execution conditions.

[0059] Specifically, in this embodiment, before the task is actually executed, the current display interface is matched with the target interface specified by the task, and the target interface object is further queried after the interface is matched. This allows for pre-execution verification of possible interface switching, refresh, or object changes after the task is generated. Furthermore, the task to be scheduled is only determined as executable when all pre-existing interactive operations have been completed, preventing the target interactive operations from being executed prematurely before their dependencies are met. Further operations are performed on the verified target interface object according to the operation type and parameters, and the task status is continuously updated and subsequent tasks are advanced based on the operation results. This forms a closed-loop execution mechanism where the interface, object, pre-existing conditions, and execution status are mutually constrained. This reduces erroneous operations caused by changes in the interactive environment during task waiting, and avoids premature, repeated, or missed execution of tasks, improving the accuracy and reliability of the final task response.

[0060] In another embodiment, when determining whether a target task is in an executable state based on the updated task response execution sequence, the corresponding execution binding method can also be determined based on the degree of dependence of the interactive operation to be executed on the target interface. The execution binding method includes at least the interface-relative execution method and the fixed resource execution method. In response to the fact that the interaction operation to be executed belongs to the interface relative execution mode, the interface identifier of the currently displayed interface is obtained according to the aforementioned embodiment, and the corresponding interaction operation to be executed is executed when the current displayed interface matches the target interface and the target interface object can be queried; Among them, the interface relative execution mode refers to the execution mode in which the correct execution of the interactive operation requires the current display state of the target interface or the current position of the target interface object in the interface as a condition. For example, the operation is performed on the buttons, toggle controls or interface objects that are only valid in the corresponding interface instance on the current page. In response to the fact that the interaction operation to be executed belongs to the fixed resource execution mode, if the candidate interface object already corresponds to a unique standard resource identifier, even if the interface identifier of the currently displayed interface does not match the interface identifier of the target interface, the similar interface object in the currently displayed interface will not be re-identified as the target interface object. Instead, the previously determined candidate interface object, standard resource identifier and current resource availability status will be checked to see if they meet the execution conditions corresponding to the interaction operation to be executed. If the conditions are met, the interactive operation to be executed will be performed according to the specified resource identifier and the operation type and operation parameters corresponding to the interactive operation to be executed; if the conditions are not met, the corresponding operation will not be executed for the time being, and at least one of the following processing will be performed according to the reason for resource unavailability: re-matching, outputting prompt information, or terminating the target task. The fixed resource execution method is applicable when the target interactive operation directly acts on a uniquely identified book, audio, course, or other business resource, and the operation does not require the source interface to remain displayed. The fixed resource execution method can also be used for uniquely identified resources in the historical interface.

[0061] For example, when a target user says "play this book" while a book is displayed on the first screen, and switches to the second screen before the speech recognition is completed; if the "book" has been uniquely identified as the target book on the first screen based on the environmental interface timing information in the request window, and the corresponding specification resource identifier has been determined, then if the playback operation does not require the first screen to continue displaying, the playback operation can be performed after re-verifying that the specification resource can still be played, without rebinding other books currently displayed on the second screen as the target object.

[0062] Specifically, this embodiment selects different pre-execution verification methods based on whether the interactive operation depends on the current state of the source interface. This can avoid incorrectly setting "the current interface still exists" as a necessary execution condition common to all target tasks. At the same time, for fixed resource operations that do not depend on the source interface, the object and resource identity determined at the stage of voice interaction request are still used, thereby avoiding the user from mistakenly replacing the same type of object on the current page with the original referent object after switching pages.

[0063] It is worth noting that in continuous voice interaction scenarios, the interface observation window corresponding to the current voice interaction request may overlap with the asynchronous response period of a prior task. Especially when the target interaction environment experiences network transmission jitter, asynchronous interface rendering, or delayed return of background processing results, the execution result of a prior task may trigger interface switching, interface object refresh, or interface object position change precisely during the period when the target user is describing the target object. In this case, if the environmental interface timing information is generated solely based on the time relationship between the state change moment and the request window, the interface state triggered by a prior task but not yet perceived by the target user may be mistakenly identified as the reference context for the current voice interaction request, or the interface before the change actually referred to by the target user may be excluded from the matching range, leading to a causal misalignment between voice interaction information and interface state information. Therefore, in addressing the technical problem of accurately associating voice requests with dynamic interface contexts, this application also needs to consider how to identify the causal source of interface state changes when asynchronously triggered interface state changes by prior tasks are mixed into the request window, and determine the actual interface timing side corresponding to the target description information.

[0064] Therefore, in a preferred embodiment, after acquiring the corresponding voice interaction information and generating environment interface timing information based on the interface state information of the target interaction environment within the corresponding request window, the method further includes: Obtain task execution event information corresponding to at least one prior task that overlaps with the interface observation window in time. The task execution event information includes at least one of the following: operation initiation time, result feedback time, affected interface, affected interface object, and expected state change. For each interface state change event in the environmental interface timing information, candidate execution events are selected from the task execution event information if the time interval between the event time and the corresponding state change time meets the preset causal time tolerance, and the range of change of the affected interface and / or the affected interface object matches the range of change of the corresponding interface state change event. Based on the temporal proximity, the overlap of the scope of action, and the consistency between the expected state change and the actual state change, the causal correlation degree of each candidate execution event is determined. Candidate execution events that meet the preset causal correlation conditions are identified as target execution events, and the corresponding interface state change events are marked as prior task-induced events; in response to multiple candidate execution events that meet the preset causal correlation conditions, the candidate execution event with the highest causal correlation is identified as the target execution event. Based on the result feedback time corresponding to the target execution event, the interface rendering latency, and the state change time corresponding to the prior task-induced event, determine the causal boundary time that represents the corresponding interface state change that can be perceived by the target user. Obtain the timestamps of the speech frames corresponding to each speech recognition unit that constitutes the target description information in the speech interaction information, and determine the time period in which the target description occurs based on the timestamps of the speech frames; The target description occurrence time, causal boundary time, and temporal reference information in the target description information are matched to determine the interface reference time period corresponding to the target description information. Specifically, if the target description occurrence time is before the causal boundary time and the temporal reference information does not point to the operation result after the interface state change, the interface state duration period before the causal boundary time is determined as the interface reference time period. If the target description occurrence time is after the causal boundary time, or the temporal reference information points to the operation result after the interface state change, the interface state duration period after the causal boundary time is determined as the interface reference time period. In response to the overlap between the target description occurrence time and the causal boundary time, or the causal correlation degree corresponding to the target execution event not meeting the preset deterministic conditions, a first temporal matching branch and a second temporal matching branch are constructed based on the duration of the interface state before and after the causal boundary time, respectively. The target description information is matched with the first and second temporal matching branches, and the target temporal matching branch is determined based on the degree of satisfaction of the object constraint information and the executable state of the corresponding candidate interface object. If only one temporal matching branch meets the preset branch selection conditions, the corresponding temporal matching branch is determined as the target temporal matching branch. If both temporal matching branches meet or do not meet the preset branch selection conditions, interface confirmation information is output to distinguish the interface state before and after the causal boundary time, and the target temporal matching branch is determined based on the confirmation result returned by the target user for the interface confirmation information. In response to no confirmation result, the first and second temporal matching branches are maintained, and the generation of candidate tasks based on the corresponding voice interaction request is paused. Based on the interface reference time period or target time sequence matching branch, the duration period and matching priority of each interface in the environmental interface time sequence information are corrected to obtain the corrected environmental interface time sequence information, and the target interface and corresponding candidate interface objects are determined based on the corrected environmental interface time sequence information. In response to the absence of a prior task-triggered event, the timing information of the environment interface remains unchanged.

[0065] In this embodiment, task execution event information refers to data records formed by the task scheduler, operation executor, or target interactive environment during the initiation, processing, and result return of interactive operations, which can characterize the execution process of interactive operations; expected state change refers to the interface change result predetermined based on the operation type, operation parameters, and interface state before operation execution, including at least one of the following: attribute change of the target interface object, display state change, position change, and interface jump.

[0066] Preferably, the causal correlation can be obtained by normalizing the time proximity, scope overlap, and result consistency to a value range of zero to one, and then weighting and summing them according to their respective fusion weights. Specifically, the time proximity decreases as the time interval between the result feedback time of the candidate execution event and the state change time increases; the scope overlap can be determined based on interface identifier consistency, interface object identifier consistency, and the overlap ratio of the changed area; and the result consistency can be determined based on the consistency ratio between the expected changed state field and the actual changed state field. For example, the fusion weights for time proximity, scope overlap, and result consistency can be set to 0.3, 0.3, and 0.4, respectively, and the preset causal correlation condition can be set to a causal correlation of not less than 0.75. If one item cannot be calculated due to lack of corresponding data, its fusion weight can be proportionally allocated to the remaining calculable items. The aforementioned fusion weights and thresholds can also be statistically determined based on the labeled task execution events and interface state change event samples.

[0067] Preferably, the preset deterministic conditions may include a causal correlation degree not lower than a second correlation threshold, and the difference between the highest and second-highest causal correlation degrees not lower than a preset difference, wherein the second correlation threshold is higher than the first correlation threshold used in the preset causal correlation conditions. For example, the first correlation threshold, the second correlation threshold, and the preset difference can be set to 0.75, 0.85, and 0.10, respectively. The preset branch selection conditions may include an interface matching degree of the corresponding temporal matching branch not lower than a preset interface matching threshold, and at least one candidate interface object in the temporal matching branch being in an executable state and having an object matching degree that satisfies the preset matching conditions; the preset interface matching threshold can be determined based on the interface matching degree distribution of the confirmed target interface, for example, set to the fifth percentile value of the distribution.

[0068] The preset causal time tolerance limits the maximum time range during which task execution events may cause changes in the interface state. It can be set based on the latency distribution from operation initiation or result feedback to interface state changes in historical tasks. For example, the 95th percentile of the latency distribution can be summed with a preset protection duration, which can be set from 100 milliseconds to 500 milliseconds. Interface rendering latency refers to the time elapsed from when the target interactive environment receives the operation result to when the corresponding changed content reaches a displayable state. It can be determined through statistical values ​​of rendering callback time, first frame display time, or historical rendering time.

[0069] The causal boundary moment refers to the time boundary that divides the interface state change induced by the target execution event into the user-perceptible pre-state and user-perceptible post-state; the target description occurrence period refers to the time range between the timestamp of the first voice frame corresponding to the target description information and the timestamp of the last voice frame. The voice frame timestamp and the interface state change moment are preferably recorded using the same monotonic clock; when they come from different acquisition modules, clock deviation correction can be performed based on the timestamp difference corresponding to the common triggering event.

[0070] Specifically, in this embodiment, by introducing task execution event information such as the initiation of prior tasks, result feedback, and expected state changes, and performing causal matching with the actual interface state changes within the request window in three dimensions—time, scope, and result state—it is possible to identify interface mutations induced by the asynchronous return of prior tasks from the interface evolution sequence. Furthermore, based on the relationship between the causal boundary time corrected for rendering latency and the time period of the target description, the actual perceptible interface state when the target user makes an object description can be determined, avoiding treating the interface before and after the change as equivalent simply based on the request window's inclusion relationship. For situations where the voice description spans the interface mutation time or the causal relationship is uncertain, by constructing branch timing information and combining it with object constraints and object executable states for secondary disambiguation, both sides' contexts can be retained when reliable attribution is not possible, rather than making irreversible unilateral discarding. Therefore, the corrected environmental interface timing information can more accurately represent the real interface context of the current voice interaction request, improving the recognition accuracy of the target interface and candidate interface objects, and reducing object misbinding and subsequent incorrect candidate task generation caused by asynchronous responses of prior tasks.

[0071] In one specific embodiment, when the response execution result of the target task includes at least two of the following: text response, resource display content, and interactive operation to be performed, a task response list corresponding to the target task can be established. The task response list is used to record the candidate interface objects and standard resource identifiers corresponding to each response product in the same target task. When multiple response products are used to represent the same logical interaction target, the corresponding multiple response products are associated with the same candidate interface object and / or the same standard resource identifier. Before the response artifacts corresponding to the target task enter the actual response execution, the resource references contained in the text response, the resources corresponding to the resource display content, and the function resources corresponding to the interactive operations to be executed are obtained respectively, and the resources corresponding to each response artifact are compared with the standard resource identifiers recorded in the task response list. If multiple response products expressing the same logical interaction goal correspond to the same specification resource identifier, the corresponding response product is allowed to enter the subsequent response execution; if at least two response products correspond to different specification resource identifiers, the corresponding response products are not submitted for the time being, and a partial update is performed according to the type of the inconsistent response product. For example, when the candidate interface object and the specification resource identifier have been determined, but only the resource name in the text response is inconsistent with the corresponding specification resource, the target interface, candidate interface object, resource verification result, and interactive operation to be executed can be retained, and only the text response can be regenerated; when only the execution interface or operation parameter template corresponding to the operation type changes and the target resource does not change, the candidate interface object and resource verification result can be retained, and only the corresponding interactive operation to be executed can be regenerated. Specifically, by enabling different response products corresponding to the same logical interaction target to share the already determined candidate interface object or standardized resource identity, it is possible to prevent situations where text prompts, resource displays, and actual operations are legal but point to different objects. Furthermore, when a partial response product fails, only the affected part can be regenerated, reducing the repeated execution of the already completed interface object determination and resource verification process.

[0072] For example, after the interactive operation to be executed in the target task completes its response execution, a corresponding execution result record can be generated. The execution result record includes at least one of the following: target task identifier, candidate interface object identifier, operation type, operation parameters, actual target object, and execution status. For interactive operations that act on business resources, the resource identifier that the operation actually acts on can also be recorded, and the resource identifier that the operation actually acts on can be compared with the standard resource identifier determined before the target task is executed. If the two are consistent, the execution status of the target task is updated according to the execution result. If the two are inconsistent, the target task is set to an execution abnormal state, and subsequent tasks that depend on the result of the interactive operation are suspended to prevent erroneous execution results from continuing to propagate along the task response execution sequence. For the same interactive operation that is repeatedly received or repeatedly scheduled due to communication retries, a corresponding operation execution identifier can be generated based on the target task identifier, candidate interface object identifier, operation type, and normalized operation parameters. Before executing the corresponding interactive operation again, it is checked whether the same operation execution identifier that has been successfully executed already exists. If it exists, the repeated execution is suppressed and the corresponding task status is updated directly using the existing execution result.

[0073] The execution result record and operation execution identifier mentioned above can be recorded locally on the terminal or jointly by the terminal and the corresponding server. Their specific storage structure and communication method do not constitute a limitation on this application.

[0074] It should be understood that, although Figure 2The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0075] In one embodiment, such as Figure 3 As shown, a task response execution device is provided, comprising: The information acquisition module is used to respond to a voice interaction request initiated by a target user in a target interactive environment, acquire the corresponding voice interaction information, and generate environmental interface timing information based on the interface state information of the target interactive environment in the corresponding request window. The task generation module is used to parse the voice interaction information to obtain the target interaction operation and target description information, so as to determine the target interface and the corresponding candidate interface object that match the voice interaction request in the target interaction environment according to the target description information and the environmental interface timing information, and generate candidate tasks according to the target interaction operation and the candidate interface object. The associated task determination module is used to add the candidate task to the task response execution sequence corresponding to the target interaction environment, and determine whether there is an associated task in the task response execution sequence whose generation time is earlier than the candidate task and whose response execution has not yet been completed; The iteration relationship determination module is used to lock the subsequent response execution of the associated task if it exists, and to determine the task iteration relationship between the candidate task and the associated task based on the candidate interface objects and interactive operations corresponding to the candidate task and the associated task respectively. A sequence update module is used to adjust the candidate tasks and / or the associated tasks based on the task iteration relationship to update the task response execution sequence; The task execution module is used to determine the target task in the executable state and execute it based on the updated task response execution sequence.

[0076] Specific limitations regarding the task response execution device can be found in the limitations of the task response execution method above, and will not be repeated here. Each module in the aforementioned task response execution device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0077] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores task response execution data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a task response execution method.

[0078] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0079] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described task response execution methods.

[0080] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described task response execution methods.

[0081] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0082] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0083] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A task response execution method, characterized in that, The method includes: In response to a target user initiating a voice interaction request in the target interaction environment, the corresponding voice interaction information is obtained, and environmental interface timing information is generated based on the interface state information of the target interaction environment in the corresponding request window. The voice interaction information is parsed to obtain the target interaction operation and target description information. Based on the target description information and the environmental interface timing information, the target interface and the corresponding candidate interface object that match the voice interaction request in the target interaction environment are determined. Candidate tasks are generated based on the target interaction operation and the candidate interface object. The candidate task is added to the task response execution sequence corresponding to the target interaction environment, and it is determined whether there is an associated task in the task response execution sequence that was generated earlier than the candidate task and has not yet completed the response execution. If they exist, the execution of the subsequent response of the associated task is locked, and the task iteration relationship between the candidate task and the associated task is determined according to the candidate interface objects and interactive operations corresponding to the candidate task and the associated task, respectively. Based on the task iteration relationship, adjust the candidate tasks and / or the associated tasks to update the task response execution sequence; Based on the updated task response execution sequence, the target task in the executable state is determined and executed.

2. The task response execution method according to claim 1, characterized in that, The step of acquiring the corresponding voice interaction information and generating environment interface timing information based on the interface state information of the target interaction environment within the corresponding request window includes: Record the voice data generated when the target user initiates the voice interaction request, and perform voice recognition processing on the voice data to obtain text information corresponding to the voice interaction request as the voice interaction information; Record the start and end times of the voice interaction request respectively, and determine the request window based on the start and end times; Based on the preset pre-time tolerance and post-time tolerance, the start boundary and end boundary of the request window are expanded respectively to obtain the interface observation window; The interface state information generated when the target interactive environment in the interface observation window undergoes at least one interface state change, such as the display, disappearance, position change, or interface switching of an interface object, is obtained, and the state change time corresponding to each interface state information is recorded. According to the state change time, the multiple interface state information are sorted in time sequence, and for the same interface and / or the same interface object, the interface state information representing entering the corresponding interface state is matched in time sequence with the interface state information representing exiting the corresponding interface state to determine the state duration period corresponding to the interface and / or the interface object. The environmental interface timing information is generated based on the duration of the states corresponding to multiple interfaces and / or interface objects and the corresponding timing relationships.

3. The task response execution method according to claim 1, characterized in that, The step of determining the target interface and corresponding candidate interface object that match the voice interaction request in the target interactive environment based on the target description information and the environmental interface timing information includes: Parse the target description information to obtain temporal reference information and object constraint information, wherein the temporal reference information is used to characterize the temporal range of the interface corresponding to the referenced object, and the object constraint information is used to characterize at least one of the resource type, object order and object spatial location of the referenced object; Based on the matching results of the timing reference information and the duration of multiple interfaces in the timing information of the environment interface, as well as the timing relationship between the multiple interfaces, at least one interface to be matched is selected from the multiple interfaces. Extract at least one of the resource type identifier, object sequence index, and object location region corresponding to the interface object contained in the at least one interface to be matched, and generate interface object feature information; The interface object feature information is compared with the object constraint information to determine the resource type sub-matching degree based on the consistency of the resource type identifier, the object order sub-matching degree based on the difference between the object order indices, and the spatial location sub-matching degree based on the positional distance or region overlap between the object location region and the constraint location region represented by the object constraint information. Based on the constraint terms contained in the object constraint information, determine the fusion weights corresponding to the resource type sub-matching degree, the object order sub-matching degree, and the spatial location sub-matching degree, and perform weighted fusion on the corresponding sub-matching degrees to obtain the object matching degree corresponding to the interface object; The interface matching degree corresponding to the interface to be matched is determined based on the object matching degree of the interface objects within the same interface to be matched, and the target interface is determined from the at least one interface to be matched based on the interface matching degree. The interface objects in the target interface whose object matching degree meets the preset matching conditions are determined as the candidate interface objects.

4. The task response execution method according to claim 3, characterized in that, The step of generating candidate tasks based on the target interactive operation and the candidate interface object includes: Determine the corresponding operation type and operation parameters based on the target interactive operation; Based on the object matching degree corresponding to the candidate interface object, the candidate interface objects are filtered and sorted to obtain a candidate object sequence; Based on the interface state information corresponding to the target interface, the executability of the candidate object sequence is determined, and candidate interface objects in the executable state are retained. The retained candidate interface objects are associated and combined with the target interface, the operation type and the operation parameters to generate multiple candidate tasks, wherein the candidate tasks are used to instruct the execution of the target interaction operation on the corresponding candidate interface object on the target interface; The order of the multiple candidate tasks is determined based on the object matching degree.

5. The task response execution method according to claim 1, characterized in that, Determining whether there exists an associated task in the task response execution sequence that was generated earlier than the candidate task and has not yet completed its response execution includes: Obtain the response execution status of multiple tasks in the task response execution sequence, excluding the candidate tasks, and filter out the tasks in the queue that have not yet completed response execution from the multiple tasks in the queue; Obtain the candidate tasks and the target interface and candidate interface objects corresponding to each of the selected tasks in the list; The target interface and candidate interface object corresponding to the candidate task are compared with the target interface and candidate interface object corresponding to each of the listed tasks, respectively. Based on the comparison results, determine the task to be associated from the selected tasks that operates on the same target interface and / or the same candidate interface object as the candidate task. Obtain the generation time corresponding to each of the tasks to be associated, and sort the multiple tasks to be associated in chronological order according to the generation time; Based on the result of the time sequence sorting, the task to be associated with the latest generation time is determined from a plurality of tasks to be associated, and / or the task to be associated with the latest generation time time is determined to satisfy a preset duration condition, and is used as the associated task. Suspend the subsequent response execution of the associated task, and maintain the current response execution state of the associated task in the task response execution sequence.

6. The task response execution method according to claim 1, characterized in that, The step of determining the task iteration relationship between the candidate task and the associated task based on the candidate interface objects and interaction operations corresponding to the candidate task and the associated task respectively includes: Obtain at least one pending interactive operation in the associated tasks that has not yet completed its response execution, and the target interactive operation corresponding to the candidate task; The target interactive operation is compared with each of the interactive operations to be executed; If the target interaction operation and at least one of the interaction operations to be executed correspond to the same candidate interface object, and the two have the same operation type but different operation parameters, then the corresponding interaction operation to be executed is determined as the first operation to be iterated. In response to the target interaction operation and at least one of the interaction operations to be executed corresponding to the same candidate interface object, and the target interaction operation and the interaction operations to be executed having a mutually exclusive execution relationship, the corresponding interaction operation to be executed is determined as the second operation to be iterated. In response to the existence of the first operation to be replaced and / or the second operation to be replaced, it is determined that there is a task substitution relationship between the candidate task and the associated task; Obtain the interface state information required for the target interactive operation corresponding to the candidate task, and determine whether the interface state information includes the operation result generated by the interactive operation that the associated task has completed. If included, the completed interactive operation that produces the operation result is determined as the preceding interactive operation; the operation parameters corresponding to the target interactive operation are matched with the operation result produced by the preceding interactive operation, and when the operation result satisfies the operation parameters corresponding to the target interactive operation, it is determined that there is a task dependency relationship between the candidate task and the associated task. Based on the determination results of the task substitution relationship and the task dependency relationship, the task iteration relationship between the candidate task and the associated task is determined.

7. The task response execution method according to claim 6, characterized in that, The step of adjusting the candidate tasks and / or the associated tasks to update the task response execution sequence based on the task iteration relationship includes: Based on the interactive operations that have been completed and the interactive operations that have not yet been completed in the associated tasks, the current execution boundary of the associated tasks is determined, and the sequence of interactive operations to be executed is determined based on the current execution boundary. In response to the existence of the task substitution relationship between the candidate task and the associated task, the first to be iterated operation and / or the second to be iterated operation are deleted from the sequence of interactive operations to be executed. When there is no task dependency relationship between the candidate task and the associated task, the operation position corresponding to the earliest operation position before deletion of each deleted operation is determined as the target insertion position. In response to the existence of the task dependency relationship between the candidate task and the associated task, the first position to be executed after the current execution boundary is determined as the target insertion position; Insert the target interaction operation corresponding to the candidate task into the target insertion position, and maintain the original relative order between the interaction operations to be executed that have not been deleted in the sequence of interaction operations to be executed, to obtain the updated sequence of interaction operations to be executed. Remove the target interaction operation that has been inserted at the target insertion position from the pending interaction operations corresponding to the candidate task, and remove the candidate task from the task response execution sequence when there are no other pending interaction operations for the candidate task; The updated sequence of interactive operations to be executed is appended to the interactive operations that have been completed in the associated task, and the task response execution sequence is updated according to the adjusted associated task and the candidate task.

8. The task response execution method according to claim 1, characterized in that, The step of determining and executing the target task in an executable state based on the updated task response execution sequence includes: Obtain the interface identifier of the currently displayed interface and match the interface identifier with the interface identifier corresponding to the target interface; In response to the interface identifier of the currently displayed interface matching the interface identifier of the target interface, the target interface object is queried in the currently displayed interface according to the object identifier of the target interface object; In response to finding the target interface object in the current display interface, the execution status of the pre-interaction operation corresponding to the current task to be scheduled is obtained; if all the pre-interaction operations are in the completed state, the current task to be scheduled is determined as the target task in the executable state. Based on the operation type and operation parameters corresponding to the interactive operation to be executed, the interactive operation to be executed is performed on the target interface object in the currently displayed interface, and the execution status corresponding to the target task is updated based on the execution result of the interactive operation to be executed. In response to the completion of all pending interactive operations corresponding to the target task, the target task is set to a completed state, and the target tasks in the executable state are determined according to the order of tasks following the target task in the task response execution sequence.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images