Computer interface interaction control method and device based on visual perception, medium and product

Abstract

The application discloses a visual perception-based computer interface interaction control method and device, a medium and a product, which comprises the following steps: receiving a task instruction, obtaining a current interface image and at least one historical interface image, obtaining historical interaction information, and constructing a time sequence interface state; performing semantic analysis on the interface based on the current interface image to generate a plurality of candidate interaction regions; performing time sequence consistency analysis on each candidate interaction region based on the historical interface image and the historical interaction information, and determining a target candidate region according to the analysis result; positioning the target candidate region to determine a target interaction position, and performing a corresponding interface operation based on the target interaction position; performing consistency evaluation on the interface state after the operation is performed, and correcting the current operation when the evaluation result does not satisfy a preset condition; and repeatedly performing the above steps until the task instruction is completed.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to a computer interface interactive control method, device, medium and product based on visual perception. Background Technology

[0002] With the development of multimodal models, visual language models, and intelligent agent technology, automated interaction based on graphical user interfaces has gradually become an important application direction of artificial intelligence technology. Related technologies typically acquire interface images and combine them with task instructions to identify and understand interface elements, thereby generating operations such as clicking, inputting, scrolling, or dragging to achieve functions such as web browsing, software operation, and cross-application task execution.

[0003] In existing technologies, interface automation methods typically include rule-based methods based on structural information and intelligent methods based on visual understanding. Methods based on structural information rely on the underlying structure of the interface or preset rules, resulting in poor adaptability to interface changes or cross-environment applications. While methods based on visual understanding possess some generalization ability, they still have shortcomings in complex real-world interface scenarios. For example, some technologies analyze only single-frame interface images, making it difficult to effectively utilize changes in the interface over time, thus making them susceptible to the influence of instantaneous interface states. In high-resolution or control-intensive interface environments, the accuracy of target element positioning is insufficient, easily leading to interaction deviations. Furthermore, when operations do not achieve the expected results, these technologies lack effective hierarchical adjustment mechanisms, often resorting to overall replanning, resulting in low processing efficiency. Simultaneously, in real desktop environments, some solutions depend on the underlying structure or specific interfaces, limiting their versatility across multiple applications and environments. Summary of the Invention

[0004] One objective of this application is to provide a computer interface interaction control method, device, medium, and product based on visual perception, at least to solve the problems in the prior art where interface information is difficult to identify stably, interaction positioning accuracy is insufficient in complex interfaces, and the interaction process lacks an effective adjustment mechanism, resulting in low stability and accuracy of automatic interface operation.

[0005] To achieve the above objectives, some embodiments of this application provide the following aspects:

[0006] This application provides a computer interface interaction control method based on visual perception, the method comprising:

[0007] Receive task instructions, and obtain the current interface image and at least one historical interface image, while also obtaining historical interaction information to construct the temporal interface state;

[0008] Based on the current interface image, semantic parsing of the interface is performed to generate multiple candidate interaction areas;

[0009] Based on the historical interface images and historical interaction information, a temporal consistency analysis is performed on each candidate interaction region, and the target candidate region is determined based on the analysis results.

[0010] The target candidate region is located to determine the target interaction position, and the corresponding interface operation is performed based on the target interaction position;

[0011] Perform a consistency evaluation on the interface state after the operation is executed, and correct the current operation if the evaluation result does not meet the preset conditions;

[0012] Repeat the above steps until the task instruction is completed.

[0013] Secondly, some embodiments of this application also provide an electronic device, the electronic device comprising: one or more processors; and a memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method described above.

[0014] Thirdly, some embodiments of this application also provide a computer-readable medium having computer program instructions stored thereon, which can be executed by a processor to implement the steps of the method described above.

[0015] Fourthly, some embodiments of this application also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described above.

[0016] Compared with related technologies, the solution provided in this application introduces a temporal consistency analysis mechanism based on historical interface images and historical interaction information to comprehensively evaluate and rank candidate interaction regions, thereby achieving more accurate target selection. This approach improves the stability and reliability of interactive target recognition even when the interface is dynamically changing, enabling the system to achieve effective interactive control in complex interface environments. Simultaneously, by constructing a closed-loop control mechanism that includes positioning, execution, evaluation, and correction, and dynamically adjusting when the operation does not achieve the expected results, the continuity and robustness of the interaction process are improved. Through these methods, the overall stability and execution efficiency of interface interaction control can be effectively improved, enhancing the system's adaptability in practical application scenarios. Attached Figure Description

[0017] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0018] Figure 1 A flowchart illustrating a computer interface interaction control method based on visual perception, provided as an exemplary embodiment of this disclosure;

[0019] Figure 2 A flowchart of another computer interface interaction control method based on visual perception provided for an exemplary embodiment of this disclosure;

[0020] Figure 3 An exemplary structural diagram of the electronic device provided for some embodiments of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Figure 1 An exemplary flowchart of a computer interface interaction control method based on visual perception is provided for an exemplary embodiment of this disclosure. The method includes:

[0023] S101. Receive task instructions, obtain the current interface image and at least one historical interface image, and obtain historical interaction information to construct the time sequence interface state.

[0024] Specifically, task instructions to be executed can be received through a user input interface or a system task scheduling module. These instructions can be natural language descriptions, templated task descriptions, structured task parameters, or combinations thereof, used to describe the computer interface operation task to be performed. During task execution, the interface image at the current moment can be captured, and historical interface images from several consecutive moments prior to the current moment can be acquired to form a short-term visual sequence. The number of historical interface images can be set according to the actual application scenario, for example, 2 frames, 3 frames, 5 frames, 8 frames, or more, thereby reflecting the changes in the interface over time.

[0025] Simultaneously, historical interaction information can also be acquired. This historical interaction information may include, but is not limited to, historical mouse position trajectories, historical click locations and click types, historical keyboard input content or shortcut key codes, the start and end positions of historical drag operations, historical action execution results, historical success or failure markers, and task progress status. This historical interaction information is used to characterize the user's or system's operational behavior on the interface and its results.

[0026] Based on this, the current interface image, historical interface images, and historical interaction information can be uniformly organized and associated to construct a temporal interface state. This temporal interface state comprehensively represents the changes in the interface over continuous time and the evolution of interactive behavior, providing contextual information support for the subsequent selection, sorting, and interface operation decisions of candidate interaction areas.

[0027] S102. Based on the current interface image, perform semantic parsing on the interface to generate multiple candidate interaction areas.

[0028] Specifically, based on the current interface image, a global semantic analysis of the interface can be performed using a visual encoder, object detection model, multimodal model, or a combination thereof to extract candidate interactive regions. These candidate interactive regions may include buttons, input boxes, menus, icons, tabs, scroll bars, checkboxes, pop-up close controls, confirmation controls, toolbar controls, and other visual element areas that can trigger interaction.

[0029] In one embodiment, at least one or more of the following information can be extracted from the current interface image: text area and corresponding text recognition results, icon area and its semantic description, position and category information of candidate interaction area, cursor position, focus control position, window hierarchy relationship and interface structure information such as modal box, drop-down box or floating layer, so as to obtain the semantic attributes and spatial distribution relationship of each element in the interface.

[0030] Subsequently, the task instructions and interface semantic information can be combined to perform a preliminary relevance assessment on the candidate interaction regions to generate corresponding initial relevance scores, thereby forming a set of candidate target regions. In one embodiment, each candidate interaction region may be accompanied by one or more of the following attribute information: region bounding box coordinates, candidate action type, region category label, text recognition result, icon semantic description, initial confidence level, and window hierarchy information.

[0031] The above method can extract candidate regions with interactive significance from the current interface image, and provide structured input information for subsequent temporal consistency analysis and target region sorting.

[0032] S103. Based on the historical interface images and historical interaction information, perform temporal consistency analysis on each candidate interaction region, and determine the target candidate region based on the analysis results.

[0033] Specifically, the changes in corresponding areas in historical interface images at different points in time, along with the operation processes recorded in historical interaction information, can be used to comprehensively analyze each candidate interaction area. This analysis allows for the evaluation of the candidate interaction areas' stability over time, their matching degree with task instructions, and their correlation with historical interaction processes. Based on this, evaluation results reflecting the priority of each candidate interaction area can be generated. The candidate interaction areas can then be ranked according to these evaluation results to determine the target candidate area for performing the current operation.

[0034] S104. Locate the target candidate region to determine the target interaction position, and perform the corresponding interface operation based on the target interaction position.

[0035] Specifically, a specific interaction location can be determined within the target candidate area. This interaction location can be a coordinate point or a specific position within the area. Subsequently, corresponding operation instructions, such as clicks, inputs, or other interactive behaviors, can be generated based on the target interaction location to perform corresponding operations on the interface. The operation instructions can be sent and executed through the system input interface or an automated execution module to complete the interaction with the target interface elements.

[0036] S105. Perform a consistency evaluation on the interface state after the operation is executed. If the evaluation result does not meet the preset conditions, correct the current operation.

[0037] Specifically, after an operation is performed, the updated interface image can be acquired and compared with the interface state before the operation to determine whether the current operation has achieved the expected effect. The consistency assessment can be based on changes in interface elements, such as changes in position, content, or state. When the assessment result indicates that the operation has not achieved the expected result, the current operation can be corrected, for example, by redetermining the interaction position within the original candidate area, or by re-analyzing and selecting candidate interaction areas, thereby improving the accuracy and effectiveness of subsequent operations.

[0038] S106. Repeat the above steps until the task instruction is completed.

[0039] Specifically, the current interface state can be updated after each operation, and subsequent processing can continue based on the updated interface state. By repeatedly executing the above steps, multiple operation steps corresponding to the task instruction can be completed step by step until the task objective is achieved.

[0040] In the above embodiments, continuous identification and selection of interaction targets can be achieved even when the interface is dynamically changing. A closed-loop control mechanism is formed through the evaluation and correction of operation results, thereby improving the stability and accuracy of the interface interaction process. Simultaneously, by introducing temporal interface states as a decision-making basis, adaptability in complex interface environments can be effectively enhanced, making the interaction control process more reliable.

[0041] In one embodiment, the timing consistency analysis includes:

[0042] Based on the spatial stability of the candidate interaction region in historical interface images, the degree of semantic matching with task instructions, and the degree of correlation with historical interaction information, the candidate interaction region is comprehensively evaluated to obtain a comprehensive evaluation result.

[0043] Based on the comprehensive evaluation results, the candidate interaction regions are sorted to determine the target candidate regions.

[0044] Specifically, for each candidate interaction region, its corresponding location can be located in the historical interface images, and the changes of that region in multiple historical interface images can be analyzed. The spatial stability can be obtained by analyzing the changes in the spatial position of the candidate interaction region at different time points, such as determining whether its position has shifted significantly, whether the region's range has changed, and whether it continues to exist in consecutive historical interface images, thereby reflecting the stability of the candidate interaction region in the time dimension.

[0045] Simultaneously, semantic matching analysis can be performed on candidate interaction regions in conjunction with task instructions. Specifically, semantic information corresponding to candidate interaction regions, such as text information or functional attributes, can be obtained and matched with the target content described by the task instructions to determine the relevance between the candidate interaction region and the current task. In this way, candidate interaction regions that are more relevant to the task objective can be selected.

[0046] Furthermore, historical interaction information can be used to perform correlation analysis on candidate interaction areas. This historical interaction information reflects the operational steps performed during task execution and their corresponding interface changes. By analyzing the relationship between candidate interaction areas and historical operations—such as whether they are located within historical operation paths or are associated with historical operation results—the rationality of candidate interaction areas in the current task can be further evaluated.

[0047] After obtaining the spatial stability, semantic matching degree, and correlation degree mentioned above, the evaluation results can be uniformly processed, such as by normalization or weighted fusion, to form a comprehensive evaluation result for each candidate interaction region. Based on the comprehensive evaluation result, multiple candidate interaction regions can be ranked, and the candidate interaction region with the higher score can be selected as the target candidate region, thereby providing a basis for subsequent interaction operations.

[0048] Through the above implementation methods, the candidate interaction area can be evaluated by comprehensively considering the time dimension, semantic information and historical interaction process, thereby improving the accuracy and stability of target selection and enabling the system to achieve more reliable interactive control in complex interface environments.

[0049] In one embodiment, the spatial stability is determined based on the positional offset, size change, and number of consecutive frames of the candidate interaction region in the historical interface image.

[0050] The degree of semantic matching is determined based on the degree of matching between the task instruction and the text information or icon semantic information corresponding to the candidate interaction area.

[0051] The degree of correlation is determined based on the spatial relationship or operation sequence between the operation trajectory recorded in the historical interaction information and the candidate interaction area.

[0052] Specifically, regarding spatial stability, candidate interaction regions can be matched and tracked across multiple historical interface images to obtain their position and size information at different time points, and their positional offset and size changes between consecutive interface images can be calculated. Simultaneously, the number of frames in which the candidate interaction region appears in consecutive historical interface images can be counted to reflect its continuous existence over time. Based on this, the spatial stability of the candidate interaction region can be characterized by combining the positional offset, size changes, and the number of consecutive frames, thereby distinguishing between stable regions and transiently changing regions.

[0053] Regarding the semantic matching degree, textual or icon semantic information corresponding to the candidate interaction areas can be obtained and matched with the task instructions. Specifically, the semantic content expressed by the candidate interaction areas can be compared with the task instructions based on methods such as text similarity calculation, semantic tag matching, or semantic vector representation to determine the degree of matching. During this process, indicators for characterizing semantic relevance can be generated based on the matching results, thereby providing a basis for subsequent comprehensive evaluation.

[0054] Regarding the degree of correlation, the relationship between candidate interaction areas and historical operations can be analyzed based on the operation trajectories and sequences recorded in historical interaction information. Specifically, the spatial location of the candidate interaction area in the interface and its relationship with the historical operation path can be determined, such as whether it is located in the continuation direction of the historical operation path or whether it is located in an interface area related to an already executed operation. Simultaneously, the rationality of the candidate interaction area in the operation sequence can be analyzed, such as whether it conforms to the logical flow of task execution, thereby determining the degree of correlation between it and the historical interaction process.

[0055] After obtaining spatial stability, semantic matching degree, and correlation degree, these indicators can be used as inputs to comprehensively evaluate candidate interaction regions, thereby providing a more accurate basis for subsequent ranking and target selection. Through this method, candidate interaction regions can be analyzed in detail across multiple dimensions, improving the reliability of target recognition and the accuracy of interaction decisions.

[0056] Furthermore, in one embodiment, spatial stability scores can be performed separately. Visual semantic matching score and historical correlation ratings Normalization is performed, followed by weighted summation based on preset weights to obtain the target time series score. The normalization process can employ linear normalization, interval mapping, or standardization based on historical sample distribution to reduce the impact of differences in different scoring units on the final ranking results.

[0057] Specifically, for the candidate interaction regions generated in step S102, target selection is not directly based on single-frame results. Instead, cross-frame matching is performed on each candidate interaction region by combining the current interface image with continuous historical interface images to form corresponding temporal trajectories. The cross-frame matching can be based on at least one of the following: intersection-over-union ratio (IoU), center point distance, region category consistency, OCR text consistency, or icon semantic consistency. When a candidate interaction region cannot maintain stable matching in adjacent historical frames, its subsequent score can be reduced or it can be removed, thereby reducing the interference of unstable regions on the decision-making results.

[0058] Based on this, a target temporal scoring function can be constructed for any candidate interaction region i. This is used to comprehensively evaluate the performance of the candidate interaction region in terms of time, semantics, and historical behavior. The target temporal scoring function can be expressed as:

[0059]

[0060] in, This represents the spatial stability score of the candidate interaction region. This represents the visual semantic matching score between the candidate interaction area and the current task. The score represents the correlation between the candidate interaction area and the historical successful operation trajectory. α, β, and γ are the corresponding weight coefficients, which can be set or dynamically adjusted according to the task type, interface complexity, or historical execution effect.

[0061] The space stability score To measure the stability of candidate interaction regions in continuous interface images, one or more of the following metrics can be used for calculation: the mean cross-over ratio (COR) of the candidate interaction region over several consecutive frames, the displacement variance of the candidate interaction region's center coordinates over consecutive frames, the rate of change of the candidate interaction region's width or height, the change in the area of ​​the candidate interaction region, the fluctuation of the candidate interaction region's aspect ratio, the number of consecutively visible frames of the candidate interaction region, and the frequency with which the candidate interaction region is occluded or disappears.

[0062] In one implementation, a high spatial stability score can be assigned when a candidate interaction region maintains a high cross-over ratio, a small center displacement, and a low size change in consecutive frames; conversely, a lower spatial stability score is assigned when a candidate interaction region appears only briefly or its position changes significantly.

[0063] The visual semantic matching score To measure the semantic relevance between candidate interaction regions and task instructions, calculations can be performed based on one or more of the following information: the similarity between the text representation of the task instruction and the local image representation of the candidate interaction region, the semantic similarity between the task instruction text and the OCR-recognized text of the candidate interaction region, the correspondence between the icon semantic labels of the candidate interaction region and the task keywords, and the semantic matching degree between the interface context area where the candidate interaction region is located and the task target.

[0064] In one embodiment, task instructions can be converted into task vectors, and image features, text information, and semantic labels of candidate interaction regions can be converted into region semantic vectors. By calculating cosine similarity or other matching indicators, a visual semantic matching score can be obtained.

[0065] The historical correlation score To measure the degree of correlation between candidate interaction areas and historical successful interaction trajectories, calculations can be performed based on one or more of the following information: the consistency of the current candidate interaction area and the historical successful target area in terms of spatial location; the consistency of the operation sequence in the current operation stage and the historical successful path; the consistency of the operation type corresponding to the current candidate interaction area and the historical operation type; the consistency of the interface level where the candidate interaction area is located and the level where the historical successful target is located; and the adjacency or logical relationship between the candidate interaction area and the previous operation target.

[0066] In one implementation, a higher historical association score can be assigned when the candidate interaction area is consistent with the historical successful path in both spatial location and operation sequence.

[0067] Obtain the target temporal score corresponding to each candidate interaction region. Then, the candidate interaction regions can be reordered based on the scores. Priority is given to candidate interaction regions that are stable over time, highly semantically related to task instructions, and exhibit strong consistency in historical behavior paths as target candidate regions.

[0068] In one implementation, the top few candidate interactive areas with the highest scores can be selected for subsequent processing. The number of candidates can be set according to the complexity of the interface or the availability of computing resources.

[0069] Through the above implementation methods, candidate interaction areas can be comprehensively evaluated from multiple dimensions based on multi-frame information and historical interaction information, thereby effectively reducing the impact of dynamic interface changes, short-term interference or visual confusion on target selection and improving the accuracy and stability of the interaction control process.

[0070] In one embodiment, locating the target candidate region to determine the target interaction location includes:

[0071] Obtain the bounding box of the target candidate region, and crop the corresponding interface region based on the bounding box to obtain the cropped region;

[0072] The cropped area is subjected to resolution enhancement and / or multi-scale scaling processing;

[0073] The target interaction location is determined based on the processed image.

[0074] Specifically, based on the target candidate region determined in step S103, the bounding box information of that region in the current interface image can be obtained first. The bounding box may include the coordinates of the top-left corner of the candidate region, as well as parameters such as width and height. Based on this, the current interface image can be cropped using the bounding box to obtain a local image region containing the target candidate region. To avoid missing target information due to inaccurate candidate region boundaries or slight offsets of interface elements, the bounding box can be appropriately expanded during the cropping process to include the contextual information surrounding the candidate region.

[0075] After obtaining the cropped region, resolution enhancement or multi-scale scaling can be applied to it. Specifically, image magnification, interpolation resampling, or multi-scale image pyramid construction can be used to transform the cropped region at different scales, thereby improving the discernibility of fine-grained features in the local area. In one embodiment, images at different scales can be processed separately, and the analysis results at each scale can be combined to improve positioning accuracy.

[0076] Subsequently, the target interaction location can be determined based on the processed image. Specifically, the target interaction element can be further analyzed within the cropped area, such as identifying the location of key structures, the center of text, or the location of significant feature points within the area, and thereby determining the specific interaction location for performing the operation. The target interaction location can be a single coordinate point or a region location that meets preset conditions.

[0077] In one embodiment, the interactive positions obtained at different scales can also be fused, for example, by combining multiple candidate positions using a weighted method to obtain the final target interactive position. This approach allows for more precise positioning within the candidate region, thereby improving the accuracy of subsequent interface operations.

[0078] Through the above implementation methods, local fine analysis can be performed based on the candidate region, which can effectively improve the accuracy of interaction position determination, reduce positioning errors caused by resolution limitations or complex interface details, and thus further improve the reliability of interface interaction control.

[0079] Furthermore, in one embodiment, the localization of the target candidate region to determine the target interaction location can be achieved using a hierarchical local fine localization strategy.

[0080] Specifically, for the candidate interaction regions ranked higher in step S103, global interaction coordinates are not directly output. Instead, a phased local fine-tuning positioning process is performed based on the target candidate regions.

[0081] First, for high-priority candidate regions, their bounding boxes in the current interface image can be obtained, and these bounding boxes can be expanded outward according to preset rules to form a local search region. The expansion rules can include fixed pixel value expansion, expansion proportional to the candidate box width and height, or a combination of both. In one embodiment, expansion can be performed at 10% to 40% of the candidate box width and height, or a minimum and maximum expansion pixel value can be set. Alternatively, the expansion ratio can be adaptively set according to the control type of the candidate region; for example, a larger expansion ratio is used for icon-type controls, and a smaller expansion ratio is used for text input controls. Furthermore, when the candidate region is located at the edge of the interface, boundary constraints can be used to limit the expansion range to prevent the cropped area from exceeding the interface range. By expanding the candidate region, contextual information surrounding the target, such as label text, adjacent controls, boundary outlines, or menu hierarchy, can be preserved in the local search region, thereby improving the accuracy of subsequent fine-grained positioning.

[0082] After obtaining the local search region, image enhancement and multi-scale resampling processing can be performed on the region to improve the discriminability of fine-grained targets within the local area. The enhancement processing may include resolution enhancement, interpolation-based image magnification, multi-scale scaling, contrast enhancement, sharpening, text region enhancement, and sub-region hierarchical sampling. In one embodiment, local images at different scales, such as 1x, 2x, and 4x scales, can be generated and input into a local analysis process to obtain multiple candidate fine-localization results. Subsequently, the interaction positions obtained at each scale can be mapped back to the local search region coordinate system according to the corresponding scaling ratio, and combined with the positional relationship of the local search region in the original interface image, the candidate interaction coordinates in the original interface can be calculated. For the candidate interaction coordinates obtained at multiple scales, the final interaction position can be determined based on methods such as coordinate clustering, consensus voting, confidence-weighted averaging, or reprojection error.

[0083] Furthermore, secondary visual analysis can be performed on the enhanced local image to obtain more precise interaction locations. This secondary visual analysis can be implemented in various ways, such as performing fine-grained object detection, keypoint prediction, or image segmentation on the local image. It can also combine local text recognition results with image texture information for joint analysis, or generate local interaction coordinates based on a visual language model. Through this secondary analysis process, precise interaction information can be output, such as click location coordinates, drag start and end positions, text input focus position, scroll area position, and interaction positions of menu sub-items.

[0084] In one embodiment, a hierarchical localization process can be adopted: first, candidate interaction regions are generated in the global interface image; second, candidate regions with higher priority are selected based on the temporal consistency analysis results; then, the boundaries of the candidate regions are expanded to form local search regions; next, image enhancement or multi-scale processing is performed on the local search regions; subsequently, secondary visual analysis is performed in the local image; finally, the precise interaction position is output. By dividing the target localization process into two stages—global candidate screening and local fine localization—it can effectively adapt to high-resolution interfaces, densely populated control areas, and scenarios involving small targets, thereby improving the accuracy and stability of interactive localization.

[0085] Through the above implementation methods, a local search region construction mechanism based on boundary expansion is introduced when locating target candidate regions. This mechanism can retain necessary contextual information around the candidate regions, thereby avoiding the semantic information loss problem caused by relying solely on the original candidate boxes. Combined with resolution enhancement and multi-scale resampling processing, the recognizability of small targets and fine-grained interface elements can be significantly improved, enabling stable and accurate localization results even in high-resolution interfaces or scenes with dense controls. Furthermore, by fusing localization results at different scales, the error impact of a single scale can be further reduced, improving overall localization accuracy.

[0086] Furthermore, by performing secondary visual analysis within a local area and employing a hierarchical localization strategy, the target localization process is divided into a two-stage process of "global candidate screening + local fine-tuning localization." This effectively reduces the interference of global noise on the localization results and enhances the adaptability to complex interface structures. Compared to performing one-time localization on the entire interface image, the above method improves the accuracy and robustness of interactive position determination while ensuring computational efficiency, thereby enhancing the stability and reliability of the entire interface interaction control process.

[0087] In one embodiment, the consensus assessment includes:

[0088] Perform a difference analysis on the interface images before and after the operation, and determine whether the operation result meets the preset conditions based on changes in the position of interface elements, changes in text content, or changes in interface state.

[0089] Specifically, an interface image can be acquired before the operation is performed, and an interface image can be acquired after the operation is completed. Based on the interface images before and after the operation, difference analysis can be performed to identify the areas in the interface that have changed and the types of changes. The difference analysis can be achieved through pixel-level comparison, structural feature comparison, or semantic information comparison, thereby obtaining interface change information.

[0090] In one embodiment, changes in the position of interface elements can be analyzed, such as determining whether the target area has undergone expected displacement or whether the interface layout has changed; changes in text content can also be analyzed, such as identifying whether the text has been updated, whether the expected text has appeared, or whether the input operation has been completed; simultaneously, changes in interface state can be analyzed, such as determining whether a control has changed from an unselected state to a selected state, whether a button has changed from an unavailable state to an available state, or whether the interface has transitioned. By comprehensively judging the above-mentioned change information, it can be determined whether the current operation has met the preset conditions.

[0091] In one implementation, corresponding preset conditions can be set for different types of operations. For example, for click operations, whether the target interface successfully navigates or whether the state of the target control changes can be used as the judgment condition; for input operations, whether the input content is displayed correctly can be used as the judgment condition; and for scrolling operations, whether the interface content is shifted can be used as the judgment condition. By associating operation types with corresponding judgment rules, the accuracy of consistency evaluation can be improved.

[0092] In a further implementation, the difference analysis results can be quantified, for example, by generating an evaluation index representing the degree of interface change, and comparing the evaluation index with a preset threshold to determine whether the operation result meets preset conditions. When the evaluation result meets the preset conditions, the current operation can be determined to have been successfully executed; when the evaluation result does not meet the preset conditions, subsequent correction steps can be triggered.

[0093] Through the above implementation methods, the results of interface operations can be effectively judged, and whether the operation has achieved the expected effect can be identified in a timely manner, thereby providing a basis for subsequent operation adjustments and improving the accuracy and reliability of the entire interactive control process.

[0094] Furthermore, in one embodiment, an action result consistency score can be constructed. This is used for a comprehensive quantitative evaluation of the operation results. The consistency score of the action results can be expressed as:

[0095]

[0096] in, This indicates the consistency score for interface changes. Indicates accessibility score, The score indicates the progress of the task. , , The weighting coefficient can be set or dynamically adjusted according to the task type or execution stage.

[0097] Specifically, the interface change consistency score To measure whether the interface changes after an operation meet expectations, calculations can be based on one or more of the following metrics: whether the target control is activated, whether the new page or window loads successfully, whether the menu is expanded, whether the specified text is successfully entered into the input box, whether the interface focus shifts to the target location, whether an error message box appears, and whether the interface experiences unexpected jumps or becomes unresponsive. A higher consistency score is assigned when the interface changes meet expectations; conversely, a lower score is assigned.

[0098] The reachability score This measure assesses the accessibility of a target element or elements required for subsequent operations after an operation has been performed. It can be calculated based on one or more of the following metrics: whether the target area is within the current visible area, whether the target control is not obscured, whether the target control is interactive, and whether key interface elements required for subsequent tasks are displayed. A higher accessibility score is assigned when the target element is visible and interactive.

[0099] The task progress score To measure whether the current operation is advancing the task towards its target state, the following factors can be used for evaluation: whether the current interface state is closer to the task completion state, whether the preset task stage transition conditions are met, and whether the blocking conditions in the previous operation step have been eliminated. An operation that effectively advances the task execution process can be assigned a higher progress score.

[0100] In one embodiment, the above scores can be normalized and then weighted and fused based on preset weights to obtain the final action result consistency score. Subsequently, the score can be compared with a preset threshold. When the score meets the threshold condition, the current operation is determined to be successful; when the score is lower than the threshold, the operation is determined to have failed to achieve the expected effect, and a subsequent operation correction process is triggered.

[0101] Through the above implementation methods, the operation results can be comprehensively evaluated from multiple dimensions such as interface changes, element accessibility, and task progress, thereby achieving accurate judgment of interface interaction effects and improving the stability and reliability of the system in complex interaction scenarios.

[0102] In one embodiment, performing the corresponding interface operation based on the target interaction location includes:

[0103] An interactive action sequence is generated based on the target interaction location, task instructions, and current interface state. The interactive action sequence includes at least an action type, an interaction location, and execution parameters.

[0104] The interactive action sequence is then mapped to control instructions that can be executed by the operating system or automated execution module.

[0105] Specifically, based on the spatial coordinates of the target interaction location within the interface, combined with the operation objective described in the task instruction and the current interface state, an action sequence for performing interface interactions can be generated. These interaction actions can include mouse movement, single-click, double-click, right-click, drag, scroll, keyboard input, or combination of shortcut keys, etc. Different types of actions can be combined according to task requirements to form a complete operation flow.

[0106] In one embodiment, a single interactive action can be represented in a structured manner, with each interactive action including at least an action type, an interaction location, and execution parameters. The action type identifies the specific operation form, such as a click or input; the interaction location indicates the spatial position where the action is executed; and the execution parameters may include information such as keyboard input content, action duration, action interval, or waiting time. Based on this, multiple interactive actions can be organized according to the task execution order to form an ordered sequence of interactive actions.

[0107] Furthermore, corresponding control commands can be generated based on the interactive action sequence. These control commands can conform to the calling specifications of the operating system input interface or automated execution module, for example, by being converted into mouse events, keyboard events, or other control signals, thereby driving the interface to perform corresponding operations. In one embodiment, the interactive action sequence can be mapped to a standard input event sequence or automated script commands and sent to the target application through the system interface.

[0108] In one embodiment, the execution parameters in the interactive action sequence can also be adjusted, such as setting the waiting time or execution rhythm between actions to adapt to different interface response speeds, thereby improving the stability of operation execution. Furthermore, the action sequence can be dynamically adjusted based on the current interface state, for example, by extending the waiting time or re-triggering some operations when the interface does not respond promptly.

[0109] Through the above implementation methods, abstract interaction goals can be transformed into executable action sequences, and standardized control commands can drive the interface to complete operations, thereby achieving precise control over the interface interaction process. Furthermore, by flexibly configuring the action sequences and execution parameters, the system's adaptability and execution reliability in different interface environments can be improved.

[0110] In one embodiment, the correction of the current operation further includes:

[0111] Prioritize redetermining the target interaction location;

[0112] If the preset conditions are still not met after redetermining the target interaction location, the target candidate region is redetermined.

[0113] If the operation result still does not meet the preset conditions after redetermining the target candidate region, the interface operation method is adjusted.

[0114] Perform a recovery operation when the interface state is abnormal.

[0115] Specifically, when the consistency evaluation result in step S105 indicates that the current operation does not meet the preset conditions, the interaction position can be re-determined within the already determined target candidate area. This re-determination of the interaction position can be based on a re-analysis of the target candidate area using the current interface image. For example, the candidate interaction point positions can be recalculated within this area, or the original interaction position can be adjusted in conjunction with the updated interface state to obtain a new target interaction position. By correcting the position within the same candidate area, operational accuracy can be improved and computational overhead reduced without changing the candidate area selection result.

[0116] In one embodiment, the target candidate region can be re-localized based on the updated interface image, for example, by re-executing the localization process or refining the existing localization results, thereby obtaining the updated target interaction location. Subsequently, interface operations can be performed based on the updated target interaction location, and the corresponding execution results can be obtained.

[0117] If the result still does not meet the preset conditions after position correction within the same candidate area, the candidate interaction area can be further redefined. Specifically, the analysis process of the candidate interaction area can be re-executed, or the candidate area set can be updated based on the current interface state to select a new target candidate area. Subsequently, positioning and interface operation steps can be performed on the new target candidate area to adjust the interaction path.

[0118] Furthermore, in one embodiment, the correction of the current operation further includes:

[0119] After redetermining the target interaction location, the updated interface state is re-evaluated for consistency.

[0120] If the preset conditions are still not met after re-evaluation, the candidate interaction areas are reordered, and the target candidate area and target interaction position are re-determined based on the reordering results.

[0121] Specifically, after redefining the target interaction location and executing the corresponding interface operation, an updated interface image can be obtained, and a consistency evaluation process can be performed based on this interface state. The re-evaluation can employ the same judgment method as the initial evaluation to analyze interface changes and determine whether the current operation meets preset conditions. By introducing a re-evaluation mechanism, the corrected operation results can be effectively verified, thereby avoiding misjudgments caused by insufficient single correction.

[0122] If the consistency evaluation results still do not meet the preset conditions, a re-sorting process for candidate interaction regions can be triggered. Specifically, based on the current interface image, updated historical interface information, and recent interaction results, a new temporal consistency analysis can be performed on the candidate interaction regions to update the evaluation results of each candidate region and generate a new ranking result. On this basis, the target candidate region can be re-determined according to the updated ranking result.

[0123] Subsequently, the localization process can be re-executed based on the new target candidate region to determine the corresponding target interaction location, and interface operations can be performed based on that location. Through this process, a dynamic transition from local correction to global decision update can be achieved, enabling the system to adjust the target selection strategy in real time based on the operation results.

[0124] In one embodiment, the aforementioned re-evaluation and candidate region reordering process can form a closed-loop mechanism, progressively optimizing target selection and interaction paths through multiple iterations until the operation result meets preset conditions or reaches a preset upper limit for the number of adjustments. This approach improves the adaptability and execution reliability of interactive control in complex interface environments.

[0125] Furthermore, in one embodiment, when the consistency assessment result indicates that the current target candidate region is reasonable, but the operation execution result does not meet expectations, action type adjustment processing can be performed.

[0126] Specifically, in step S104, an interactive action sequence can be regenerated based on the target interactive position to adjust the action type, execution order, or execution parameters, thereby making the executed interface operation more consistent with the current interface state. After the adjusted action is executed, step S105 can be executed again for consistency evaluation.

[0127] Furthermore, in one embodiment, when the interface state becomes abnormal, such as displaying an error message, becoming unresponsive, or entering an unexpected interface, a recovery operation can be performed. The recovery operation can be implemented in the interface operation execution phase corresponding to step S104, such as performing a return operation, closing a pop-up window, triggering a preset shortcut key, or reactivating the interface window, to restore the interface to a state where the task can continue to be performed. After completing the recovery operation, steps S103 to S105 can be re-executed based on the updated interface state.

[0128] In one embodiment, the above-described hierarchical error correction process can be executed step by step in a preset order, and step S105 is performed for consistency evaluation after each level of error correction. The error correction process terminates when the evaluation result meets preset conditions; otherwise, it proceeds to the next level of error correction strategy. In this way, the correction range can be gradually expanded while ensuring computational efficiency.

[0129] Through the above implementation methods, different types of errors, such as local positioning errors, candidate region selection deviations, and improper action execution, can be processed in layers, thereby avoiding unnecessary overall replanning operations, reducing computational overhead, and improving the stability and robustness of the interface interaction control process.

[0130] Furthermore, in one embodiment, after completing the interface interaction corresponding to the task instruction, the interface state information, candidate interaction area sorting result, target candidate area selection result, target interaction position, interaction action sequence and consistency evaluation result during the interaction process are recorded in the interaction trajectory pool.

[0131] Based on the historical records in the interaction trajectory pool, at least one parameter in the subsequent interaction process is updated. The parameters include weight parameters in the temporal consistency analysis, candidate interaction area screening threshold, target candidate area positioning related parameters, interaction action execution parameters, or operation correction strategy parameters.

[0132] Specifically, after completing the interaction process corresponding to the task instruction described in step S106, relevant information in this round of interaction can be recorded and updated to achieve the accumulation of historical trajectories and adaptive optimization of control strategies.

[0133] Information such as interface state, candidate interaction area sorting, target candidate area selection, target interaction location, executed interaction action sequence, and action result consistency score involved in this round of operation can be uniformly recorded and stored in a pre-built interaction trajectory pool. Simultaneously, result information generated during the operation can also be recorded, such as whether the operation was successful, failure type, and task progress. The recorded content in the trajectory pool may include, but is not limited to, the current task category, current interface type, candidate area sorting result, finally selected target candidate area, precise interaction coordinates, actual executed action sequence, action execution result marker, failure reason category, post-operation consistency score, and the distribution of subsequent valid candidate areas.

[0134] In one implementation, relevant parameters in subsequent steps can be updated and optimized based on historical success cases accumulated in the interaction trajectory pool. For example, the weight parameters involved in the temporal consistency analysis in step S103 can be adjusted according to historical execution results to optimize the influence of different evaluation factors in the sorting process; the screening threshold for candidate interaction areas can also be dynamically adjusted to improve the effectiveness of candidate area screening. In addition, the boundary expansion ratio involved in local positioning in step S104 can be adaptively adjusted according to the historical performance of different types of controls, and the scaling factor in the multi-scale processing process can be optimized.

[0135] Furthermore, parameters in the hierarchical error correction strategy can be adjusted based on historical execution data. For example, the retry threshold for local relocation and the execution priority of recovery operations under different interface states can be updated to improve the system's response efficiency in abnormal situations. In one embodiment, the above parameters can be classified and managed according to task category, interface type, or application scenario to achieve differentiated strategy configuration for different scenarios.

[0136] Through the above implementation methods, historical experience can be continuously accumulated during multiple rounds of interaction, and the system control strategy can be continuously optimized based on historical trajectories. This allows the system to gradually adapt to different interface styles, resolutions, and application types, thereby improving its cross-scenario generalization ability and overall interactive control performance.

[0137] Furthermore, in one embodiment, combined with Figure 2 As shown, the method disclosed herein can be executed according to a closed-loop control process to achieve iterative processing between interface understanding, target decision-making, operation execution, and result feedback.

[0138] Specifically, at the beginning of the task, task input and interface state acquisition are performed, receiving task instructions and acquiring the current interface image, historical interface images, and historical interaction information to construct a temporal interface state. Subsequently, the global understanding and candidate region generation stage is entered, where semantic parsing of the current interface image is performed to identify and generate multiple candidate interaction regions.

[0139] Based on this, temporal scoring and re-ranking processing are performed. Temporal consistency analysis is conducted on candidate interaction regions based on historical interface images and historical interaction information, and the candidate interaction regions are ranked according to the analysis results to determine the target candidate region. Next, the local region cropping and fine-tuning stage begins. Fine-tuning is performed within the target candidate region to determine the target interaction position, and the corresponding interface operation is generated and executed based on this position.

[0140] After the operation is completed, the post-action feedback verification phase begins. This phase assesses the consistency of the interface state after the operation to determine if the current operation achieved the expected result. If the assessment result meets preset conditions, the current task step is considered successfully executed, and the historical trajectory update phase begins, recording and updating the current interaction process. If the assessment result does not meet preset conditions, the operation correction process begins, implementing corresponding adjustment strategies based on the failure type, thus forming a closed-loop control mechanism.

[0141] In one specific application embodiment, the method disclosed herein can be applied to high-resolution professional software interface scenarios. In such interfaces, toolbar buttons are small, controls are densely arranged, and there is considerable visual interference. If the interaction position is determined solely based on a single global frame image, click misalignment or erroneous operations are likely to occur. In this embodiment, the system first generates multiple candidate interaction regions based on the current interface image, and combines a preset number of recent historical interface images and historical interaction information to perform temporal consistency analysis on each candidate interaction region to obtain the corresponding ranking results. For target candidate regions with higher ranking priority, their bounding boxes can be expanded outward according to a preset ratio to form a local search region. Subsequently, multi-scale images are generated for the local search region, and a local parsing process is performed on each scale image to obtain multiple candidate interaction positions. Finally, the target interaction position is determined based on the consistency of the positioning results at different scales. Through the above method, even if the target control occupies a small proportion in the global interface, a more accurate positioning result can still be obtained after local enhancement, thereby improving the accuracy of interactive operations.

[0142] In another specific embodiment, the method disclosed herein can be applied to dynamic menu expansion scenarios. In this scenario, the menu controls in the interface undergo an expansion process, and a single frame image is insufficient to accurately reflect the stable state. In this embodiment, the system acquires continuous interface images before and after performing menu-related operations, and performs temporal consistency analysis on candidate interaction areas based on the interface images and historical interaction information. By using indicators such as spatial stability, it distinguishes between transiently changing states and stable presentation states, thereby improving the accuracy of identifying sub-menu targets. When the interface change after the initial operation does not meet the preset conditions, the system prioritizes re-performing the positioning process within the current target candidate area to determine the new target interaction position, rather than directly reselecting the entire candidate area. This approach reduces misjudgments caused by dynamic interface changes and improves the stability of menu-related operations.

[0143] In another specific embodiment, the method disclosed herein can be applied to cross-window task scenarios. In this scenario, users need to switch between multiple application interfaces and complete continuous operations. After performing a window switching operation, the system can acquire an updated interface image and determine whether the current interface state meets preset conditions based on consistency evaluation. When the evaluation result indicates that the target window has not switched correctly, the system prioritizes re-performing positioning processing near the current target candidate area to correct possible click deviations; when local correction still does not achieve the expected effect, it can return to the candidate interaction area analysis stage, reorder the candidate areas related to multiple window switching, and redetermine the target candidate area and interaction position. If necessary, a restorative operation can also be performed to eliminate abnormal interface states. Through the above-mentioned hierarchical processing method, the cumulative impact of erroneous interface states on subsequent operations can be avoided, thereby improving the reliability of cross-window task execution.

[0144] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0145] Furthermore, some embodiments of this application also provide an electronic device. The electronic device can be various forms of digital computer, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device can also be various forms of mobile devices, such as cellular phones, smartphones, wearable devices, and other similar computing devices.

[0146] The electronic device includes: one or more processors; and a memory storing computer program instructions that, when executed, cause the processor to perform the steps of the methods provided in any one or more of the above embodiments. Figure 3 An exemplary structural diagram of the electronic device is disclosed. The electronic device includes one or more processors 1101, a memory 1102, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations. The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0147] The electronic device may further include an input device 1103 and an output device 1104. The processor 1101, memory 1102, input device 1103 and output device 1104 may be connected by a bus or other means, as shown in the figure, which is connected by a bus.

[0148] Input device 1103 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1104 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display, a light-emitting diode display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0149] To provide interaction with the user, the electronic device can be a computer. The computer has: a display device (e.g., a cathode ray tube or LCD monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback); and input from the user can be received in any form (e.g., voice input or tactile input).

[0150] In this embodiment, a computer-readable medium stores a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in any one or more of the above embodiments. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into that device. The aforementioned computer-readable medium carries one or more computer-readable instructions.

[0151] The memory 1102 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 1102, thereby implementing the program instructions / modules corresponding to the methods provided in any one or more of the embodiments described above in this application.

[0152] The memory 1102 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 1102 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1102 may optionally include memory remotely located relative to the processor 1101, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0153] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0154] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technologies, read-only optical discs, digital versatile optical discs or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0155] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0156] In the above embodiments, all or part of the implementation can be achieved through software, hardware, firmware, or any combination thereof. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the above steps or functions. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. In addition, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0157] The computer program product provided in this application includes one or more computer programs / instructions. When executed by a processor, these computer programs / instructions generate, in whole or in part, the processes or functions described in this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0158] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0159] The scope of this application is defined by the appended claims rather than the foregoing description, and is therefore intended to encompass all variations falling within the meaning and scope of equivalents of the claims. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural.

[0160] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily made by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims, and the above embodiments should be regarded as exemplary and non-limiting.

Claims

1. A computer interface interaction control method based on visual perception, characterized in that, The method includes: Receive task instructions, and obtain the current interface image and at least one historical interface image, while also obtaining historical interaction information to construct the temporal interface state; Based on the current interface image, semantic parsing of the interface is performed to generate multiple candidate interaction areas; Based on the historical interface images and historical interaction information, a temporal consistency analysis is performed on each candidate interaction region, and the target candidate region is determined based on the analysis results. The target candidate region is located to determine the target interaction position, and the corresponding interface operation is performed based on the target interaction position; Perform a consistency evaluation on the interface state after the operation is executed, and correct the current operation if the evaluation result does not meet the preset conditions; Repeat the above steps until the task instruction is completed.

2. The method according to claim 1, characterized in that, The time-series consistency analysis includes: Based on the spatial stability of the candidate interaction region in historical interface images, the degree of semantic matching with task instructions, and the degree of correlation with historical interaction information, the candidate interaction region is comprehensively evaluated to obtain a comprehensive evaluation result. Based on the comprehensive evaluation results, the candidate interaction regions are sorted to determine the target candidate regions.

3. The method according to claim 2, characterized in that, The spatial stability is determined based on the positional offset, size change, and number of consecutive frames of the candidate interaction region in the historical interface image. The degree of semantic matching is determined based on the degree of matching between the task instruction and the text information or icon semantic information corresponding to the candidate interaction area. The degree of correlation is determined based on the spatial relationship or operation sequence between the operation trajectory recorded in the historical interaction information and the candidate interaction area.

4. The method according to claim 1, characterized in that, The step of locating the target candidate region to determine the target interaction location includes: Obtain the bounding box of the target candidate region, and crop the corresponding interface region based on the bounding box to obtain the cropped region; The cropped area is subjected to resolution enhancement and / or multi-scale scaling processing; The target interaction location is determined based on the processed image.

5. The method according to claim 1, characterized in that, The consistency assessment includes: Perform a difference analysis on the interface images before and after the operation, and determine whether the operation result meets the preset conditions based on changes in the position of interface elements, changes in text content, or changes in interface state.

6. The method according to claim 1, characterized in that, The step of performing the corresponding interface operation based on the target interaction location includes: An interactive action sequence is generated based on the target interaction location, task instructions, and current interface state. The interactive action sequence includes at least an action type, an interaction location, and execution parameters. The interactive action sequence is then mapped to control instructions that can be executed by the operating system or automated execution module.

7. The method according to claim 1, characterized in that, The correction of the current operation includes: The target interaction position is redefined; If the preset conditions are still not met after redetermining the target interaction location, the target candidate region is redetermined. If the operation result still does not meet the preset conditions after redetermining the target candidate region, the interface operation method is adjusted. Perform a recovery operation when the interface state is abnormal.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; and A memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images