Pointer intelligent iterative positioning method and system combined with multi-modal large language model
By combining a multimodal large language model for semantic parsing and interface coherence judgment, the problems of pointer positioning and interaction coherence in dynamic interfaces are solved, and the accuracy and robustness of pointer operations are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUANGDONG SHUNLI TECH CO LTD
- Filing Date
- 2026-04-29
- Publication Date
- 2026-05-29
Smart Images
Figure CN122111291A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of pointer positioning technology, specifically to a pointer intelligent iterative positioning method and system that combines a multimodal large language model. Background Technology
[0002] With the rapid development of artificial intelligence technology, human-computer interaction methods are becoming increasingly diversified. Natural language, as an intuitive and convenient interaction method, is being used more and more widely in intelligent control systems. Among them, pointer positioning and operation technology based on natural language commands aims to allow users to control the screen pointer to complete operations such as clicking, dragging, and selecting using everyday language, greatly improving the user-friendliness and efficiency of human-computer interaction.
[0003] However, existing technologies still face many challenges in handling semantic understanding of complex natural language instructions, accurate pointer positioning in dynamically changing interfaces, and contextual coherence in multi-turn interactions. Traditional pointer positioning methods often rely on predefined templates or rules, which have limited ability to parse ambiguous or complex natural language instructions with complex spatial relationships, making it difficult to accurately understand the user's true intentions. When faced with frequent updates to interface elements, window switching, or dynamic content loading, changes in the screen image may lead to pointer positioning deviations or failures, and there is a lack of effective interface coherence judgment mechanisms. Summary of the Invention
[0004] This application provides a pointer intelligent iterative positioning method and system that combines a multimodal large language model, which solves the technical problems of limited semantic understanding of complex natural language instructions, easy deviation in pointer positioning under dynamic interfaces, and lack of an effective interface coherence discrimination mechanism in the prior art.
[0005] The technical solution to the above-mentioned technical problems in this application is as follows: Firstly, this application provides a pointer-based intelligent iterative localization method combining a multimodal large language model, the method comprising: The task structure is obtained by semantically parsing the user's natural language instructions using a multimodal large language model. In response to the task structure, screen images and historical execution information are periodically acquired using the sliding window method to obtain a series of screen images and a series of trajectory summaries; The screen image series is analyzed to extract interface layout features, and interface continuity is determined based on the interface layout features. If the coherence determination result is coherent, then the task structure, the screen image series, and the trajectory summary series are input into the multimodal large language model for alignment determination and pointer movement instruction generation. Execute pointer movement instructions, iteratively acquire screen images to update pointer position until alignment is determined, generate pointer operation instructions based on the task structure and execute them.
[0006] Secondly, this application provides a pointer-based intelligent iterative localization system that combines a multimodal large language model, including: The semantic parsing module is used to perform semantic parsing on the acquired user's natural language instructions using a multimodal large language model to obtain the task structure; The data processing module is used to respond to the task structure by periodically acquiring screen images and historical execution information using the sliding window method to obtain a series of screen images and a series of trajectory summaries. The feature extraction module is used to parse the series of screen images, extract interface layout features, and determine interface coherence based on the interface layout features. The instruction generation module is used to input the task structure, the screen image series, and the trajectory summary series into the multimodal large language model if the coherence discrimination result is coherent, and to perform alignment discrimination and pointer movement instruction generation. The instruction execution module is used to execute pointer movement instructions, iteratively acquire screen images to update the pointer position until the alignment judgment result is aligned, and generate pointer operation instructions according to the task structure and execute them.
[0007] This application provides one or more technical solutions, which have at least the following technical effects or advantages: This application provides a pointer intelligent iterative positioning method and system that combines a multimodal large language model. First, it performs deep semantic parsing on the user-input natural language commands, extracting core elements such as the operation object, the type of action to be executed, and the spatial constraints between the operation objects, thereby constructing a structured task structure. This effectively solves the problem of insufficient understanding of complex, ambiguous, or unclear commands by traditional methods. Second, when responding to the task structure, it periodically collects screen images and historical execution information using a sliding window method, forming a series of screen images and a series of trajectory summaries. This helps the multimodal large language model to more comprehensively understand the execution context of the current task. Third, it parses the series of screen images, extracts the binary layout features of interactive elements, and uses this to determine interface continuity, accurately judging whether the interface is in a continuous state. Finally, it inputs the task structure, the series of screen images, and the series of trajectory summaries into the multimodal large language model to generate pointer movement commands, achieving intelligent pointer navigation. Finally, the generated pointer movement instructions are executed, and the pointer position is continuously updated by iteratively acquiring screen images until the multimodal large language model determines that the pointer has accurately aligned with the target. The final pointer operation instructions are generated and executed according to the task structure, so that the pointer can gradually approach the target.
[0008] Through the above technical solutions, this application achieves accurate semantic parsing based on a multimodal large language model, stable pointer positioning under dynamic interfaces, and ensures interactive coherence, thereby improving the accuracy and robustness of natural language-driven pointer operations. Attached Figure Description
[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0010] Figure 1 This is a flowchart illustrating the pointer-based intelligent iterative localization method combining a multimodal large language model provided in an embodiment of this application. Figure 2 This is a schematic diagram of the structure of the pointer intelligent iterative positioning system that combines a multimodal large language model, as provided in the embodiments of this application.
[0011] The components represented by each number in the attached diagram are explained below: Semantic parsing module 11, data processing module 12, feature extraction module 13, instruction generation module 14, instruction execution module 15. Detailed Implementation
[0012] This application provides a pointer intelligent iterative positioning method and system that combines a multimodal large language model to address the technical problems of limited semantic understanding of complex natural language instructions, easy deviation in pointer positioning under dynamic interfaces, and lack of effective interface coherence discrimination mechanism in the prior art.
[0013] Example 1, as Figure 1 As shown, embodiments of this application provide a pointer-based intelligent iterative localization method combining a multimodal large language model, including: S10: Semantically analyze the acquired user's natural language instructions using a multimodal large language model to obtain the task structure; The task structure serves as a structured task description, and the description items of the task structure at least include the operation object, the type of action to be performed, and the spatial constraint relationship of the operation object.
[0014] In this embodiment, the user's natural language instructions typically contain unstructured text information. Directly using these instructions to drive pointer positioning and operations results in semantic ambiguity and unclear intent. By using a multimodal large language model to perform semantic parsing, the logical relationships of the contextual information in the instructions can be understood, transforming the natural language instructions into a structured task structure that the computer can understand and execute.
[0015] For example, when a user enters "click the settings button in the upper right corner of the page, and then drag the brightness slider to 50%", the multimodal large language model can accurately identify "settings button" as the primary operation object, "click" as the action to be performed, "brightness slider" as the secondary operation object, and "drag to 50%" as the corresponding action type and specific parameters to be performed, and further sort out the order between "clicking the settings button" and "dragging the brightness slider".
[0016] S20: In response to the task structure, periodically collect screen images and historical execution information using the sliding window method to obtain a series of screen images and a series of trajectory summaries; In this embodiment of the application, after responding to the task structure, a sliding window method is used to collect data in order to achieve dynamic tracking of the pointer operation process.
[0017] The screen image series consists of screen images from multiple consecutive acquisition cycles arranged in chronological order, used to reflect the dynamic changes of the interface; the trajectory summary series is a structured extraction of historical execution information, including the coordinates of key points on the pointer's historical movement path, the types of actions executed, the execution timestamps of each action, and the status feedback information generated during the operation.
[0018] Specifically, step S20 in the method includes: Based on a preset acquisition period, screen images are acquired using the sliding window method to obtain a series of screen images including the current screen image frame and N consecutive preceding screen image frames, where N is greater than or equal to 1. Combining the sliding window method, starting from the current moment, backtrack to extract the first M items of pre-execution information, and output the historical execution information, where M is greater than or equal to N; Based on the historical execution information, the pointer's epoch movement information and epoch click information are extracted, and serialized based on the timestamp of the prior execution information to output a structured trajectory summary series.
[0019] In this embodiment, the acquisition period is first determined, for example, set to 100 milliseconds / frame, and the window size of the sliding window is set to 5 frames, i.e. N=5. Then, at each acquisition moment, the current screen image frame and the previous 4 consecutive screen images are acquired to form a series of screen images containing 5 frames. This can ensure the granularity of capturing dynamic changes in the interface without increasing the processing burden due to excessive data volume.
[0020] Secondly, for the collection of historical execution information, the value of M is set to be greater than or equal to N. For example, M=10, that is, starting from the current moment, the first 10 items of pre-execution information are retrieved back. The pre-execution information includes all operation records of the pointer in the past period.
[0021] Next, the historical execution information is processed to extract pointer epoch movement information, such as the direction and distance of pointer movement within each time unit, as well as epoch click information, such as the coordinate position of the click and the type of click, such as left click, right click, etc.
[0022] Then, according to the timestamp order of the pre-execution information, the extracted information is serialized to form a structured trajectory summary series, which reflects the historical trajectory and operation details of the pointer operation.
[0023] S30: Analyze the series of screen images, extract interface layout features, and determine interface coherence based on the interface layout features. In this embodiment, to determine whether the interface is in a stable state during pointer operation and to avoid positioning deviations caused by sudden interface changes, a series of screen images are analyzed. Then, interface layout features are extracted, and the degree of change in the interface layout is quantified by calculating the cosine similarity of the interface layout feature vectors corresponding to adjacent frame images.
[0024] Specifically, step S30 in the method includes: Extract the binarized layout features of interactive elements in the current screen image frame and the previous screen image frame; Based on the binarized layout features, calculate the typical change indicators between the current screen image frame and the preceding screen image frame; When the typical change indicator does not exceed the first preset threshold, the determination interface remains continuous and subsequent steps are executed.
[0025] In this embodiment, the current screen image frame and the preceding screen image frame are first preprocessed, including grayscale conversion, noise reduction and other operations, to highlight the outline of the interactive elements.
[0026] Then, an object detection algorithm is used to identify interactive elements in the image, such as buttons, input boxes, sliders, etc., and to determine their position coordinates and circumscribed rectangular area in the image coordinate system.
[0027] Next, based on the positional information of the interactive elements, a binary layout matrix is constructed. Each element in the matrix represents whether an interactive element exists in the corresponding image region; a value of 1 indicates its presence, and a value of 0 indicates its absence. This forms a digital representation of the interface layout features. Typical change indicators can be obtained by calculating the cosine similarity between the binary layout matrices of the current screen image frame and the preceding screen image frames. Typical change indicators include at least one of the following: structural similarity index, area change rate, mean square error, and cosine similarity.
[0028] For example, when cosine similarity is used as a typical change indicator, if the calculated cosine similarity value is greater than or equal to the first preset threshold of 0.95, it is determined that the interface layout has not changed significantly and is in a continuous state; conversely, if the cosine similarity value is less than the first preset threshold, it is determined that the interface has undergone a sudden change, such as window switching, pop-up window, or page refresh. At this time, the interface re-initialization process will be triggered to re-collect the screen image series and historical execution information to ensure the accuracy of subsequent pointer positioning.
[0029] Furthermore, parsing the screen image series, extracting interface layout features, and determining interface coherence based on the interface layout features also includes: When the typical change indicator exceeds the first preset threshold but does not exceed the second threshold, the detection feature points are selected in the current screen image frame and the preceding screen image frame according to the binarized layout features and in combination with the preset feature point rules. Based on the determined detection feature points, the feature descriptors of the current screen image frame and the previous screen image frame are calculated respectively to obtain the first feature matrix and the second feature matrix, wherein the feature descriptor is the feature point spacing; Based on the first feature matrix and the second feature matrix, multi-point matching is performed on the current screen image frame and the preceding screen image frame, and the geometric transformation matrix is obtained according to the multi-point matching result; The current screen image frame and the preceding screen image frame are aligned according to the geometric transformation matrix, and the overlap area ratio is calculated based on the image alignment result. If the overlapping area ratio is greater than the empirical area ratio based on confidence level, the output coherence judgment result is coherent; otherwise, the output coherence judgment result is incoherent.
[0030] In this embodiment, firstly, when the typical change indicator exceeds the first preset threshold but does not exceed the second threshold, it indicates that the interface may have undergone local changes or slight overall adjustments, rather than drastic abrupt changes. At this time, more refined feature point matching and geometric transformation analysis are performed to determine the continuity of the interface.
[0031] Preset feature point rules can include prioritizing the selection of key corner points of interactive elements as detection feature points. For example, for a rectangular button, its four vertices can be selected as detection feature points; for a circular slider, its center and two intersection points with the slider track can be selected as detection feature points.
[0032] Next, based on the determined detection feature points, the feature descriptor for each feature point in the current screen image frame and the preceding screen image frames is calculated. Specifically, the feature descriptor uses the feature point spacing, i.e., the Euclidean distance between each feature point and all other feature points in the image is calculated and organized into a vector in a specific order to form the feature descriptor for that feature point. Then, the feature descriptors of all feature points in the current frame are combined into a first feature matrix, and the feature descriptors of all feature points in the preceding frames are combined into a second feature matrix.
[0033] Subsequently, based on the first and second feature matrices, multi-point matching is performed between the current screen image frame and the preceding screen image frames. Algorithms such as the FLANN matcher can be used to find the most similar feature point in the second feature matrix for each feature point in the current frame as a matching point pair. To improve matching accuracy, a distance threshold can be set to filter out high-quality matching point pairs; for example, the distance threshold can be set to a ratio of the nearest neighbor distance to the second nearest neighbor distance of less than 0.8.
[0034] Furthermore, based on the filtered multi-point matching results, for example, a random sampling consensus algorithm is used to estimate the geometric transformation matrix between two frames of images. This matrix can be an affine transformation matrix or a perspective transformation matrix, used to describe the spatial transformation relationship of the current frame relative to the preceding frame, such as rotation, translation, scaling, or projection.
[0035] After obtaining the geometric transformation matrix, the matrix is used to perform geometric transformation on the current screen image frame or the preceding screen image frame to align the two frames. Then, the area of the overlapping region of the two aligned frames is calculated and divided by the area of the smaller image to obtain the overlap area ratio.
[0036] If the calculated overlap area ratio is greater than the empirical area ratio based on confidence level (which can be obtained from a large amount of experimental data, for example, if it is set to 0.7), it is considered that although there are certain changes in the interface, the core area still remains coherent, and the coherence judgment result is coherent; otherwise, it is determined that the interface coherence has been broken, the coherence judgment result is discontinuous, and the interface re-initialization process is triggered.
[0037] Specifically, extracting the binarized layout features of interactive elements in the current screen image frame and the preceding screen image frame also includes: Perform interface element detection on the current screen image frame and the preceding screen image frame respectively, and identify the main interactive elements; For each of the main interactive elements, a bounding rectangle region is determined, and within the bounding rectangle region, an edge detection method and a preset envelope constraint are combined to determine the element envelope of each main interactive element. Based on the envelopes of each element, a corresponding binarized layout image is generated on the blank image, and the output is the binarized layout feature, wherein the pixels within the element envelopes are assigned a first pixel value, and the background area of the blank image is assigned a second pixel value.
[0038] In this embodiment, a deep learning-based object detection model, such as YOLOv5, is first used to detect interface elements in the current screen image frame and the preceding screen image frames. This model has learned the features of a large number of common interface interaction elements during the training phase, such as buttons, input boxes, drop-down menus, sliders, checkboxes, radio buttons, tabs, icons, etc., and can accurately identify the categories of the main interactive elements and their approximate positions in the image.
[0039] Secondly, for each identified primary interactive element, its bounding rectangle region is determined based on its bounding box coordinates, and edge information is extracted using the Canny edge detection algorithm. Simultaneously, pre-defined envelope constraints are introduced. For example, for buttons with regular shapes, the envelope constraint can be a minimum bounding rectangle or a minimum bounding ellipse; for text input boxes, the envelope constraint can be a rectangle encompassing all text display areas; for irregularly shaped icons, their element envelope is determined by the connectivity and closure of edge pixels. By combining edge detection with envelope constraints, potentially interfering pixels within the bounding rectangle region are eliminated, determining the element envelope of each primary interactive element. This element envelope is a precise depiction of the area actually occupied by the interactive element.
[0040] Then, based on the element envelopes of each major interactive element, a blank image of the same size as the original screen image is created. On this blank image, all pixels within the area covered by each element envelope are assigned a first pixel value, such as 255, representing white, while the background area of the blank image, i.e., the area not covered by any element envelope, is assigned a second pixel value, such as 0, representing black. The resulting binarized image presents the layout position and shape outline of all major interactive elements in the interface in a black-and-white contrast, thus outputting a binarized layout feature for subsequent interface coherence determination.
[0041] S40: If the coherence determination result is coherent, then the task structure, the screen image series and the trajectory summary series are input into the multimodal large language model to perform alignment determination and pointer movement instruction generation; In this embodiment of the application, if the coherence discrimination result is coherent, it indicates that the current interface is in a stable state. At this time, the task structure, screen image series and trajectory summary series are used as inputs and jointly input to the multimodal large language model to realize alignment discrimination and pointer movement instruction generation.
[0042] Furthermore, the multimodal large language model, with its comprehensive understanding of text, images, and sequence information, aligns and distinguishes the objects to be operated on in the task structure with the interface elements in the series of screen images.
[0043] Specifically, step S40 in the method includes: Extract the current position coordinates of the pointer and the current screen image frame from the series of screen images; The current position coordinates, the current screen image frame, the task structure, and the trajectory summary series are encoded into a multimodal input. The multimodal large language model is invoked to analyze and process the multimodal input to obtain a structured alignment discrimination result, wherein the alignment discrimination result includes at least the alignment Boolean value and the movement command.
[0044] In this embodiment of the application, firstly, the pointer in the image coordinate system is obtained as the current position coordinate by the pointer detection algorithm of the current screen image frame in the screen image series, and at the same time, the current screen image frame is extracted as the visual input.
[0045] Secondly, the current position coordinates are represented as coordinate vectors, the current screen image frame is converted into image feature vectors through the image encoding module, the task structure is parsed into a text description sequence and converted into a text feature vector, and the trajectory summary series is presented in the form of time series feature vectors.
[0046] Subsequently, the feature vectors of different modalities are fused to form a multimodal input vector of a unified dimension, ensuring that the multimodal large language model can simultaneously process location, image, task target and historical trajectory information.
[0047] Furthermore, when calling the multimodal large language model, the model first processes the multimodal input through a cross-modal attention mechanism, semantically associating the task objective described in the text with the interface elements in the image, and combining the historical operation patterns in the trajectory summary series to determine whether the current pointer position is aligned with the operation object specified in the task structure.
[0048] If the alignment Boolean value is true, it indicates that no pointer movement is required, and the model outputs an empty movement command. If the alignment Boolean value is false, the model generates a structured movement command containing the movement direction, movement distance, and target coordinates based on the current position coordinates, the coordinates of the target element in the image, and the optimal path planning of the historical movement trajectory. For example, "move 200 pixels to the right of the screen to coordinates (850, 420)" to achieve precise pointer positioning and operation guidance.
[0049] Furthermore, before encoding the current position coordinates, the current screen image frame, the task structure, and the trajectory summary series into a multimodal input, the process also includes: Based on the task structure, extract the operation object; Obtain historical interaction records, and traverse the historical interaction records using the operation object as the index to perform dual position extraction, and obtain dual position extraction results, wherein the dual position extraction results include the operation object's edge appearance boundary and the operation object's click boundary; Based on the dual-position extraction results, and combined with statistical analysis methods, the typical ratio of the apparent boundary of the operation object to the click boundary of the operation object is determined as the position correction coefficient. The position correction coefficient is added to the task structure and associated with the operation object for storage.
[0050] In this embodiment of the application, firstly, the specific object to be operated is parsed from the task structure. For example, the "save button" in "click the 'save' button" is the object to be operated.
[0051] Secondly, historical interaction records are obtained. These records contain screen image data and pointer click coordinates from past interactions with similar or identical objects. Using the extracted objects as indexes, the historical interaction records are traversed, performing dual-location extraction: the apparent boundary of the object's edges refers to the visual boundary of the object detected by image recognition algorithms in the screen image, typically defined by the edge coordinates of its circumscribed rectangle; the click boundary of the object is the effective area boundary formed by the set of pointer coordinates when the user actually clicked the object in historical interactions. For example, for a rectangular button, its apparent boundary might be the visible rectangular outline of the button, while the click boundary might be the central area of the button that the user typically clicks or a specific area.
[0052] Then, based on the dual-position extraction results, statistical analysis methods are used, such as calculating the offset between the geometric center of the apparent boundary of the operation object and the geometric center of the click boundary, or analyzing the distribution pattern of the click coordinates within the apparent boundary of the operation object, to determine the typical ratio between the apparent boundary of the operation object and the click boundary of the operation object. This ratio is then used as a position correction coefficient. For example, if statistics show that when a user clicks a certain type of button, the center of the actual click position is usually located 10% above the center of the apparent boundary of the operation object, then this typical ratio can be used for subsequent correction of the target coordinates.
[0053] For example, when the object being operated on is a button with a text label, the Y-axis coordinate of the user's click position shown in the historical interaction record is generally 5% lower than the Y-axis coordinate of the geometric center of the edge appearance boundary, meaning that the center of the click boundary is offset downwards by 5% relative to the center of the edge appearance boundary. In this case, 5% of the edge appearance boundary height is calculated as a vertical correction value, and this correction value, together with the typical ratio, is used as a position correction coefficient, which is stored in the task structure and associated with this type of button object.
[0054] Finally, the calculated position correction coefficient is associated with the operation object in the task structure and stored so that when generating pointer movement instructions, the coordinates of the target element can be fine-tuned in combination with the coefficient, so that the pointer is finally positioned in a click area that is more in line with the user's operating habits, thereby further improving the accuracy of pointer positioning and the convenience of operation.
[0055] S50: Execute the pointer movement instruction, iteratively acquire screen images to update the pointer position until the alignment judgment result is aligned, generate pointer operation instructions according to the task structure and execute them.
[0056] The pointer operation instructions are generated based on the operation object, the type of action to be executed, and the position correction coefficient in the task structure.
[0057] In this embodiment, firstly, based on the generated pointer movement instructions, the pointer is driven to move on the screen via the underlying input control interface. During the pointer movement, the current screen image frame is iteratively acquired at preset time intervals, such as every 100 milliseconds, and the pointer's current position coordinates are updated in real time using a pointer detection algorithm.
[0058] Secondly, the updated position coordinates are compared with the target coordinates in the movement command. At the same time, the multimodal large language model is called again to perform alignment determination on the new screen image frame, the current position coordinates, and the task structure. If the alignment determination result is "aligned", the pointer movement stops; if it is not aligned, the movement command is adjusted according to the new position deviation, and the pointer continues to move until the pointer is positioned in the effective area of the target operation object.
[0059] Once the pointers are aligned, specific pointer operation instructions are generated based on the operation object parsed from the task structure, the type of action to be executed, and the previously associated stored position correction coefficients.
[0060] For example, if the task structure indicates "click the 'Save' button", and the position correction factor indicates that the optimal click position of such a button is 10% above the center of its apparent boundary, the generated pointer operation instruction will include the correction of the target coordinates, and simulate the corresponding mouse or touch operation through the input control interface, such as sending the signal of pressing and releasing the left mouse button to the corrected target coordinates, thereby completing the precise execution of the operation specified by the task structure.
[0061] In summary, compared with existing technologies, this application achieves intelligent iterative optimization of pointer positioning by combining the comprehensive understanding and reasoning capabilities of multimodal large language models for text, image, and trajectory information.
[0062] In summary, the embodiments of this application have at least the following technical effects: This application provides a pointer intelligent iterative positioning method combining a multimodal large language model. First, it performs deep semantic parsing on the user-input natural language commands, extracting core elements such as the operation object, the type of action to be executed, and the spatial constraints between the operation objects, thereby constructing a structured task structure. This effectively solves the problem of insufficient understanding of complex, ambiguous, or unclear commands by traditional methods. Second, when responding to the task structure, it periodically collects screen images and historical execution information using a sliding window method, forming a series of screen images and a series of trajectory summaries. This helps the multimodal large language model to more comprehensively understand the execution context of the current task. Third, it parses the series of screen images, extracts the binary layout features of interactive elements, and uses this to determine interface continuity, accurately judging whether the interface is in a continuous state. Finally, it inputs the task structure, the series of screen images, and the series of trajectory summaries into the multimodal large language model to generate pointer movement commands, achieving intelligent pointer navigation. Finally, the generated pointer movement instructions are executed, and the pointer position is continuously updated by iteratively acquiring screen images until the multimodal large language model determines that the pointer has accurately aligned with the target. The final pointer operation instructions are generated and executed according to the task structure, so that the pointer can gradually approach the target.
[0063] Through the above technical solutions, this application achieves accurate semantic parsing based on a multimodal large language model, stable pointer positioning under dynamic interfaces, and ensures interactive coherence, thereby improving the accuracy and robustness of natural language-driven pointer operations.
[0064] Example 2, as Figure 2As shown, based on the same inventive concept as the pointer intelligent iterative localization method combining a multimodal large language model provided in Embodiment 1, this application also provides a pointer intelligent iterative localization system combining a multimodal large language model, including: Semantic parsing module 11 is used to perform semantic parsing on the acquired user's natural language instructions through a multimodal large language model to obtain the task structure; Data processing module 12 is used to respond to the task structure by periodically acquiring screen images and historical execution information using the sliding window method to obtain a series of screen images and a series of trajectory summaries. The feature extraction module 13 is used to parse the series of screen images, extract interface layout features, and determine interface coherence based on the interface layout features. The instruction generation module 14 is used to input the task structure, the screen image series and the trajectory summary series into the multimodal large language model if the coherence discrimination result is coherent, and to perform alignment discrimination and pointer movement instruction generation. The instruction execution module 15 is used to execute pointer movement instructions, iteratively acquire screen images to update the pointer position until the alignment judgment result is aligned, generate pointer operation instructions according to the task structure and execute them.
[0065] Furthermore, in one embodiment of the application, the task structure is used as a structured task description, and the description items of the task structure at least include the operation object, the type of action to be performed, and the spatial constraint relationship of the operation object.
[0066] In one embodiment, the data processing module 12 is specifically used for: Based on a preset acquisition period, screen images are acquired using the sliding window method to obtain a series of screen images including the current screen image frame and N consecutive preceding screen image frames, where N is greater than or equal to 1. Combining the sliding window method, starting from the current moment, backtrack to extract the first M items of pre-execution information, and output the historical execution information, where M is greater than or equal to N; Based on the historical execution information, the pointer's epoch movement information and epoch click information are extracted, and serialized based on the timestamp of the prior execution information to output a structured trajectory summary series.
[0067] In one embodiment, the feature extraction module 13 is specifically used for: Extract the binarized layout features of interactive elements in the current screen image frame and the previous screen image frame; Based on the binarized layout features, calculate the typical change indicators between the current screen image frame and the preceding screen image frame; When the typical change indicator does not exceed the first preset threshold, the determination interface remains continuous and subsequent steps are executed.
[0068] Furthermore, parsing the screen image series, extracting interface layout features, and determining interface coherence based on the interface layout features also includes: When the typical change indicator exceeds the first preset threshold but does not exceed the second threshold, the detection feature points are selected in the current screen image frame and the preceding screen image frame according to the binarized layout features and in combination with the preset feature point rules. Based on the determined detection feature points, the feature descriptors of the current screen image frame and the previous screen image frame are calculated respectively to obtain the first feature matrix and the second feature matrix, wherein the feature descriptor is the feature point spacing; Based on the first feature matrix and the second feature matrix, multi-point matching is performed on the current screen image frame and the preceding screen image frame, and the geometric transformation matrix is obtained according to the multi-point matching result; The current screen image frame and the preceding screen image frame are aligned according to the geometric transformation matrix, and the overlap area ratio is calculated based on the image alignment result. If the overlapping area ratio is greater than the empirical area ratio based on confidence level, the output coherence judgment result is coherent; otherwise, the output coherence judgment result is incoherent.
[0069] Furthermore, in one embodiment, extracting the binarized layout features of interactive elements in the current screen image frame and the preceding screen image frame further includes: Perform interface element detection on the current screen image frame and the preceding screen image frame respectively, and identify the main interactive elements; For each of the main interactive elements, a bounding rectangle region is determined, and within the bounding rectangle region, an edge detection method and a preset envelope constraint are combined to determine the element envelope of each main interactive element. Based on the envelopes of each element, a corresponding binarized layout image is generated on the blank image, and the output is the binarized layout feature, wherein the pixels within the element envelopes are assigned a first pixel value, and the background area of the blank image is assigned a second pixel value.
[0070] Furthermore, the instruction generation module 14 is specifically used for: Extract the current position coordinates of the pointer and the current screen image frame from the series of screen images; The current position coordinates, the current screen image frame, the task structure, and the trajectory summary series are encoded into a multimodal input. The multimodal large language model is invoked to analyze and process the multimodal input to obtain a structured alignment discrimination result, wherein the alignment discrimination result includes at least the alignment Boolean value and the movement command.
[0071] Furthermore, in one embodiment, before encoding the current position coordinates, the current screen image frame, the task structure, and the trajectory summary series into a multimodal input, the method further includes: Based on the task structure, extract the operation object; Obtain historical interaction records, and traverse the historical interaction records using the operation object as the index to perform dual position extraction, and obtain dual position extraction results, wherein the dual position extraction results include the operation object's edge appearance boundary and the operation object's click boundary; Based on the dual-position extraction results, and combined with statistical analysis methods, the typical ratio of the apparent boundary of the operation object to the click boundary of the operation object is determined as the position correction coefficient. The position correction coefficient is added to the task structure and associated with the operation object for storage.
[0072] Furthermore, the pointer operation instructions are generated based on the operation object, the type of action to be executed, and the position correction coefficient in the task structure.
Claims
1. A pointer-based intelligent iterative localization method combining a multimodal large language model, characterized in that: include: The task structure is obtained by semantically parsing the user's natural language instructions using a multimodal large language model. In response to the task structure, screen images and historical execution information are periodically acquired using the sliding window method to obtain a series of screen images and a series of trajectory summaries; The screen image series is analyzed to extract interface layout features, and interface continuity is determined based on the interface layout features. If the coherence determination result is coherent, then the task structure, the screen image series, and the trajectory summary series are input into the multimodal large language model for alignment determination and pointer movement instruction generation. Execute pointer movement instructions, iteratively acquire screen images to update pointer position until alignment is determined, generate pointer operation instructions based on the task structure and execute them.
2. The pointer-based intelligent iterative localization method combining a multimodal large language model as described in claim 1, characterized in that, The task structure is used as a structured task description, and the description items of the task structure include at least the operation object, the type of action to be performed, and the spatial constraint relationship of the operation object.
3. The pointer-based intelligent iterative localization method combining a multimodal large language model as described in claim 1, characterized in that, In response to the task structure, screen images and historical execution information are periodically acquired using the sliding window method to obtain a series of screen images and a series of trajectory summaries, including: Based on a preset acquisition period, screen images are acquired using the sliding window method to obtain a series of screen images including the current screen image frame and N consecutive preceding screen image frames, where N is greater than or equal to 1. Combining the sliding window method, starting from the current moment, backtrack to extract the first M items of pre-execution information, and output the historical execution information, where M is greater than or equal to N; Based on the historical execution information, the pointer's epoch movement information and epoch click information are extracted, and serialized based on the timestamp of the prior execution information to output a structured trajectory summary series.
4. The pointer-based intelligent iterative localization method combining a multimodal large language model as described in claim 1, characterized in that, The screen image series is analyzed to extract interface layout features, and interface continuity is determined based on these features, including: Extract the binarized layout features of interactive elements in the current screen image frame and the previous screen image frame; Based on the binarized layout features, calculate the typical change indicators between the current screen image frame and the preceding screen image frame; When the typical change indicator does not exceed the first preset threshold, the determination interface remains continuous and subsequent steps are executed.
5. The pointer-based intelligent iterative localization method combining a multimodal large language model as described in claim 4, characterized in that, The process of parsing the screen image series, extracting interface layout features, and determining interface coherence based on the interface layout features also includes: When the typical change indicator exceeds the first preset threshold but does not exceed the second threshold, the detection feature points are selected in the current screen image frame and the preceding screen image frame according to the binarized layout features and in combination with the preset feature point rules. Based on the determined detection feature points, the feature descriptors of the current screen image frame and the previous screen image frame are calculated respectively to obtain the first feature matrix and the second feature matrix, wherein the feature descriptor is the feature point spacing; Based on the first feature matrix and the second feature matrix, multi-point matching is performed on the current screen image frame and the preceding screen image frame, and the geometric transformation matrix is obtained according to the multi-point matching result; The current screen image frame and the preceding screen image frame are aligned according to the geometric transformation matrix, and the overlap area ratio is calculated based on the image alignment result. If the overlapping area ratio is greater than the empirical area ratio based on confidence level, the output coherence judgment result is coherent; otherwise, the output coherence judgment result is incoherent.
6. The pointer-based intelligent iterative localization method combining a multimodal large language model as described in claim 4, characterized in that, Extracting the binarized layout features of interactive elements in the current screen image frame and the preceding screen image frames also includes: Perform interface element detection on the current screen image frame and the preceding screen image frame respectively, and identify the main interactive elements; For each of the main interactive elements, a bounding rectangle region is determined, and within the bounding rectangle region, an edge detection method and a preset envelope constraint are combined to determine the element envelope of each main interactive element. Based on the envelopes of each element, a corresponding binarized layout image is generated on the blank image, and the output is the binarized layout feature, wherein the pixels within the element envelopes are assigned a first pixel value, and the background area of the blank image is assigned a second pixel value.
7. The pointer-based intelligent iterative localization method combining a multimodal large language model as described in claim 1, characterized in that, If the coherence determination result is coherent, then the task structure, the screen image series, and the trajectory summary series are input into the multimodal large language model for alignment determination and pointer movement instruction generation, including: Extract the current position coordinates of the pointer and the current screen image frame from the series of screen images; The current position coordinates, the current screen image frame, the task structure, and the trajectory summary series are encoded into a multimodal input. The multimodal large language model is invoked to analyze and process the multimodal input to obtain a structured alignment discrimination result, wherein the alignment discrimination result includes at least the alignment Boolean value and the movement command.
8. The pointer-based intelligent iterative localization method combining a multimodal large language model as described in claim 7, characterized in that, Before encoding the current position coordinates, the current screen image frame, the task structure, and the trajectory summary series into a multimodal input, the process also includes: Based on the task structure, extract the operation object; Obtain historical interaction records, and traverse the historical interaction records using the operation object as the index to perform dual position extraction, and obtain dual position extraction results, wherein the dual position extraction results include the operation object's edge appearance boundary and the operation object's click boundary; Based on the dual-position extraction results, and combined with statistical analysis methods, the typical ratio of the apparent boundary of the operation object to the click boundary of the operation object is determined as the position correction coefficient. The position correction coefficient is added to the task structure and associated with the operation object for storage.
9. The pointer-based intelligent iterative localization method combining a multimodal large language model as described in claim 8, characterized in that, The pointer operation instructions are generated based on the operation object, the type of action to be executed, and the position correction coefficient in the task structure.
10. A pointer-based intelligent iterative localization system combining a multimodal large language model, characterized in that: The pointer-based intelligent iterative localization method combining a multimodal large language model as described in any one of claims 1-9 includes: The semantic parsing module is used to perform semantic parsing on the acquired user's natural language instructions using a multimodal large language model to obtain the task structure; The data processing module is used to respond to the task structure by periodically acquiring screen images and historical execution information using the sliding window method to obtain a series of screen images and a series of trajectory summaries. The feature extraction module is used to parse the series of screen images, extract interface layout features, and determine interface coherence based on the interface layout features. The instruction generation module is used to input the task structure, the screen image series, and the trajectory summary series into the multimodal large language model if the coherence discrimination result is coherent, and to perform alignment discrimination and pointer movement instruction generation. The instruction execution module is used to execute pointer movement instructions, iteratively acquire screen images to update the pointer position until the alignment judgment result is aligned, and generate pointer operation instructions according to the task structure and execute them.