Mouse triggered cross-application file context processing method

Abstract

This application discloses a mouse-triggered cross-application file context processing method, system, and computer-readable storage medium, relating to the fields of human-computer interaction, operating system context capture, and artificial intelligence office processing. The client software initiates a resident context capture process, periodically detects the foreground window and determines the application type, and obtains the currently active file, webpage address, or selected file through cross-process reading or file handle enumeration based on application automation interfaces, accessibility interfaces, file manager interfaces, and desktop list controls. A unified context object is constructed and written to the cache and pushed when the object changes. In response to mouse smart button triggers, natural language instructions are obtained and the target context object is read. The structured operation intent is obtained through intent parsing and routed to the execution module according to the context type and intent type to perform file processing, webpage content processing, document content processing, content generation, document write-back, or system operations on the current operation object.

Description

Technical Field

[0001] This application relates to the fields of computer human-computer interaction, operating system context capture, and artificial intelligence office processing technology, specifically to a mouse-triggered cross-application file context processing method, system, and computer-readable storage medium. Background Technology

[0002] With the improvement of the capabilities of large-scale pre-trained language models in natural language understanding, content generation, and intent parsing, assistant software based on artificial intelligence models is gradually being applied in office scenarios. Users frequently need to handle objects in their daily work, including: one or more files selected in a file manager or on the desktop; applications open in the foreground and in focus, along with their currently active document, webpage, or window content; and pure command tasks such as content generation or system operations issued by the user when no specific object is detected. Therefore, users hope to be able to initiate AI processing requests for objects they are currently operating on or focused on through convenient interactive entry points.

[0003] In the existing technology, the solutions for the above-mentioned needs mainly include the following categories.

[0004] The first type is based on the operating system's built-in voice assistant or the application's built-in intelligent assistant. This type of solution typically requires the user to explicitly state the target application name, function name, or object name in the command. The system matches and launches the corresponding application based on the application or function name in the command text, and then the application's built-in assistant module completes the processing. The object being processed in this solution is primarily determined by the command text, which is disconnected from the objects in the user's current operating environment. Therefore, when a user wants to process "this current document," "this current webpage," or "the files just selected," they still need to repeatedly specify the object in the command; if the application being processed does not have a built-in assistant module or does not provide a callable interface, it is difficult to complete the processing.

[0005] The second category is interaction schemes based on text selection within an application window, floating toolbars in editable areas, the Document Object Model (DOM), or webpage metadata. These schemes typically utilize internal application interfaces, webpage interfaces, or text selection location information to present an operation entry point near the selected or edited content. They are suitable for rewriting, polishing, translating, or formatting local content within a single application window. However, the scope of these schemes is usually limited to a single application window, making it difficult to handle selected files in file managers or on the desktop across application boundaries, and also difficult to handle files or webpage objects already opened in other applications. For applications that do not expose the corresponding interfaces, this type of scheme also has weak compatibility.

[0006] The third type involves the user manually specifying the target object during the command input phase. For example, the user needs to drag a file into the assistant software, select the target file through a file selection dialog box, paste the file path or web address, and then the assistant software reads and processes it. Although this type of solution can be used across applications to some extent, each interaction requires the user to perform the steps of "switching to assistant software—specifying the target object—entering the command—returning to the original operation scenario," failing to fully utilize the operational context in which the user is currently in a file, webpage, or document environment, resulting in high interaction costs.

[0007] Furthermore, while existing desktop automation or robotic process automation (RPA) solutions can execute preset tasks by simulating user actions or reading window information, they are typically centered around pre-configured process scripts, focusing on completing repetitive tasks according to fixed steps. For AI-driven processing requests issued by users during their work process, specifically targeting the current task, these solutions still struggle to provide a sufficiently natural and coherent interactive experience.

[0008] The above solutions share a common problem: the target processing object typically originates from user command text, a single application's internal interface, or additional manual input by the user, rather than consistently from the user's current actual operating environment. For the same user, their current focus might be a selected file in a file manager, a file on the desktop, an open document in a foreground application, a webpage in a browser, or a pure command task without a clearly defined object. Existing solutions struggle to achieve seamless switching between these scenarios, easily leading users to repeatedly specify the object, manually select the object, or switch tools.

[0009] Meanwhile, in desktop operating systems, the foreground application identifier, the currently active file path, the current webpage address, the file path selected in the file manager, and the file path selected on the desktop belong to different applications, windows, processes, or system controls. Different types of applications also have objectively different capabilities in providing the currently open file path; some applications even provide an unstable, standard interface for obtaining this information. Furthermore, although both the file manager window and the operating system desktop are used to display selected files, they differ in how they read the selected item. Therefore, it presents objective technical difficulties for AI assistants to accurately determine the object they wish to process when a user triggers a command.

[0010] In summary, existing technologies lack an effective mechanism that enables AI assistants to objectively and accurately determine the objects to be processed in the user's current real-world operating environment, and to maintain consistent trigger entry points and coherent processing flows across different office scenarios such as files, applications, web pages, and pure commands. This is a technical problem that urgently needs to be solved. Summary of the Invention

[0011] The purpose of this application is to provide a mouse-triggered cross-application file context processing method, system, and computer-readable storage medium to at least solve the technical problems in the prior art where artificial intelligence assistants have difficulty in determining the object to be processed in the user's current real operating environment, and have difficulty in maintaining a consistent trigger entry and coherent processing flow across different office scenarios such as files, applications, web pages, and pure commands.

[0012] In a first aspect, this application provides a mouse-triggered cross-application file context processing method, applied to an electronic device running a Windows operating system and communicating with a mouse, wherein the mouse includes smart buttons and the electronic device runs client software, characterized in that it includes:

[0013] The client software initiates a persistent context capture process, which periodically acquires the current foreground window and obtains the window identifier, window title, window class name, process identifier, and process name of the current foreground window. Based on at least one of the window title, window class name, process identifier, and process name, the application type corresponding to the current foreground window is determined.

[0014] Candidate targets are obtained by performing at least one context retrieval method based on application type and / or window state, the method including: obtaining the currently active file path through an application automation interface or file handle enumeration; obtaining the current webpage address of the browser through an accessibility interface; obtaining the selected file path of the file explorer through a file manager interface; obtaining the selected file path of the desktop through cross-process reading of the desktop list control;

[0015] A unified context object is constructed based on the target object candidates, and the target context object is determined based on at least one of the source type, foreground window state, and detection time. An empty context object is constructed when no candidates are obtained.

[0016] When the unified context object changes, it is written to the cache and a structured context update message is sent via inter-process communication;

[0017] In response to a smart key press, the client software obtains a natural language instruction, reads the target context object or an empty context object from the cache, inputs it into the intent parser to obtain a structured operation intent, and routes it to the execution module according to the context type and intent type to perform at least one of the following: file processing, web page content processing, document content processing, content generation, document write-back, or system operation.

[0018] In one possible implementation, the mouse further includes a microphone, and the trigger events for the mouse smart buttons include long-press events and short-press events. When the trigger event is a long-press event, the client software activates the mouse's microphone to capture the user's voice, and converts the user's voice into the natural language command after the smart button is released. When the trigger event is a short-press event, the client software pops up a text input window and uses the text entered by the user in the text input window as the natural language command. The voice command corresponding to the long-press event and the text command corresponding to the short-press event both reuse the same target context object determination process and operation routing process. Thus, both voice and text command entry points share the same context capture and routing mechanism, expanding the command input methods without adding context processing logic.

[0019] In one possible implementation, the persistent context capture process calls the Windows window interface to obtain the window identifier, window title, window class name, process identifier, and process name of the current foreground window, and determines the application type corresponding to the current foreground window according to a preset application type table. The preset application type table includes the correspondence between process name, window class name, window title keywords, application type, and context acquisition method. The application type includes at least one of document editing applications, table editing applications, presentation editing applications, PDF reading applications, browser applications, image processing applications, code editing applications, and text editing applications. When the application type corresponding to the current foreground window is a document editing application, table editing application, or presentation editing application, the persistent context capture process obtains the currently active file path through the Component Object Model interface. When the application type corresponding to the current foreground window is a browser application, the persistent context capture process finds the browser address bar control through the accessibility interface and determines the value in the address bar control as the browser's current webpage address. Therefore, appropriate acquisition methods are adopted for different application types to address the objective differences in the ability of different types of applications to expose the currently open file path.

[0020] In one possible implementation, obtaining the selected file path in the resource manager via the file manager interface includes: determining whether the current foreground window is a resource manager window; when the current foreground window is a resource manager window, obtaining the folder view corresponding to the resource manager window through the Shell.Application component object model interface, and calling the SelectedItems method of the folder view to obtain one or more file paths selected by the user in the resource manager window. Obtaining the selected file path on the desktop via cross-process reading of the desktop list control includes: searching for the desktop view window under the desktop management window, and searching for the desktop list control under the desktop view window; when the desktop list control is not found under the desktop management window, traversing the working window carrying the desktop view, and searching for the desktop view window and its desktop list control under the working window; determining the resource manager process carrying the desktop list control, allocating cross-process memory in the resource manager process, sending a list item read message to the desktop list control, reading the text of the selected desktop icon from the resource manager process, and determining the desktop file path or shortcut target path based on the text. Therefore, even though the desktop list control belongs to the File Explorer process and cannot be directly read through the ordinary file manager interface, this application can still obtain the path of the selected file on the desktop.

[0021] In one possible implementation, when the current active file path cannot be obtained through the application automation interface, the resident context capture process obtains the current active file path through file handle enumeration. The file handle enumeration includes: enumerating file handles opened by processes in the system; filtering file handles of target processes based on the process identifier; copying the file handles of the target processes to the current process; querying the object path corresponding to the copied file handles; converting the device path in the object path to a Windows file system path; extracting candidate filenames from the window title, and determining the current active file path corresponding to the current foreground window based on the matching results of the candidate filenames and the filenames in the Windows file system path. When the current active file path is not obtained through file handle enumeration, the resident context capture process performs command-line parameter parsing and / or recent file record reverse lookup to determine the current active file path based on the candidate filenames extracted from the window title. Therefore, this application improves compatibility with applications that provide unstable standard automation interfaces and increases the success rate of identifying the current active file path through a multi-level fallback acquisition method.

[0022] In one possible implementation, the unified context object includes at least the contextId field, contextType field, sourceType field, hwnd field, pid field, and timestamp field. The unified context object also includes at least one of the pathList field, url field, processName field, windowTitle field, priority field, and confidence field. The contextType field is used to indicate the context type, which includes at least one of the following: currently active file, file selected in Windows Explorer, file selected on the desktop, current webpage address, and empty context. The sourceType field is used to indicate the context source, which includes at least one of the following: component object model interface, Shell.Application interface, desktop list control, accessibility interface, file handle enumeration, command-line argument parsing, and recent file record reverse lookup. The determination of the target context object based on at least one of the following: source type, foreground window state, and detection time, includes: when the current foreground window is a file explorer window and a file path is selected in the file explorer, the candidate object corresponding to the selected file path in the file explorer is determined as the target context object; when the current foreground window is a desktop window and a file path is selected on the desktop, the candidate object corresponding to the selected file path on the desktop is determined as the target context object; when there is no selected file path in the foreground state but a currently active file path exists, the candidate object corresponding to the currently active file path is determined as the target context object; when there is neither a selected file path nor a currently active file path in the foreground state but a current webpage address exists in the browser, the candidate object corresponding to the current webpage address in the browser is determined as the target context object; when none of the above candidate objects exist, an empty context object is determined as the target context object. The resident context capture process performs current foreground window detection and target object candidate acquisition according to a preset period, and when the unified context object changes, it sends a structured context update message to the user interface process of the client software through inter-process communication; when the unified context object does not change, it does not send duplicate structured context update messages. Therefore, when multiple candidate objects coexist at the same time, this application can determine a target context object based on the source type, foreground window state, and detection time, avoiding object misalignment during subsequent intent parsing and operation execution; and by pushing only when the object changes, it reduces inter-process communication overhead.

[0023] In one possible implementation, the structured operation intent includes at least an intent type and operation parameters; the structured operation intent also includes at least one of the following: target object identifier, target file path list, target webpage address, output type, output location, write location, confidence level, and whether user confirmation is required; the intent type includes at least one of the following: question and answer, summary, format conversion, multi-file merging, presentation generation, mind map generation, document write-back, pure instruction content generation, or system operation. The client software performs routing based on the context type and the intent type, including: when the context type is the currently active file, the file selected in the file explorer, or the file selected on the desktop, and the intent type is format conversion, routing to the file format conversion module; when the context type is the file selected in the file explorer or the file selected on the desktop, and the target file path list includes multiple file paths, and the intent type is merging multiple files, routing to the multi-file merging module; when the context type is the currently active file and the intent type is question and answer or summary, routing to the document parsing module and the question and answer summary module; when the context type is the current webpage address and the intent type is question and answer or summary, routing to the webpage content processing module and the question and answer summary module; when the context type is empty... In the following context, when the intent type is system operation, routing is directed to the system operation module; wherein, when the context type is the current webpage address, the webpage content processing module obtains at least one of the webpage title, webpage body, webpage summary, or user-selected text based on the current webpage address, and performs question-and-answer or summary operations in conjunction with the natural language instruction; when the context type is the current active file, file selected in File Explorer, file selected on the desktop, or current webpage address, and the intent type is presentation generation or mind map generation, routing is directed to the corresponding content generation module; when the context type is an empty context and the intent type is pure instruction content generation, routing is directed to the content generation module; when the context type is the current active file and the intent type is document write-back, routing is directed to the document write-back module.

[0024] Furthermore, when the context type is a file selected in File Explorer or a file selected on the desktop, the target file path list includes multiple file paths of different file types, and the intent type is presentation generation, the content generation module calls the corresponding content extractor according to the file type corresponding to each file path to extract the file content of the multiple different file types, and performs summary synthesis on the extracted content according to the operation parameters to generate one or more page presentation files; the different file types are determined by at least one of file extension, MIME type, or content extractor type, and the multiple different file types include at least two of document files, table files, presentation files, PDF files, Markdown files, text files, and image files; the operation parameters include at least one of presentation page number, summary granularity, output theme, and output layout. When the intent type is document write-back, the client software obtains the content of the currently active document according to the target context object, generates content to be written according to the natural language instruction and the content of the currently active document, and writes the content to be written into the currently active document through at least one of the component object model interface, official plugin interface, accessibility interface, or input simulation interface.

[0025] Secondly, this application provides a mouse-triggered cross-application file context processing system, including a mouse and an electronic device running a Windows operating system. The mouse includes smart buttons, a microphone, a main control chip, and a communication module. The smart buttons are used to receive user-triggered operations, the microphone is used to collect user voice, and the main control chip is used to send button status signals and / or voice signals to the electronic device through the communication module. The electronic device runs client software and a resident context capture process started by the client software. The resident context capture process is used to periodically acquire the current foreground window and acquire the window identifier, window title, window class name, process identifier, and process name of the current foreground window. Based on at least one of these, it determines the application type corresponding to the current foreground window, acquires target object candidates based on the application type and / or the current window state, constructs a unified context object based on the target object candidates, and writes it to the context cache and sends a structured context update message through inter-process communication when the unified context object changes. The client software is used to acquire natural language commands input by the user in response to the triggering event of the mouse smart button, reads the target context object or empty context object from the context cache, generates a structured operation intent based on the natural language command and the target context object or empty context object, and selects the corresponding execution module based on the context type and intent type. The execution module is used to perform at least one of file processing, web page content processing, document content processing, multi-file content aggregation processing, content generation processing based on the target context object, document write-back processing, or system operation. The system is used to execute the method described in the first aspect above.

[0026] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0027] Compared with the prior art, this application has the following beneficial effects.

[0028] Firstly, this application uses the smart mouse button as a unified hardware trigger entry point, and enables the client software to adopt a consistent processing flow for currently active files, files selected in the file explorer or desktop, current web pages in the browser, and pure command tasks without a clear object. Users do not need to switch tools between different office scenarios, nor do they need to manually upload, drag and drop, or repeatedly specify target objects in commands, thereby maintaining a consistent trigger entry point and a coherent processing flow across scenarios such as files, applications, web pages, and pure commands.

[0029] Secondly, the objects to be processed in this application originate from the detection results of the current state of the operating system by the resident context capture process, rather than from the instruction text, the internal interface of a single application, or the user's additional manual input. Therefore, the processing objects are decoupled from the instruction text, allowing the user to directly initiate processing on the current real operation objects such as "the current document", "the current webpage", or "the files that were just selected".

[0030] Third, considering the objective differences in the ability of different types of applications to expose the currently open file path, and the technical difficulties such as the desktop list control belonging to the resource manager process and not being able to be read through the ordinary file manager interface, this application distributes appropriate acquisition methods according to application type, such as component object model interface, accessibility interface, file manager interface, or cross-process reading of desktop list control. When the currently active file path cannot be obtained through the application automation interface, multiple fallback methods are used in sequence, such as file handle enumeration, command line parameter parsing, and recent file record reverse lookup, which improves the compatibility with various applications and the success rate and accuracy of identifying the current operation object.

[0031] Fourth, this application uses a resident context capture process to continuously detect and write the changed unified context object into a cache at a preset period, so that the client software can directly read the target context object from the cache when the key is triggered, without having to start the detection process again after the trigger, thereby shortening the time consumption between the trigger event and the context acquisition result and improving the interaction response speed; at the same time, update messages are pushed only when the object changes, reducing inter-process communication overhead.

[0032] Fifth, this application normalizes multiple candidate objects into a unified context object, and determines a target context object when multiple candidate objects coexist based on the source type, foreground window state, and detection time, thus avoiding object misalignment in subsequent intent parsing and operation execution; furthermore, through structured operation intent and routing based on context type and intent type, the same natural language instruction is correctly distributed to the corresponding execution module in different contexts, and supports cross-format processing such as extracting content from multiple objects of different file types and then summarizing it to generate presentations. Attached Figure Description

[0033] Figure 1 A schematic diagram of the overall flow of the mouse-triggered cross-application file context processing method provided in the embodiments of this application;

[0034] Figure 2 A schematic diagram illustrating the periodic detection, caching, and push process of the resident context capture process provided in this application embodiment;

[0035] Figure 3This is a schematic diagram of the target object candidate acquisition and multi-level fallback process based on application type provided in the embodiments of this application;

[0036] Figure 4 A schematic diagram illustrating the process of a desktop list control reading the path of a selected file on the desktop across processes, as provided in an embodiment of this application.

[0037] Figure 5 This is a flowchart illustrating the process of determining the target context object based on a unified context object, as provided in an embodiment of this application.

[0038] Figure 6 A schematic diagram illustrating the routing of the structured operation intent parsing and execution module provided in an embodiment of this application;

[0039] Figure 7 This is a schematic diagram of the structure of a mouse-triggered cross-application file context processing system provided in an embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0041] In the description of this application, the term "foreground window" refers to the window in the operating system that is currently in focus and receives user input; "target object candidate" refers to a file path or web address that the resident context capture process obtains in a round of detection and may become a processing object; "unified context object" refers to a data object obtained by normalizing the target object candidates in a round of detection according to a unified data structure; "target context object" refers to the object determined from the unified context object that will be provided to intent parsing in this instance; "empty context object" refers to an object constructed when no target object candidate is detected, indicating that there is no explicit processing object. The following interface names, field names, and specific function names are used to describe one or more optional implementations of this application; unless otherwise expressly defined in the claims, they should not be construed as further limitations on the scope of protection of this application.

[0042] I. Overall Methodology

[0043] See Figure 1This application provides a mouse-triggered cross-application file context processing method, applied to an electronic device running a Windows operating system and connected to a mouse. The electronic device runs client software, and the method includes steps S1 to S6. In S1, the client software starts a resident context capture process, which periodically acquires the window identifier, window title, window class name, process identifier, and process name of the current foreground window, and determines the application type corresponding to the current foreground window accordingly. In S2, the resident context capture process selects one or more of its preset context acquisition methods based on the application type and / or the current window state to acquire target object candidates. In S3, a unified context object is constructed based on the target object candidates, and the target context object is determined; if no candidate is acquired, an empty context object is constructed. In S4, the unified context object of this round is compared with the previous round; if changes occur, they are written to the context cache and pushed via inter-process communication. In S5, in response to a mouse smart button trigger event, the client software acquires the natural language command input by the user and reads the target context object or an empty context object from the context cache. In S6, the natural language instructions and the read object input intent parser are used to obtain the structured operation intent, and the corresponding execution module is routed to perform the corresponding operation according to the context type and intent type.

[0044] It should be noted that S1 to S4 are executed continuously and periodically by the resident context capture process, regardless of whether the user triggers a key press; S5 to S6 are driven by key press events. Therefore, the target context object required for key press triggering has been pre-detected and cached, and the client software can directly read it without needing to restart the detection process after the trigger.

[0045] II. Periodic Detection, Caching, and Pushing of Persistent Processes

[0046] See Figure 2 In one implementation, a resident context capture process performs detection cyclically according to a preset period, which is between 400ms and 1000ms, for example, 600ms. In each round, the process first obtains foreground window information and determines the application type, then obtains target object candidates, and subsequently constructs a unified context object for this round. The process serializes the unified context object for this round into a preset structured format (e.g., a JSON string) and compares it with the structured result of the previous round: when the two are inconsistent, it is determined that the context has changed, the unified context object for this round is written to the context cache, and a structured context update message is sent to the client software via inter-process communication; when the two are consistent, duplicate update messages are not sent to reduce inter-process communication overhead.

[0047] In one implementation, the client software includes a main process and a user interface process. A resident context capture process outputs structured context update messages to the main process. The main process parses these messages, updates its maintained latest context cache, and compares the current message with the previous one. If they are inconsistent, it broadcasts a context update event to the user interface process via inter-process communication. Multiple inter-process communication channels can be configured, such as: a first channel for real-time context push from the main process to the user interface process (e.g., `fileContext:update`), a second channel for the user interface process to request the latest context from the main process (e.g., `fileContext:getLatest`), and a third channel for newly created user interface windows to request a context push from the main process (e.g., `fileContext:requestPush`). In one implementation, after each preset number of rounds (e.g., approximately 30 rounds), the resident context capture process proactively releases the component object model interface references generated during the detection process to reclaim resources.

[0048] III. Obtaining Front-End Window Information and Determining Application Type

[0049] In one implementation, the resident context capture process calls the Windows window interface to obtain foreground window information. For example, it obtains the window identifier through the interface for obtaining the foreground window handle, the window title through the interface for obtaining the window title, the window class name through the interface for obtaining the window class name, and the process identifier through the interface for obtaining the process identifier to which the window belongs. The process then queries its executable file name based on the process identifier as the process name. The process determines the application type according to a preset application type table, which records the correspondence between process name, window class name, window title keywords, application type, and context acquisition method. Application types include, but are not limited to, document editing applications, table editing applications, presentation editing applications, PDF reading applications, browser applications, image processing applications, code editing applications, and text editing applications.

[0050] IV. Application Type-Based Target Object Candidate Acquisition and Multi-Level Backup

[0051] See Figure 3 The resident context capture process selects the appropriate acquisition method based on the application type.

[0052] For document editing, spreadsheet editing, or presentation editing applications, the active file path is obtained through the Component Object Model (COM) interface. For example, for a document editing application, the file path of its active document object (e.g., ActiveDocument) is obtained; for a spreadsheet editing application, the file path of its active workbook object (e.g., ActiveWorkbook) is obtained; and for a presentation editing application, the file path of its active presentation object (e.g., ActivePresentation) is obtained. For similar applications with multiple candidate ProgIDs, multiple preset ProgIDs are tried sequentially to obtain the active object, and the active file path corresponding to the current foreground window is determined by combining the window title and the active object name.

[0053] For browser applications, the current webpage address is obtained through accessibility interfaces (such as interface automation interfaces): a tree of interface elements is constructed with the window identifier of the current foreground window as the root node. Candidate controls of type edit box are searched in the element tree, their values ​​are read, and when the value matches the webpage address format, the control is identified as the address bar control and its value is identified as the current webpage address.

[0054] When the current active file path cannot be obtained through application automation interfaces such as the Component Object Model (COM) interface (e.g., the application does not provide the interface, the interface is not open, or although it responds, it does not return a valid path), the resident context capture process executes a multi-level fallback. The first level is file handle enumeration: enumerating file handles opened by processes in the system (e.g., through NtQuerySystemInformation), filtering the handles of the target process based on the process identifier, copying them to the current process (e.g., through DuplicateHandle), querying the object path corresponding to the copied handle (e.g., through NtQueryObject), converting the device path in the object path to a file system path with a drive letter (e.g., obtaining the mapping relationship between device path and drive letter through QueryDosDevice and converting accordingly), and then extracting candidate filenames from the window title and matching them with the filenames in the above file system path to determine the current active file path. During the file handle enumeration process, the resident context capture process can filter non-disk file objects, directory objects, pipe objects, and system device objects, and set a timeout for the query of handle object paths to avoid blocking the detection process due to abnormal target process handles. The second level is command-line argument parsing: obtaining the command-line arguments of the target process, extracting the file path parameters, and determining the path by matching candidate filenames. The third level is recent file record reverse lookup: parsing the target path pointed to by shortcuts in the recently used file directories of the operating system, and determining the path by matching candidate filenames. This fallback process is performed level by level, returning upon success at any level.

[0055] V. Retrieving Selected Files from File Manager and Desktop

[0056] For a selected file in the file manager, the persistent context capture process determines whether the current foreground window is the file explorer window. If so, it obtains the folder view corresponding to the window through the Shell.Application component object model interface, and calls the SelectedItems method of the folder view to get one or more file paths selected by the user.

[0057] See Figure 4 For selected files on the desktop, since the desktop list control belongs to the resource manager process, it cannot be read directly through the ordinary file manager interface. This embodiment adopts cross-process reading: First, it searches for the desktop view window (e.g., the SHELLDLL_DefView window) and the desktop list control (e.g., the SysListView32 control) under the desktop management window (e.g., the Progman window); if not found, it traverses the working window (e.g., the WorkerW window) that carries the desktop view and searches for the desktop view window and desktop list control under it; then it determines the resource manager process that carries the list control, allocates cross-process memory in the process (e.g., through VirtualAllocEx), sends a list item read message (e.g., the LVM_GETITEMTEXT message) to the desktop list control, and then reads back the text of the selected desktop icon from the process (e.g., through ReadProcessMemory).

[0058] In one implementation, after reading the text of the selected desktop icon from the desktop list control, the resident context capture process searches for the file path corresponding to the text in the current user desktop directory and the public desktop directory; when the file path corresponds to a shortcut file, the target path pointed to by the shortcut file is parsed, and the target path is used as the desktop selected file path; when the system hides file extensions or there are multiple candidate files with the same name, the final desktop selected file path is determined according to the actual extension in the file system, the desktop item index, the file modification time, or the user confirmation result.

[0059] VI. Construction of the Unified Context Object and Determination of the Target Context Object

[0060] The resident context capture process normalizes the target object candidates acquired in this round into a unified context object. In one implementation, the unified context object includes at least one of the following: a contextId field for identifying the object, a contextType field for indicating the context type, a sourceType field for indicating the context source, an hwnd field for recording the foreground window handle, a pid field for recording the process identifier, and a timestamp field for recording the detection time; it may also include at least one of the following: a pathList field for recording a list of file paths, a url field for recording web page addresses, a processName field for recording the process name, a windowTitle field for recording the window title, a priority field for recording the priority, and a confidence field for recording the confidence level.

[0061] The context type includes at least one of the following: currently active file, file selected in Windows Explorer, file selected on the desktop, current webpage address, and empty context; the context source includes at least one of the following: component object model interface, Shell.Application interface, desktop list control, accessibility interface, file handle enumeration, command line argument parsing, and recent file record reverse lookup.

[0062] For example, when a user selects multiple files in File Explorer, the contextType of the unified context object can be the file selected in File Explorer, the sourceType can be the Shell.Application interface, the pathList can include the file paths of the multiple files, the hwnd can record the file explorer window handle, the pid can record the file explorer process identifier, the timestamp can record the current detection time, the priority can record the priority of the candidate object, and the confidence can record the confidence level of the candidate object.

[0063] See Figure 5When multiple candidate objects exist simultaneously, the process determines a target context object based on the foreground window state, source type, and detection time. In one implementation: if the current foreground window is a file explorer window and a selected file path exists, it is determined as the target context object; if the current foreground window is a desktop window and a selected file path exists, it is determined as the target context object; if no selected file path exists in the foreground but a currently active file path exists, the currently active file path is determined as the target context object; if neither a selected file path nor a currently active file path exists, but a current webpage address exists, the current webpage address is determined as the target context object; if none of the above candidate objects exist, an empty context object is determined as the target context object.

[0064] VII. Command Input Method

[0065] In one implementation, the mouse also includes a microphone, and the smart button's trigger events include long-press events and short-press events. When a long-press event occurs, the client software activates the mouse's microphone to capture the user's voice and converts the voice into a natural language command after the button is released. When a short-press event occurs, the client software pops up a text input window and uses the user's input text as a natural language command. Both types of commands reuse the same target context object determination process and operation routing process.

[0066] 8. Intent Resolution and Execution Module Routing

[0067] See Figure 6 The client software inputs natural language instructions along with the read target context object or empty context object into the intent parser. The intent parser can employ a large-scale artificial intelligence model, a rule parser, or a combination of both, and outputs a structured operation intent according to a preset structured format. The structured operation intent includes at least the intent type and operation parameters, and may also include at least one of the following: target object identifier, target file path list, target webpage address, output type, output location, write location, confidence level, and whether user confirmation is required. The intent type includes at least one of the following: question and answer, summary, format conversion, multi-file merging, presentation generation, mind map generation, document write-back, pure instruction content generation, and system operation.

[0068] For example, when the natural language instruction is "summarize these files into a single PPT slide" and the target context object's pathList includes multiple file paths, the intent type in the structured operation intent can be presentation generation, the target file path list can be the pathList, the operation parameters can include summary synthesis and a page count of one, and the output type can be a presentation file.

[0069] The client software routes structured operation intents to the corresponding execution modules based on context type and intent type. Examples of routing rules are as follows: When the context type is "currently active file," "file selected in File Explorer," or "file selected on the desktop," and the intent type is "format conversion," the program routes to the file format conversion module. When the context type is "file selected in File Explorer" or "file selected on the desktop," the target file path list includes multiple file paths, and the intent type is "multi-file merging," the program routes to the multi-file merging module. When the context type is "currently active file," and the intent type is "question and answer" or "summary," the program routes to the document parsing module and the question and answer summary module. When the context type is "current webpage address," and the intent type is "question and answer" or "summary," the program routes to the webpage content processing module and the question and answer summary module. When the context type is "currently active file," "file selected in File Explorer," "file selected on the desktop," or "current webpage address," and the intent type is "presentation generation" or "mind map generation," the program routes to the corresponding content generation module. When the context type is "empty context," and the intent type is "pure instruction content generation," the program routes to the content generation module; when the context type is "empty context," and the intent type is "system operation," the program routes to the system operation module. When the context type is "current active file" and the intent type is "document write-back", the route is routed to the "document write-back" module.

[0070] Regarding the implementation of the webpage content processing module: When the target context object is the current webpage address, the webpage content processing module obtains the webpage content based on that address. The methods for obtaining the content include, but are not limited to: reading the document object model (DOM) of the current tab through a browser interface and extracting the main text, or having the client software request the page from that address and then parse its document structure. When the webpage requires login status, relies on dynamic script rendering, or has access restrictions, it prioritizes reading the DOM or visible text content of the current tab through a browser interface, browser extension interface, or browser debugging interface. After obtaining the page, this module removes non-text elements such as navigation bars and advertisements, extracts at least one of the following: webpage title, webpage main text, webpage summary, or user-selected text, and then combines this with natural language commands to hand over the question-and-answer summary module for question-and-answer or summary execution.

[0071] Regarding the implementation of the question-and-answer summary module: This module uses the document content extracted by the document parsing module or the webpage content extracted by the webpage content processing module as context, and inputs it along with natural language instructions into the AI ​​big data model. When the intent type is question-and-answer, the big data model generates an answer to the user's question based on the above content; when the intent type is summary, the big data model generates a summary based on the above content at the granularity indicated by the operation parameters. The answer or summary is then displayed to the user via the client software.

[0072] Regarding the implementation of the mind map generation module: This module inputs the content of the object to be processed (such as the content of an active document, the content of a selected file, or the body of a webpage) and natural language instructions into an artificial intelligence model. The model outputs outline data with hierarchical relationships, which includes multiple nodes and the parent-child relationships between nodes. This module converts the outline data into a mind map data structure (such as a tree structure or corresponding structured descriptive data), and then renders and generates a mind map file based on it. The mind map file can be in the form of at least one of mind map files, scalable vector graphics files, or image files.

[0073] Regarding generating presentations from multiple files across formats: When the context type is "File selected in Explorer" or "File selected on the desktop," the target file path list includes multiple file paths of different file types, and the intent type is "Presentation generation," the content generation module calls the corresponding content extractor based on the file type of each file path to extract the content of multiple different file types. Then, based on the operation parameters, it performs a summary synthesis of the extracted content to generate a one-page or multi-page presentation file. The file type is determined by at least one of the following: file extension, MIME type, or content extractor type. Multiple different file types include at least two of the following: document files, table files, presentation files, PDF files, Markdown files, text files, and image files. The operation parameters include at least one of the following: presentation page number, summary granularity, output theme, and output layout.

[0074] Regarding the implementation of the document write-back module: When the intent type is document write-back, the client software obtains the content of the currently active document based on the target context object, generates the content to be written from the large model based on the natural language instructions and the content of the currently active document, determines the current cursor position, the end position of the document, the specified paragraph position, or the specified title position based on the writing position in the structured operation intent, and writes the content to be written into the currently active document through at least one of the component object model interface, official plugin interface, auxiliary function interface, or input simulation interface.

[0075] Regarding the implementation of the system operation module: When the context type is empty and the intent type is system operation, the client software performs operations such as application startup, opening system settings, volume adjustment, or display settings adjustment through the operating system interface.

[0076] IX. System Structure

[0077] See Figure 7This application also provides a mouse-triggered cross-application file context processing system, including a mouse 10 and an electronic device 20 running a Windows operating system. The mouse 10 includes a smart button 11, a microphone 12, a main control chip 13, and a communication module 14: the smart button 11 receives user-triggered operations, the microphone 12 collects user voice, and the main control chip 13 sends button status signals and / or voice signals to the electronic device 20 via the communication module 14. The electronic device 20 runs client software 21 and a resident context capture process 22 started by the client software 21, and has a context cache 23. The resident context capture process 22 performs the aforementioned foreground window detection, application type determination, target object candidate acquisition, and unified context object construction, and writes the unified context object to the context cache 23 and sends a structured context update message via inter-process communication when the unified context object changes. The client software 21 receives natural language instructions in response to the trigger event of the smart button 11, reads the target context object or empty context object from the context cache 23, generates a structured operation intent via the intent parser 26, and the operation routing module 27 selects the corresponding execution module 28. The execution module 28 includes at least one of the following modules: file format conversion module, multi-file merging module, document parsing module, question and answer summary module, web page content processing module, content generation module, document write-back module, and system operation module.

[0078] 10. Application Examples

[0079] For ease of understanding, several application examples are given below, which do not constitute a limitation of this application.

[0080] Example 1: A user selects a document file in File Explorer, long-presses the smart button and says "Convert to PDF". The context type is File Explorer selected file and the intent type is format conversion. The program is routed to the file format conversion module to output a PDF file.

[0081] Example 2: The user selects multiple PDF files and says "merge into one", which routes to the multi-file merging module to output the merged file.

[0082] Example 3: The user's foreground is a browser application. The user says "summarize this webpage". The target context object is the current webpage address. The webpage content processing module retrieves the main text and the question-and-answer summary module outputs a summary.

[0083] Example 4: The user's front end is a document editing application. The user says "Add a conclusion at the end of the document". The intent type is document write-back. The document write-back module writes the content generated by the large model at the end of the document.

[0084] Example 5: A user selects a document file (e.g., a Word document), a spreadsheet file (e.g., an Excel spreadsheet), a Markdown file, and a text file simultaneously in File Explorer. They then long-press the smart button and say, "Summarize these files into a single PPT slide." The resident context capture process obtains the paths to these multiple files through the file manager interface and constructs a unified context object containing these file paths. The client software inputs the natural language command and the target context object into the intent parser, obtaining a structured operation intent with the intent type "Presentation Generation," the operation parameter "Summary Synthesis," and the page number "One Page." The content generation module calls the corresponding content extractor for each file type to extract the document text, table data, Markdown-level text, and text file content. The extracted results are then summarized and synthesized to generate a one-page presentation file.

[0085] Example 6: When a user is on the desktop with no file selected, they briefly press the smart button and enter "increase screen brightness". The target context object is an empty context object and the intent type is system operation. The system operation module is then routed to perform the display settings adjustment.

[0086] Example 7: The user selects several files and says "Generate a mind map of these contents". The mind map generation module generates a hierarchical outline from the large model based on the extracted content and renders it as a mind map file.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for handling cross-application file context triggered by a mouse, applied to an electronic device running a Windows operating system and communicating with a mouse, wherein the mouse includes smart buttons and the electronic device runs client software, characterized in that, include: The client software initiates a persistent context capture process, which periodically acquires the current foreground window and obtains the window identifier, window title, window class name, process identifier, and process name of the current foreground window. Based on at least one of the window title, window class name, process identifier, and process name, the application type corresponding to the current foreground window is determined. Candidate targets are obtained by performing at least one context retrieval method based on application type and / or window state, the method including: obtaining the currently active file path through an application automation interface or file handle enumeration; obtaining the current webpage address of the browser through an accessibility interface; obtaining the selected file path of the file explorer through a file manager interface; obtaining the selected file path of the desktop through cross-process reading of the desktop list control; A unified context object is constructed based on the target object candidates, and the target context object is determined based on at least one of the source type, foreground window state, and detection time. An empty context object is constructed when no candidates are obtained. When the unified context object changes, it is written to the cache and a structured context update message is sent via inter-process communication; In response to a smart key press, the client software obtains a natural language instruction, reads the target context object or an empty context object from the cache, inputs it into the intent parser to obtain a structured operation intent, and routes it to the execution module according to the context type and intent type to perform at least one of the following: file processing, web page content processing, document content processing, content generation, document write-back, or system operation.

2. The method according to claim 1, characterized in that, The mouse also includes a microphone, and the trigger events of the mouse smart buttons include long press events and short press events; When the triggering event is a long press event, the client software activates the mouse's microphone to collect the user's voice, and converts the user's voice into the natural language command after the smart button is released; When the triggering event is a short press event, the client software pops up a text input window and uses the text entered by the user in the text input window as the natural language command; The voice command corresponding to the long press event and the text command corresponding to the short press event both reuse the target context object determination process and operation routing process described in claim 1.

3. The method according to claim 1, characterized in that, The persistent context capture process calls the Windows window interface to obtain the window identifier, window title, window class name, process identifier, and process name of the current foreground window, and determines the application type corresponding to the current foreground window according to the preset application type table; The preset application type table includes the correspondence between process name, window class name, window title keywords, application type, and context acquisition method; The application types include at least one of the following: document editing applications, table editing applications, presentation editing applications, PDF reading applications, browser applications, image processing applications, code editing applications, and text editing applications; When the application type corresponding to the current foreground window is a document editing application, a table editing application, or a presentation editing application, the resident context capture process obtains the current active file path through the component object model interface; When the application type corresponding to the current foreground window is a browser application, the resident context capture process searches for the browser address bar control through the accessibility interface and determines the value in the address bar control as the browser's current webpage address.

4. The method according to claim 1, characterized in that: The step of obtaining the path of the selected file in File Explorer through the file manager interface includes: Determine whether the current foreground window is a Windows Explorer window; When the current foreground window is a file explorer window, the folder view corresponding to the file explorer window is obtained through the Shell.Application component object model interface; Call the SelectedItems method of the folder view to get one or more file paths selected by the user in the Explorer window; The method of obtaining the path of the selected file on the desktop through cross-process reading of the desktop list control includes: Locate the Desktop View window under the Desktop Management window, and then locate the Desktop List control within the Desktop View window; If the desktop list control is not found in the desktop management window, the working window that carries the desktop view is traversed, and the desktop view window and its desktop list control are searched in the working window. Determine the File Explorer process that hosts the desktop list control; Allocate cross-process memory within the resource manager process; Send a list item read message to the desktop list control; The text of the selected desktop icon is read from the resource manager process, and the desktop file path or shortcut target path is determined based on the text.

5. The method according to claim 1 or 3, characterized in that, When the current active file path cannot be obtained through the application automation interface, the resident context capture process obtains the current active file path through file handle enumeration; The file handle enumeration includes: Enumerate the file handles opened by processes in the system; Filter the file handles of the target processes based on the process identifier; Copy the file handle of the target process to the current process; Query the object path corresponding to the handle of the copied file; Convert the device path in the object path to a Windows file system path; Extract candidate filenames from the window title, and determine the current active file path corresponding to the current foreground window based on the matching results of the candidate filenames and the filenames in the Windows file system path; When the current active file path is not obtained through file handle enumeration, the resident context capture process performs command line parameter parsing and / or recent file record lookup to determine the current active file path based on the candidate file names extracted from the window title.

6. The method according to claim 1, characterized in that, The unified context object includes at least the contextId field, contextType field, sourceType field, hwnd field, pid field, and timestamp field; the unified context object also includes at least one of the pathList field, url field, processName field, windowTitle field, priority field, and confidence field; The contextType field is used to represent the context type, which includes at least one of the following: currently active file, file selected in File Explorer, file selected on the desktop, current webpage address, and empty context. The sourceType field is used to indicate the context source, which includes at least one of the following: component object model interface, Shell.Application interface, desktop list control, accessibility interface, file handle enumeration, command line argument parsing, and recent file record reverse lookup; Determining the target context object based on at least one of the following: source type, foreground window state, and detection time, includes: When the current foreground window is the resource manager window and there is a selected file path in the resource manager, the candidate object corresponding to the selected file path in the resource manager is determined as the target context object; When the current foreground window is a desktop window and there is a selected file path on the desktop, the candidate object corresponding to the selected file path on the desktop is determined as the target context object; If there is no selected file path in the foreground state but there is a currently active file path, the candidate object corresponding to the currently active file path will be determined as the target context object; If there is no selected file path and currently active file path in the foreground state, but the current webpage address of the browser exists, the candidate object corresponding to the current webpage address of the browser will be determined as the target context object; If the path to the selected file in File Explorer, the path to the selected file on the desktop, the path to the currently active file, and the current webpage address in the browser do not exist, an empty context object will be determined as the target context object; The resident context capture process performs current foreground window detection and target object candidate acquisition according to a preset period. When the unified context object changes, it sends a structured context update message to the user interface process of the client software through inter-process communication. When the unified context object does not change, it does not send duplicate structured context update messages.

7. The method according to claim 1, characterized in that, The structured operation intent includes at least an intent type and operation parameters; the structured operation intent also includes at least one of the following: target object identifier, target file path list, target webpage address, output type, output location, write location, confidence level, and whether user confirmation is required; The intent types include at least one of the following: question and answer, summary, format conversion, multi-file merging, presentation generation, mind map generation, document write-back, pure command content generation, or system operation; The client software performs routing based on the context type of the target context object or empty context object and the intent type of the structured operation intent, including: When the context type is currently active file, file selected in File Explorer, or file selected on the desktop, and the intent type is format conversion, the route is routed to the file format conversion module; When the context type is "File selected in Explorer" or "File selected on desktop", and the target file path list includes multiple file paths, and the intent type is "Multi-file merge", the route is routed to the multi-file merge module. When the context type is the current active file and the intent type is question and answer or summary, the route is routed to the document parsing module and the question and answer summary module. When the context type is the current webpage address and the intent type is question and answer or summary, the route is routed to the webpage content processing module and the question and answer summary module. When the context type is the current active file, the file selected in the file explorer, the file selected on the desktop, or the current webpage address, and the intent type is presentation generation or mind map generation, the route is routed to the corresponding content generation module. When the context type is an empty context and the intent type is pure instruction content generation, the route is routed to the content generation module; When the context type is "current active file" and the intent type is "document write-back", the route is routed to the document write-back module. When the context type is empty context and the intent type is system operation, the route is routed to the system operation module; Wherein, when the context type is file selected in Explorer or file selected on the desktop, the target file path list includes multiple file paths of different file types, and the intent type is presentation generation, the content generation module calls the corresponding content extractor according to the file type corresponding to each file path, extracts the file content of the multiple different file types, and performs summary synthesis on the extracted content according to the operation parameters to generate one or more page presentation files; The different file types are determined by at least one of file extension, MIME type, or content extractor type; the multiple different file types include at least two of document files, table files, presentation files, PDF files, Markdown files, text files, and image files; The operation parameters include at least one of the following: presentation page count, abstract granularity, output topic, and output format; When the intent type is document write-back, the client software obtains the content of the currently active document based on the target context object, generates content to be written based on the natural language instruction and the content of the currently active document, and writes the content to be written into the currently active document through at least one of the component object model interface, official plugin interface, accessibility interface or input simulation interface.

8. A mouse-triggered cross-application file context processing system, characterized in that, This includes mice and electronic devices running the Windows operating system; The mouse includes a smart button, a microphone, a main control chip, and a communication module. The smart button is used to receive user-triggered operations, the microphone is used to collect user voice, and the main control chip is used to send button status signals and / or voice signals to the electronic device through the communication module. The electronic device runs client software and a persistent context capture process initiated by the client software; The persistent context capture process is used to periodically acquire the current foreground window and acquire the window identifier, window title, window class name, process identifier, and process name of the current foreground window. Based on at least one of the window title, window class name, process identifier, and process name, the application type corresponding to the current foreground window is determined. Based on the application type and / or the current window state, target object candidates are acquired. A unified context object is constructed based on the target object candidates. When the unified context object changes, it is written to the context cache and a structured context update message is sent through inter-process communication. The client software is used to obtain natural language instructions input by the user in response to the triggering event of the smart mouse button, read the target context object or empty context object from the context cache, generate a structured operation intent based on the natural language instructions and the target context object or empty context object, and select the corresponding execution module based on the context type of the target context object or empty context object and the intent type of the structured operation intent. The execution module is used to perform at least one of the following: file processing, web page content processing, document content processing, multi-file content aggregation processing, content generation processing based on target context object, document write-back processing, or system operation. Wherein, when the target context object includes multiple file paths of different file types, and the intent type of the structured operation intent is presentation generation, the execution module extracts content from the multiple files of different file types respectively, and synthesizes the extracted content into a summary according to the operation parameters in the structured operation intent to generate one or more page presentation files; The system is used to perform the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

Images