Browser AI integration system

Abstract

This approach avoids the inconvenience of having to use a dedicated automation browser or specific tool to integrate with the AI ​​agent, prevents vendor lock-in to AI providers, and suppresses the exposure of unauthorized content. [Solution] The browser operation integration function 10 interacts with the browser as an in-browser function or as an external operation integration unit. The Gateway module 20 is an intermediary function that performs request relaying, target verification, sharing permission management, audit recording, etc., and can be deployed within the browser, add-on mechanism, AI execution environment, native application, server, or cloud environment. The Gateway module 20 is available from the AI ​​agent 90 in an invocation format that is independent of specific AI services, AI models, execution environments, or tool protocols. Only targets explicitly permitted for sharing by the user are treated as targets for operation or annotation, and if the acquired target identification information does not correspond to the state at the time of permission, the sharing permission is revoked.

Description

Technical Field

[0002] ,

[0001] The present invention relates to a browser cooperation technology that provides the content of a tab on a browser to an AI agent or an automation program and enables the execution of browser operations from the AI agent or the automation program. In particular, the present invention relates to a browser AI cooperation technology that performs cooperation with an AI agent only for tabs explicitly permitted by a user, automatically invalidates the sharing permission in response to changes from the acquisition target identification information at the time of permission, and further visualizes the reference target or operation target of the AI agent on the tab by an annotation layer. In the present invention, the Gateway module refers to an intermediary function that performs at least one of relay of requests, target confirmation, sharing permission management, conversion, audit recording, or processes equivalent thereto between the browser operation cooperation function and the AI agent. The browser, the AI agent, and the Gateway module may be arranged on the same information processing device, or may be distributed and arranged on a plurality of information processing devices capable of communicating with each other. The Gateway module may be implemented in the browser, in the add-on mechanism of the browser, in the operating system, in the AI agent execution environment, in the native application, on the server, in the cloud environment, or as a combination thereof.

Background Art

[0002] In recent years, technologies have been used in which AI agents read information on a browser and assist with operations such as form input, clicking, scrolling, and file attachment. Commercial browsers or browser operation integration functions include configurations that send page content to an AI service on the cloud, or configurations that summarize or manipulate the current page using the browser's built-in AI functions. For example, Non-Patent Documents 1 to 4 describe technologies in which AI refers to page content or assists with operations on the browser via a browser or browser extension. Furthermore, automation frameworks such as headless browsers, automated test drivers, and remote control frameworks are widely used to control browsers from programs. In addition, to allow users to point out objects on the screen in a manner similar to human conversation, the experience of adding annotations to screenshots or browser screens and having the AI ​​agent refer to those annotations is becoming important. For example, Patent Documents 1 to 5 and Patent Document 7 describe technologies related to AI, generative AI, business screens, UI elements, RPA, automated operations, or operation assistance. Patent Document 6 also describes technologies for creating, organizing, displaying, or connecting document annotations within a web browser.

[0003] On the other hand, existing browser AI integration technologies do not adequately provide a configuration that explicitly limits which tabs are shown to the AI ​​agent on a tab-by-tab basis, while still using the user's usual browser profile and login status, and that automatically revokes the sharing permission upon changes in the identification information of the tab being acquired, closing the tab, or explicit revocation. Furthermore, configurations that grant comprehensive access permissions to the browser operation integration function, or configurations that send page content to the relay server without tab-by-tab sharing permission boundaries, leave a risk of leakage of browsing history, business data, customer information, authenticated page content, etc. In addition, if it is not recorded which browser content or converted content is sent back to the AI ​​agent, and which operation requests or operation commands pass through the Gateway, it becomes difficult to isolate the cause in the event of information leakage, operation error, or prompt injection.

[0004] In the past, there have been cases where convenient browser operation integration features have transmitted browsing data due to acquisitions, changes in ownership, or policy changes. Therefore, in the integration of browsers and AI agents, it is important to have code auditability, controllability of communication paths and data destinations, recording of operation history, technical boundaries that make tabs other than those selected by the user invisible to the AI ​​agent, and feedback that allows the user to visually confirm what the AI ​​agent is trying to reference or manipulate. In AI browsers or browser-equivalent environments, the behavior of host access permissions for standard browser add-on APIs may differ from that of general browsers, and interoperability that allows for stable display of annotation layers across multiple execution paths is also required. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Patent No. 7827819 [Patent Document 2] Japanese Patent Publication No. 2025-118097 [Patent Document 3] Japanese Patent Publication No. 2025-160869 [Patent Document 4] Japanese Patent Publication No. 2025-161728 [Patent Document 5] Japanese Patent Publication No. 2025-097253 [Patent Document 6] Japanese Patent Publication No. 2025-098236 [Patent Document 7] Japanese Patent Publication No. 2026-045283 [Non-patent literature]

[0006] [Non-Patent Document 1] OpenAI, Browser AI related announcement, https: / / openai.com / index / introducing-chatgpt-atlas /

[0007]

Non-licensed Document 2

[0008] [Non-licensed document 3] Google, Gemini in Chrome, https: / / www.google.com / chrome / ai /

[0009]

Non-licensed Document 4

[0010]

Non-licensed Document 5

[0011]

Non-licensed Document 6

[0012]

Non-licensed Document 7

[0013]

Non-licensed Document 8

[0014]

Non-Patent Document 9

[0015]

Non-Patent Document 10

[0016]

Non-Patent Document 11

[0017]

Non-Patent Document 12

[0018]

Non-Patent Document 13

[0019]

Non-Patent Document 14

[0020]

Non-Patent Document 15

Summary of the Invention

Problems to be Solved by the Invention

[0021] Existing commercial browser AI integration technologies may include structures that send page content to cloud services or integrate with the provider's backend. Such structures may lack sufficient technical boundaries to limit the target to only the tabs explicitly selected by the user, control over communication paths and destinations, and auditability regarding the transmission and reception of browser content, converted content, operation requests, and responses.

[0022] Existing browser automation frameworks or browser operation sharing tools, such as Playwright MCP and Browser-use, are primarily intended for testing, scraping, or temporary automation purposes. These often utilize dedicated browser instances, remote debugging ports, or automation profiles. Therefore, they are inconvenient for users to easily share their everyday browser profiles, login status, add-on features, and work screens with an AI agent. Furthermore, they lack a built-in permission lifecycle that allows users to explicitly select what to show the AI ​​agent on a tab-by-tab or display-by-display basis, and automatically hides shared content when it transitions to a different acquisition target than the one permitted.

[0023] Furthermore, in a configuration that grants comprehensive page access permissions to the browser operation integration function, there is a possibility that the program communicating with that function may access pages unintended by the user. Even if an AI agent requests the contents of an unshared tab through prompt injection or other means, it is difficult to prevent unintended information leakage without technical boundaries on a tab-by-tab basis.

[0024] The object of the present invention is to reduce the inconvenience of routinely switching between dedicated automation browser environments or specific browser automation frameworks, and to provide a browser-based AI integration system that allows users to flexibly interact with one or more AI agents using their browser without being locked into specific AI services, AI providers, AI models, agent execution environments, client applications, or tool protocols. Furthermore, the object of the present invention is to share only the objects explicitly permitted by the user with AI agents, to revoke sharing permission in response to changes in the acquired object identification information of the object, termination of the object, or explicit revocation, to allow users to specify areas, text, or structured elements as annotations on permitted shared objects, and to make the transmitted and received content and operation requests passing through the Gateway auditable. [Means for solving the problem]

[0025] A first aspect of the present invention is a browser-AI collaboration system that links a browser with one or more AI agents. The Gateway module is an intermediary function that performs at least one of the following processes between the browser operation collaboration function and one or more AI agents: request relaying, target verification, sharing permission management, conversion, audit recording, or equivalent processing. The browser, one or more AI agents, and Gateway module may be located on the same information processing device, or they may be distributed across multiple information processing devices that can communicate with each other. The Gateway module may be implemented within the browser, within the browser's add-on mechanism, within the operating system, within the AI ​​agent execution environment, within a native application, on a server, in a cloud environment, or as a combination thereof. The system comprises a browser operation collaboration function that is attached to a browser, operates within the browser, or collaborates with the display screen or operation target of the browser or a browser-equivalent environment, and a Gateway module that collaborates with the browser operation collaboration function via a communication path or calls within the same execution environment, and relays requests from one or more AI agents to the browser operation collaboration function. The Gateway module can be used by one or more AI agents through an invocation format that is independent of specific AI services, AI providers, AI models, agent execution environments, client applications, or tool protocols. The browser operation integration function and the Gateway module enable browsers to interact with one or more AI agents without requiring a specific browser automation framework or a dedicated automation browser environment.

[0026] In a second aspect of the present invention, the browser operation linkage function registers a specific tab, window, display area, or operation target as a shared-permitted target in the Gateway module only when the user performs an explicit sharing permission operation for that specific tab, window, display area, or operation target. The Gateway module forwards requests received from one or more AI agents to the browser operation linkage function only when the requests target a shared-permitted target. The browser operation linkage function revokes the sharing permission for a shared-permitted target if it detects that the current acquired target identification information of the shared-permitted target no longer corresponds to the acquired target identification information recorded at the time of the explicit sharing permission operation, that the shared-permitted target has been closed, terminated, or become unavailable, or that the user has performed an explicit cancellation operation. Furthermore, the browser operation linkage function does not require permanent or comprehensive access rights targeting all or an unspecified number of pages in the permission mechanism provided by the browser or an equivalent browser linkage permission mechanism, but instead acquires access to the shared-permitted target through temporary or limited-scope access rights limited to the shared-permitted target based on user operation.

[0027] In a third aspect of the present invention, the browser operation linkage function executes read operations only on the condition that the target is a shared-permitted target, and for write operations that can change the display content, input state, transition state, or external input / output state of a shared-permitted target, the content of the write operation is presented to the user and executed only after the user's approval is obtained.

[0028] In a fourth aspect of the present invention, the communication path includes at least one of the following: a wired connection, a USB connection, a short-range wireless connection, a Bluetooth connection, a local network, a private network, a virtual private network, the Internet, or a combination thereof. When the Gateway module operates on the same information processing device as the browser, the Gateway module limits the waiting for communication with the browser operation cooperation function to the same device's communication path or local communication interface, and further verifies that the source identification information or authentication information included in the connection request indicates an authorized browser cooperation source, and rejects connection requests from unauthorized sources. The Gateway module records the operation history, including granting sharing permission, revoking sharing permission, read operations, annotation operations, and write operations, as well as at least part or a summary of browser content, converted content, operation requests, annotation requests, operation commands, or response content, as an audit record. Furthermore, the Gateway module or the audit record unit may record, present, conceal, summarize, or reject the content based on predetermined audit conditions before it is provided to one or more AI agents or transferred to the browser operation cooperation function. Audit records are stored on an information processing device where a browser, one or more AI agents, or a Gateway module is located, or on a storage device capable of communicating with these, and access to the audit records is limited to authorized personnel.

[0029] In a fifth aspect of the present invention, the browser operation cooperation function performs at least one of the following processes: using at least one of a control mechanism, cooperation mechanism, or operation assistance mechanism provided by the browser to perform at least one of the following on a shared permission-granted target: information acquisition, display control, input control, external resource specification, and annotation layer execution; using an external operation cooperation unit that cooperates with the display screen, window, tab, or target element of the browser or browser-equivalent environment without going through the browser's add-on mechanism to identify the target of operation based on at least one of the acquisition target identification information, display target identification information, display area information, or location information corresponding to the shared permission-granted target, and executing an operation request or annotation request received from the Gateway module limited to that target of operation; and if execution by the first browser cooperation mechanism is refused due to browser permissions or execution environment restrictions, the second browser cooperation mechanism performs the process on the shared permission-granted target.

[0030] In a sixth aspect of the present invention, a Gateway module or an agent operation interface that works in conjunction with a Gateway module performs at least one of the following in a format that can be invoked by one or more AI agents: acquiring, reading, annotating, manipulating, managing sharing permissions, acquiring operation history, and assisting with repetitive operations for browser content or content related to shared permission targets; a Gateway module performs a predetermined conversion process before returning the browser content or content related to shared permission targets received from the browser operation linkage function to one or more AI agents; and a Gateway module reads a conversion definition including target conditions and conversion process specification information, and executes a conversion process corresponding to the target conditions if the acquired target identification information of the shared permission target matches the target conditions.

[0031] In a seventh aspect of the present invention, a Gateway module or an agent operation interface that works in conjunction with a Gateway module provides a pre-execution presentation process that records a series of operations on a browser or browser-equivalent environment display screen or target as an operation sequence, presents the operation sequence to the user before execution, and re-execution process that re-executes the operation sequence on the display screen or target, thereby enabling the generation and re-execution of operation sequences used for repeated operation verification or automated operation verification.

[0032] In an eighth aspect of the present invention, the browser operation linkage function draws an annotation layer on the shared permission target, accepts area specification by the user, text selection by the user, or annotation addition requests from at least one AI agent as annotation input, and visually displays identification indicators and comments on the shared permission target. The browser operation linkage function classifies annotations related to area specification into structural element annotations or visual area annotations, classifies annotations related to text selection into text annotations including re-identification information, and stores structural element annotations, visual area annotations, or text annotations as annotation data including at least one of location information, display target identification information, comments, creation time, target element information, string information, or display attribute information.

[0033] A ninth aspect of the present invention is a method corresponding to the system. A tenth aspect of the present invention is a program that causes one or more computers to execute the method. [Effects of the Invention]

[0034] According to the present invention, since only the objects explicitly permitted by the user are visible to the AI ​​agent, the exposure of unintended page content can be suppressed compared to configurations that assume bulk permission for the entire browser or an unspecified number of pages.

[0035] According to the present invention, if a shared item that has been permitted to be shared no longer corresponds to the item identification information obtained at the time of permission, the sharing permission is automatically revoked. This prevents the AI ​​agent from continuing to view or manipulate the item even after the user has moved to another site or a confidential site.

[0036] According to the present invention, the browser, AI agent, and Gateway module can be applied not only to configurations where they are located on the same information processing device, but also to configurations where they are distributed across smartphones, tablets, desktop terminals, cloud servers, resident services in a cloud environment, servers on a private network, etc. Furthermore, the Gateway module is not limited to an independent local application, but can be implemented within a browser, within a browser's add-on mechanism, within an operating system, within an AI agent execution environment, within a native application, on a server, in a cloud environment, or a combination of these.

[0037] According to one embodiment of the present invention, the Gateway module operates only within the same device's internal communication channel, and the source identification information or authentication information of connection requests is limited to authorized browser partners. This reduces the attack surface where unauthorized sources send messages to the Gateway module.

[0038] According to the present invention, the browser operation linkage function does not require permanent or comprehensive access rights to an unspecified number of pages, but accesses only the necessary scope to the shared-permitted targets, thus minimizing the permissions of the browser operation linkage function. Furthermore, even when using an external operation linkage unit, operation requests or annotation requests can be executed only to the display area or operation target corresponding to the shared-permitted targets.

[0039] According to the present invention, an annotation layer can be drawn on shared-permitted objects, and regions, structured elements, text selections, and annotation input originating from the AI ​​agent can be structured and sent back to the AI ​​agent. This allows users to work while visually confirming the objects referenced or manipulated by the AI ​​agent. Furthermore, since audit records, write operation approvals, and conversion processes are managed on the Gateway module side, it is possible to track operations in business use, minimize data, remove noise by object, verify transmitted and received content afterward, and isolate the cause of information leakage or erroneous operations. [Brief explanation of the drawing]

[0040] [Figure 1] This is an overall configuration diagram of a browser-AI integration system according to one embodiment of the present invention.

[0041] [Figure 2] This is a state transition diagram showing the lifecycle of sharing permissions.

[0042] [Figure 3] This sequence diagram shows the automatic revocation of sharing permissions when the target identification information changes.

[0043] [Figure 4] This is a sequence diagram showing the message flow from the agent operation interface to the browser tab.

[0044] [Figure 5] This is a comparison diagram between a comprehensive access permission add-on function and a tab-based limited access configuration according to the present invention.

[0045] [Figure 6] This is a diagram showing the plug-in architecture in the Gateway module.

[0046] [Figure 7] This diagram shows the token reduction effect when tab content is converted to Markdown.

[0047] [Figure 8] This is a diagram showing the annotation layers and annotation data generation flow drawn on the shared tab.

[0048] [Figure 9] This flowchart illustrates the source verification and audit record permission settings for connection requests in the Gateway module. [Modes for carrying out the invention]

[0049] The following describes embodiments of the present invention. The following embodiments are illustrative, and the present invention is not limited to these configurations.

[0050] As shown in Figure 1, the browser AI collaboration system 1 of this embodiment comprises a browser operation collaboration function 10, a Gateway module 20, an agent operation interface 30, a browser tab 40, a plug-in 50, an audit record 60, a permission management unit 70, a communication channel 80, an AI agent 90, an annotation layer 100, and a source verification unit 120. In this specification, "browser" includes, for example, web browsers such as Chrome, Safari, and Firefox, mobile browsers, and browser-equivalent environments. The browser operation collaboration function 10 also includes browser add-ons, mobile browser add-ons, additional functions built into the browser, an external operation collaboration unit that collaborates with automation APIs or accessibility APIs provided by the browser, or a browser collaboration mechanism equivalent thereto. The browser operation collaboration function 10 accepts explicit sharing permission operations from the user for the browser tab 40. The Gateway module 20 acts as an intermediary function between the browser operation collaboration function 10 and the agent operation interface 30, and the permission management unit 70 manages the tabs that have been granted sharing permission. The source verification unit 120 verifies the source of connection requests via the communication channel 80, and the annotation layer 100 is drawn on the shared permission tab. The Gateway module 20 is not limited to an independent local process, but may be implemented within a browser, within a browser add-on mechanism, within an operating system, within an AI agent execution environment, within a native application, on a server, in a cloud environment, or a combination thereof. The agent operation interface 30 is invoked from the AI ​​agent 90 by command line, API, message, function call, or equivalent calling format. First Embodiment

[0051] In the first embodiment, the browser 40, AI agent 90, and Gateway module 20 operate on the same desktop computer. The browser is, for example, Chrome, Safari, or Firefox. The Gateway module 20 is implemented as a native application residing in the menu bar or system tray. As an example of an internal communication path, the Gateway module 20 listens on a loopback address or a WebSocket endpoint on a predetermined local communication interface for communication with the browser operation cooperation function 10, and can use a Unix Domain Socket for communication with the agent operation interface 30. The file permissions of the Unix Domain Socket are set to allow access only by the owner user.

[0052] The browser operation integration function 10 is configured according to the browser-specific add-on API, browser integration API, or external operation integration unit, and uses temporary or scope-limited tab access permissions based on user operations, script execution API, tab management API, storage API, debug API, timer API, automation API, or accessibility API, etc. On the other hand, the browser operation integration function 10 does not request permanent or comprehensive access permissions for all or an unspecified number of pages. When a user performs an operation equivalent to "Share this tab" in the add-on UI, browser built-in UI, or external operation integration unit UI, the browser operation integration function 10 sends information identifying the tab, the URL of the tab, acquisition target identification information such as origin, display target identification information such as title, and the time of sharing permission to the Gateway module 20.

[0053] The permission management unit 70 stores the tab identification information, linkage function identifier, acquisition target identification information, display target identification information, acquisition target identification information at the time of permission, and the time of permission for sharing of the permitted tab. When an operation request is sent from the AI ​​agent 90 via the agent operation interface 30, the Gateway module 20 determines whether the target tab is registered with the permission management unit 70. If it is not registered, the Gateway module 20 rejects the operation request and does not forward it to the browser operation linkage function 10.

[0054] As shown in Figure 2, the sharing permission status includes the non-shared state, the sharing permission operation in progress state, the shared state, and the expired state. In the non-shared state, the tab content cannot be obtained from the AI ​​agent 90. In the sharing permission operation in progress state, the user performs the sharing operation via a pop-up or the like. In the shared state, it is possible to read, annotate, or manipulate the shared-permitted tab. In the expired state, the tab is no longer a shared-permitted tab.

[0055] As shown in Figure 3, the browser operation integration function 10 monitors URL update events or navigation events of the shared-permitted tab 40. If it detects that the current target identification information of the shared-permitted tab 40 no longer corresponds to the target identification information obtained at the time of permission, the browser operation integration function 10 deletes the sharing permission and sends an expiration notification to the Gateway module 20. The Gateway module 20 deletes the tab from the permission management unit 70 and records the reason for expiration in the audit record 60. Similarly, if the tab is closed or explicitly revoked by the user, the sharing permission is deleted and the reason for expiration is recorded in the audit record 60.

[0056] As shown in Figure 4, the AI ​​agent 90 invokes the agent operation interface 30 by means of a command, API call, message transmission, or function call corresponding to, for example, a read request for a shared permitted object. The agent operation interface 30 can send operation requests or annotation requests to the Gateway module 20 via the communication channel 80 as messages in JSON format or the like. The communication channel 80 may include Unix Domain Socket, named pipe, local TCP socket, WebSocket, HTTP, gRPC, standard input / output, USB connection, Bluetooth connection, short-range wireless connection, local network, private network, virtual private network, Internet, or OS-specific local or remote IPC. Furthermore, if the Gateway module 20 is implemented in the same execution environment as the browser operation cooperation function 10 or the agent operation interface 30, a function call, message queue, shared memory, or equivalent transfers within the same execution environment may be used instead of the communication channel 80. The Gateway module 20 checks for sharing permission for the target tab in the permission management unit 70, and if permission is granted, it forwards the operation request or annotation request to the browser operation cooperation function 10 via the communication channel 80 or a transfer within the same execution environment. The browser operation cooperation function 10 reads, annotates, or operates the browser tab 40 using the control mechanism, cooperation mechanism, or operation assistance mechanism provided by the browser, and returns the result to the AI ​​agent 90 via the reverse path.

[0057] Reading operations include reading the document structure, acquiring displayed images, acquiring operation logs, extracting structured data, describing target elements, acquiring communication history, and acquiring an annotation list. Writing operations include input, selection, transitions, changing display states, specifying external resources, and temporarily changing display content. For writing operations, the browser operation linkage function 10 can be configured to open an approval window presenting the user with the operation details, target tab, target identification information, input values, or location information, and execute the operation only if the user approves.

[0058] When using a debugging protocol or remote control protocol provided by the browser, the browser operation cooperation function 10 can use script evaluation commands for DOM reading or annotation layer execution, page image acquisition commands for screenshot acquisition, input event sending commands for clicks, scrolls, drags, and key presses, and file input element setting commands for file attachments. This allows operations to be performed only on tabs that have been granted sharing permission, without using permanent or comprehensive access permissions that target an unspecified number of pages.

[0059] The Gateway module 20 records all sharing permission grants, revocations, read operations, annotation operations, and write operations in the audit log 60. The audit log 60 can be stored in, for example, JSONL format, database format, event log format, or an equivalent recording format, and may include the time, target tab identifier, retrieved target identifier, operation type, revocation reason, and, if possible, the calling agent identifier. The audit log 60 may include at least one of the following for DOM, HTML, text, image information, network information or annotation data received from the browser operation linkage function 10, content converted by the plugin 50, operation requests or annotation requests received from the AI ​​agent 90, operation commands forwarded to the browser operation linkage function 10, or response content sent back to the AI ​​agent 90: full text, partial text, summary information, hash value, character count, conversion name, reason for concealment, or reason for rejection. Before returning the content to the AI ​​agent 90 or transferring it to the browser operation linkage function 10, the Gateway module 20 may conceal, summarize, and present to the user any confidential words, tokens, identifiers, personal information, or predetermined patterns, based on predetermined audit conditions, user settings, administrator settings, or determinations of the plugin 50, or refuse to transmit or transfer them. The audit record 60 may be stored in the information processing device on which the browser operates, the information processing device on which the AI ​​agent 90 operates, the information processing device on which the Gateway module 20 operates, or a storage device that can communicate with these.

[0060] As shown in Figure 5, in conventional comprehensive access permission add-on functions, the add-on function can continuously access a large number of acquisition targets, and a data leakage risk arises due to changes in ownership, updates, or backend integration. In contrast, in this embodiment, the browser operation integration function 10 does not request permanent or comprehensive access permissions for all or an unspecified number of pages, but targets only tabs that the user has allowed to share, and automatically revokes sharing permission if there is a mismatch in the acquisition target identification information. This combination makes it possible to technically limit the range of browser information that the AI ​​agent 90 can access.

[0061] As shown in Figure 6, the Gateway module 20 loads the plugin 50. The plugin 50 may be a JavaScript script running in the JavaScriptCore VM within the same process as the Gateway module 20. Alternatively, it may be an external script or executable file running in a separate process. The plugin 50 registers the transformation name using a host API equivalent to abg.registerTransform, and the Gateway module 20 performs the transformation before returning the tab content, such as HTML, obtained from the browser operation integration function 10 to the AI ​​agent 90.

[0062] The first example of Plugin 50 is markdown-plugin, which converts HTML to Markdown. The second example of Plugin 50 is a domain-specific plugin that, when it matches specific URL conditions for a web application, internal system, or website, removes sidebars, top bars, popovers, scripts, styles, administrative attributes, and other unnecessary elements, converting the main text or business-relevant areas into compact Markdown. In addition to conversions based on target conditions, Plugin 50 may also conceal sensitive terms, summarize content to be recorded in audit records 60, or reject transmissions or forwarding that meet predetermined conditions. The Gateway module 20 performs the domain-specific conversion before the generic Markdown conversion if the URL of a shared-permitted tab matches the domain conditions defined in the plugin.

[0063] The agent operation interface 30 may include a function to handle repetitive operation flows. Specifically, it records operations such as reading, taking screenshots, clicking, typing, sending keys, navigating, scrolling, dragging, attaching files, and waiting issued to shared-permitted tabs as an operation column, for example, in JSON format. It also provides a pre-execution presentation process that displays the recorded operation column without executing it, and a re-execution process that plays it back for shared-permitted tabs. This allows the AI ​​agent 90 or the user to use it for repetitive operation verification, automated operation verification, business procedure reproduction, and auditing of operation columns.

[0064] In this embodiment, the AI ​​agent 90 may be an interactive coding agent, an agent in an integrated development environment, an automation agent on a terminal, or an in-house developed agent, etc. The AI ​​agent 90 only needs to be able to call the agent operation interface 30 and is not limited to a specific AI service, a specific cloud API, a specific client implementation, or a specific calling format. In a configuration where skill files, procedure files, or tool definition files are installed, these files present the AI ​​agent 90 with instructions on how to use the agent operation interface 30. Second Embodiment

[0065] The second embodiment is an annotation layer integrated configuration. As shown in Figure 8, the browser operation cooperation function 10 draws an annotation layer 100 on the shared-permitted tab 40. The annotation layer 100 is configured as a layer separate from the normal DOM of the browser tab 40 and includes annotations with identifiers, comments, a toolbar, a draft rectangle for area selection, and a comment editing UI. The annotation layer 100 can only be started for the shared-permitted tab 40 and cannot be started from either the popup or the agent operation interface 30 on unshared tabs.

[0066] The annotation layer 100 has Area mode and Text mode. In Area mode, it accepts a rectangular area dragged by the user on the browser tab 40, or a point clicked, as annotation input. In Text mode, it accepts selected text as annotation input, even if the user selects text that spans multiple structured elements. The user can add comments to the annotation input and stop capturing the annotation layer 100 by clicking the Done button or pressing the Escape key. The annotation data 110 is retained even after stopping, and the AI ​​agent 90 can retrieve the annotation data 110 via the agent operation interface 30.

[0067] The browser operation integration function 10 determines whether a region specified in Area mode corresponds to a stable structured element, using the region's center point, the display rectangle of the target element, stable attributes, and a meaningful element type. If it corresponds to a stable structured element, the browser operation integration function 10 classifies the annotation as a structured element annotation and includes the target element identification information, identification quality, tag name, display string, color, background color, font size, and font family in the annotation data 110. If it is difficult to estimate a stable structured element, such as a canvas, video, or an ambiguous wrapper, the browser operation integration function 10 classifies the annotation as a visual region annotation and includes location information in the annotation data 110.

[0068] The browser operation integration function 10 includes, for selected text specified in Text mode, the selected text, root element identification information of the selected text, range information, highlighted fragments, and re-identification information including an index for re-identification in the annotation data 110. The text annotation is displayed on the annotation layer 100 as line-by-line highlights of the selected text, rather than as a rectangular frame. This allows the AI ​​agent 90 to receive the selection as pure text data rather than as a visual area.

[0069] Structural element annotations and text annotations are not moved or resized on the annotation layer 100 in order to maintain their ability to follow target element identification information or re-identification information. On the other hand, visual area annotations, which indicate any visual area, are allowed to be moved by dragging and resized by edge or corner handles. The browser operation coordination function 10 redraws the annotation layer 100 and updates the position information of each annotation in response to scrolling, resizing, scrolling within an iframe, or changes in the responsive layout.

[0070] The agent operation interface 30 provides start, stop, delete, list acquisition, structural element annotation addition using target element identification information, and region annotation addition using location information for the annotation layer 100. If the result of adding a region annotation is a visual region annotation, the agent operation interface 30 may save the image corresponding to that region. The return value of the annotation list includes at least one of the following: annotation type, comment, location information, target element information, selected string, re-identification information, acquisition target identification information, display target identification information, and creation time.

[0071] The annotation layer 100 can also be used for pre-verification of operations by the AI ​​agent 90. For example, if the AI ​​agent 90 suggests a button, logo, input field, or any visual area as a target for operation or replacement, the user can specify that target as an annotation, and the AI ​​agent 90 can use the target element information or location information contained in the annotation data 110 to perform subsequent reading, image acquisition, temporary display content modification, or approved write operations. This allows the AI ​​agent 90 to treat the target indicated by a human as "here" as structured target information.

[0072] When executing the annotation layer 100, the browser operation integration function 10 first attempts to execute it using the first browser integration mechanism. If this execution is denied due to permissions or execution environment restrictions in the AI ​​browser or browser-equivalent environment, the browser operation integration function 10 executes the annotation layer 100 on the shared-permitted tab 40 using the second browser integration mechanism. The first and second browser integration mechanisms include, for example, a browser add-on API, a browser integration API, a debugging protocol, a remote control protocol, or a script evaluation function of an equivalent browser control API. Third Embodiment

[0073] The third embodiment is a configuration that enhances security for communication within the same device. As shown in Figure 9, when the Gateway module 20 receives a connection request, before substantially accepting the connection and starting message processing, the source verification unit 120 verifies the source identification information or authentication information of the connection request. If the connection request is a WebSocket connection request, the source verification unit 120 determines that it is an authorized browser source only if, for example, the Origin scheme is one of chrome-extension, moz-extension, safari-web-extension, or a browser-specific scheme corresponding to the browser operation cooperation function, and the host exists and does not contain user information, paths, queries, and fragments.

[0074] The source verification unit 120 rejects the connection request if the source identification information or authentication information is missing, empty, indicates an unauthorized source such as a regular web page, or contains a malicious format that resembles an authorized browser connection source. This blocks the path by which an unauthorized source can operate the Gateway module 20 via the same device's internal communication channel.

[0075] The Gateway module 20 can set permissions so that only the owner user can access the application support directory, log directory, internal communication channels, and audit records 60. For example, the application support directory and log directory can be set to be accessible only to the owner user, and the audit records 60 can be set to be read and written only to the owner user. The audit records 60 are appended, for example, in JSONL format, and record read, write, annotation, sharing permission granting, revocation, connection status changes, summary information of content received or sent back by the Gateway module 20, hash value, reason for concealment, and reason for rejection.

[0076] The tests corresponding to the Gateway module 20 include allowing authorized browser sources, rejecting unauthorized sources or sources with missing source identification information, and rejecting malicious formats that resemble authorized browser sources. This allows the boundary conditions of the source verification unit 120 to be verified at the implementation stage. Fourth Embodiment

[0077] In the fourth embodiment, the browser may be a browser of multiple types such as Chrome, Safari, and Firefox, or a browser having an add-on mechanism equivalent thereto, additional functions built into the browser, or an external operation linkage function for the browser screen. The browser operation linkage function 10 is implemented according to the add-on API, browser linkage API, or external operation linkage API of each browser. If a debugging protocol, remote control protocol, or browser-specific debugging API provided by the browser is available, the browser operation linkage function 10 uses that protocol or API to read, annotate, or operate. If it is not available, the browser operation linkage function 10 reads, annotates, or operates by structured, limited script injection to the shared-permitted tab, or by external operation linkage limited to the display area or screen coordinates corresponding to the shared-permitted tab.

[0078] The Gateway module 20 may run on the same desktop operating system as the browser, on a mobile device such as a smartphone or tablet, on a cloud server, a resident service in a cloud environment, a server on a private network, an on-premises server, or an embedded environment. The Gateway module 20 may be implemented within a browser, within a browser add-on mechanism, within an operating system, within an AI agent execution environment, within a native application, on a server, in a cloud environment, or as a combination of these. The agent operation interface 30 may communicate with the Gateway module 20 using named pipes, local TCP sockets, WebSocket, HTTP, gRPC, standard input / output, USB connection, Bluetooth connection, short-range wireless connection, local network, private network, virtual private network, the internet, or OS-specific local or remote IPC, not limited to Unix Domain Sockets. For example, a mobile browser extension on a smartphone may operate as a browser operation cooperation function 10, and the Gateway module 20 may operate on the smartphone, a nearby desktop terminal, a server on a private network, or in a cloud environment. [Examples]

[0079] In Example 1, a user allows sharing of a Japanese CMS article open in a regular browser tab via the browser operation integration function 10 add-on UI, and in the implementation used in this example, the AI ​​agent 90 performs abg read <tab>The command equivalent to --selector article --format markdown was executed. For comparison, we used the acquisition of the entire page HTML using a browser automation framework, the acquisition of the innerHTML of the article element, and the acquisition of the innerText of the article element. Measurement was performed using the ABG repository benchmark on publicly available articles.

[0080] The measurement results showed that obtaining the entire page's HTML yielded 124,476 characters and approximately 50,000 tokens, preserving the structure but incurring significant overhead. Obtaining the `innerHTML` of the `article` element yielded 42,070 characters and approximately 17,000 tokens, preserving the structure but leaving markup overhead. The process equivalent to `abg read N --selector article --format markdown` in this example yielded 11,780 characters and approximately 5,900 tokens, obtaining a formatted output while preserving the structure. On the other hand, obtaining the `innerText` of the `article` element yielded 6,543 characters and approximately 3,800 tokens, but lost structure such as headings, links, and lists.

[0081] These results show that `abg read N --selector article --format markdown` reduced the number of characters by approximately 90.5% and the number of estimated tokens by approximately 88% compared to obtaining the entire page's HTML. On the other hand, unlike obtaining innerText, the structure such as headings, links, and lists was preserved. Therefore, this embodiment has the effect of not only providing secure communication limited to tabs that have been granted sharing permission, but also significantly reducing the amount of contextual information passed to the AI ​​agent 90. [Examples]

[0082] In Example 2, a general-purpose markdown-plugin and a domain-specific plugin were compared using a fixture for a knowledge base-type page. The domain-specific plugin has specific domain conditions in its plugin definition, and Gateway Module 20 matches the URL of the shared-permitted tab with these domain conditions before performing the domain-specific conversion before the general-purpose conversion.

[0083] The measurement results showed that the raw HTML of the knowledge base type page had 2,209 characters and approximately 553 tokens, while the general-purpose markdown-plugin reduced it to 1,242 characters and approximately 311 tokens, a reduction of approximately 44% compared to the raw HTML. Domain-specific conversion resulted in 709 characters and approximately 178 tokens, a reduction of approximately 68% compared to the raw HTML.

[0084] These results show that the domain-specific plugin 50 further reduced page-specific unnecessary elements than the generic HTML conversion, achieving approximately 68% token reduction compared to raw HTML. This allows the AI ​​agent 90 to focus on business-related content rather than noise such as application sidebars, popovers, scripts, and styles. [Explanation of Symbols]

[0085] 1. Browser AI Integration System

[0086] 10. Browser operation integration function

[0087] 20 Gateway Modules

[0088] 30 Agent Operation Interface

[0089] 40 browser tabs

[0090] 50 Plugins

[0091] 60 Audit Records

[0092] 70 Permit Management Department

[0093] 80 communication channels

[0094] 90 AI Agents

[0095] 100 Annotation Layers

[0096] 110 Annotation Data

[0097] 120 Source Verification Department [Industrial applicability]

[0098] This invention can be used for business support using AI agents, operation and maintenance of web applications, customer support, data entry, auditing, testing, compliance, internal business automation, and collaborative work with contractors or external partners. In particular, when dealing with authenticated business screens, customer data, management screens, internal knowledge bases, business screens on smartphones, etc., by sharing only tabs explicitly permitted by the user with the AI ​​agent, automatically revoking sharing permission when the acquired identification information changes, recording operation history and transmitted / received content passing through the Gateway, and visually confirming the target using an annotation layer, it is possible to introduce AI agent support while maintaining business data governance, regardless of the placement of the browser, AI agent, and Gateway module.< / tab>

Claims

1. A browser-AI collaboration system that connects a browser with one or more AI agents, A browser operation cooperation function that identifies and cooperates with a specific tab, window, display area, or operation target in at least one mode selected from the following: a mode added to the browser, a mode operating within the browser, a mode coordinating with the display screen of the browser, a mode coordinating with the operation target of the browser, a mode coordinating with the display screen of a browser-equivalent environment, and a mode coordinating with the operation target of a browser-equivalent environment, A Gateway module that communicates with the browser operation coordination function via a communication channel or calls within the same execution environment, and relays requests from one or more AI agents to the browser operation coordination function, Equipped with, The browser operation integration function registers a specific tab, window, display area, or operation target as a shared-permitted target in the Gateway module only when the user explicitly grants permission to share that specific tab, window, display area, or operation target. The Gateway module forwards the request to the browser operation integration function only if it confirms that the request received from one or more AI agents targets the shared permitted target. The browser operation linkage function revokes the sharing permission for the shared object if it detects that the currently acquired target identification information for the shared object no longer matches the target identification information acquired at the time of permission recorded at the time of the explicit sharing permission operation, that the shared object has been closed or has become unavailable for any other reason, or that the user has performed an explicit cancellation operation. The Gateway module does not forward requests targeting the object whose sharing permission has expired to the browser operation linkage function. The Gateway module is an intermediary function that performs at least one of the following between the browser operation integration function and the one or more AI agents: request relaying, target verification, sharing permission management, transformation, and audit recording. The browser, the one or more AI agents, and the Gateway module may be located on the same information processing device, or they may be distributed across multiple information processing devices capable of communicating with each other, and the Gateway module may be implemented within the browser, within the browser's add-on mechanism, within the operating system, within the AI ​​agent execution environment, within a native application, on a server, in a cloud environment, or as a combination thereof. The Gateway module is available from one or more AI agents via an invocation format that is independent of specific AI services, AI providers, AI models, agent execution environments, client applications, or tool protocols. The browser operation integration function and the Gateway module do not require a specific browser automation framework or a dedicated automation browser environment, and enable the browser to work in conjunction with the one or more AI agents. A system characterized by the following features.

2. In the system described in Claim 1, The browser operation linkage function registers the sharing permission information, including information identifying the shared-permitted target, information identifying the target to be acquired, information identifying the target to be displayed, and information identifying the target to be acquired at the time of permission, in the Gateway module at the time of the explicit sharing permission operation. The Gateway module determines, based on the shared permission information, whether the target of the request received from one or more AI agents is a shared permission target. The browser operation linkage function does not require permanent or comprehensive access rights targeting all or an unspecified number of pages in the authorization mechanism provided by the browser or an equivalent browser linkage authorization mechanism, but rather obtains access to the shared-permitted targets based on user operations using temporary or limited access rights restricted to the shared-permitted targets. A system characterized by the following features.

3. In the system described in claim 2, The aforementioned browser operation linkage function executes read operations only if the target is one of the shared-permitted targets. For write operations that can change the display content, input state, transition state, or external input / output state of the aforementioned shared-permitted target, the content of the write operation will be presented to the user, and the operation will only be executed after the user's approval. A system characterized by the following features.

4. In the system described in Claim 1, The aforementioned communication path includes at least one of the following: wired connection, USB connection, short-range wireless connection, Bluetooth connection, local network, private network, virtual private network, internet, or a combination of each of the aforementioned communication paths. When the Gateway module operates on the same information processing device as the browser, the Gateway module limits the waiting for communication with the browser operation cooperation function to the same internal communication path or local communication interface, verifies that the source identification information or authentication information included in the connection request from the browser operation cooperation function indicates an authorized browser cooperation source, and rejects connection requests from unauthorized sources. The Gateway module includes an audit record unit that records, as an audit record, at least part or a summary of the following: operation history including granting sharing permission, revoking sharing permission, read operations, annotation operations, and write operations; browser content received from the browser operation linkage function; content converted by the Gateway module; operation requests received from the one or more AI agents; annotation requests received from the one or more AI agents; operation commands forwarded to the browser operation linkage function; and response content sent back to the one or more AI agents. The audit record is stored on the information processing device on which the browser is located, the information processing device on which one or more AI agents are located, the information processing device on which the Gateway module is located, or on a storage device that can communicate with each of the information processing devices. The audit recording unit or the Gateway module performs at least one of the following actions based on predetermined audit conditions before the browser content, converted content, operation request, operation command, or response content is provided to the one or more AI agents or transferred to the browser operation linkage function: The audit record unit restricts access to the audit record to authorized entities corresponding to the browser, the one or more AI agents, or the Gateway module. A system characterized by the following features.

5. In the system described in claim 2, The browser operation linkage function includes a process that uses at least one of the control mechanism, linkage mechanism, or operation assistance mechanism provided by the browser to perform at least one of the following actions for the shared permission target: information acquisition, display control, input control, external resource specification, and annotation layer execution. Without going through the browser's add-on mechanism, an external operation linkage unit that interacts with the browser or a browser-equivalent environment's display screen, window, tab, or target element identifies the target based on at least one of the acquired target identification information, display target identification information, display area information, or location information corresponding to the shared permission target, and executes the operation request or annotation request received from the Gateway module limited to that target. If execution by the first browser integration mechanism is denied due to browser permissions or execution environment restrictions, the second browser integration mechanism performs processing on the shared permitted target, Perform at least one of the following: A system characterized by the following features.

6. In the system described in claim 2, The Gateway module or the agent operation interface that works in conjunction with the Gateway module provides at least one of the following in a format that can be invoked by the one or more AI agents: information acquisition, reading, annotation, manipulation, sharing permission management, operation history acquisition, and repetitive operation support for browser content or content related to the shared permission target; The Gateway module performs a process of converting the browser content received from the browser operation linkage function or the content related to the shared permission target by a predetermined conversion process before sending it back to the one or more AI agents. The Gateway module reads a conversion definition including target conditions and conversion process specification information, and if the target identification information of the shared permitted target matches the target conditions, it performs a conversion process corresponding to the target conditions. Perform at least one of the following: A system characterized by the following features.

7. In the system described in Claim 1, The Gateway module, or the agent operation interface that works in conjunction with the Gateway module, records a series of operations to be performed on the shared-permitted target as an operation sequence, provides a pre-execution presentation process that presents the operation sequence to the user before execution, and provides a re-execution process that, at the time of re-execution, confirms that the target of the operation sequence is the shared-permitted target and that the sharing permission is valid, thereby enabling the generation and re-execution of operation sequences used for iterative operation verification or automated operation verification. A system characterized by the following features.

8. In the system described in claim 2, The browser operation integration function draws an annotation layer on the shared permission granted object, The annotation layer accepts at least one of the following as annotation inputs: area selection by the user, text selection by the user, and annotation addition requests from one or more AI agents, and visually displays an identifier and comment corresponding to the annotation input on the shared-permitted object. The browser operation linkage function, when the annotation input is a region specification, classifies it as a structural element annotation if it can estimate the target element corresponding to the structured element in the browser, and classifies it as a visual region annotation if it cannot estimate the target element. When the aforementioned annotation input is a text selection, it is classified as a text annotation containing the selected string and re-identification information for re-identifying the selected string. The structural element annotation, the visual area annotation, or the text annotation is stored as annotation data that includes at least one of location information, display target identification information, comments, creation time, target element information, string information, or display attribute information. A system characterized by the following features.

9. A method for coordinating a browser and one or more AI agents via a browser operation coordination function and a Gateway module, The Gateway module is an intermediary function that performs at least one of the following between the browser operation integration function and the one or more AI agents: request relaying, target verification, sharing permission management, transformation, and audit recording. The browser, the one or more AI agents, and the Gateway module may be located on the same information processing device, or they may be distributed across multiple information processing devices capable of communicating with each other, and the Gateway module may be implemented within the browser, within the browser's add-on mechanism, within the operating system, within the AI ​​agent execution environment, within a native application, on a server, in a cloud environment, or as a combination thereof. The browser operation integration function includes the step of accepting an explicit sharing permission request from the user regarding a specific tab, window, display area, or object of operation, The browser operation linkage function transmits or receives shared permission information, including information identifying a specific tab, window, display area, or operation target, acquisition target identification information, display target identification information, and acquisition target identification information at the time of permission, to the Gateway module via a communication channel or a call within the same execution environment. The Gateway module receives operation requests or annotation requests from one or more AI agents in a call format that is independent of a specific AI service, AI provider, AI model, agent execution environment, client application, or tool protocol. The Gateway module determines whether the target of the operation request or the annotation request is the shared permitted target, and only if the target is the shared permitted target, forwards the operation request or the annotation request to the browser operation linkage function. The browser operation linkage function performs a read or operation on the target in response to the operation request, or draws an annotation layer on the target in response to the annotation request, The browser operation linkage function revokes the sharing permission for the target when it detects that the current target identification information of the target does not match the target identification information obtained at the time of permission, that the target has been closed or has become unavailable for any other reason, or that the user has performed an explicit cancellation operation. A method characterized by including the following.

10. A program characterized by causing one or more computers to execute the method described in claim 9.

Images