Intelligent agent remote cooperative control method and system

Abstract

The application provides an intelligent agent remote cooperative control method and system. The method comprises the following steps: a server acquires object information of a to-be-controlled object in a target system, and sends the object information to a client; the client determines a target object according to the object information, and returns a confirmation instruction; the server registers a callback for the target object according to the confirmation instruction, records an operation event performed on the target object, and generates an event record; the client generates a playback decision according to the event record; the server analyzes the event record to obtain playback content according to the playback decision, and sends an operation instruction to the target object according to the playback content to perform the operation event on the target object. The application realizes accurate control and background operation of the target object, solves the problem that a traditional remote control scheme cannot obtain window characteristics and cannot realize background batch operation due to the dependence on screen image recognition, and improves the accuracy and execution efficiency of remote cooperative control.

Description

Technical Field

[0001] This invention relates to the field of remote assistance technology for intelligent agents, and particularly to a method and system for remote collaborative control of intelligent agents. Background Technology

[0002] Current mainstream remote collaborative control solutions, such as Microsoft RDP (Remote Display Protocol) and Sunflower, work by encoding the screen of a remote machine into a video or image stream and transmitting it to the master control unit. The master control unit then packages keyboard and mouse operations into network commands and sends them to the controlled end for execution. These solutions all use screen image transmission combined with global input simulation to achieve remote control. However, this approach has significant shortcomings in practical applications: First, the master control unit needs to use OCR (Optical Character Recognition) technology to recognize the screen image and manually calculate relative coordinates to locate the target window, making it impossible to directly obtain the window's structural information and characteristics, thus hindering background operations. Second, image recognition-based control relies on the real-time view of the window; when the window is obscured or minimized, operations cannot be performed correctly. Finally, these solutions cannot send keyboard or mouse events independently to specific windows, nor can they register window-level event callbacks, making it difficult to achieve precise control of multiple windows and efficient batch operations, thus limiting the overall efficiency and stability of collaborative control. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide a method and system for remote collaborative control of intelligent agents, which can directly obtain the structural information of the target object and realize precise control and background operation of the target object.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for remote collaborative control of intelligent agents, characterized in that it includes: The server obtains the object information of the object to be controlled in the target system and sends the object information to the client; The client determines the target object based on the object information and returns a confirmation command; The server registers a callback for the target object according to the confirmation instruction, records the operation events performed on the target object, and generates an event log. The client generates a replay decision based on the event log; The server parses the event record to obtain the replay content based on the replay decision, and sends an operation instruction to the target object based on the replay content to execute the operation event on the target object.

[0005] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows: A remote collaborative control system for intelligent agents includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the various steps of the aforementioned remote collaborative control method for intelligent agents.

[0006] The beneficial effect of this invention is that it obtains object information of the object to be controlled in the target system through the server and sends it to the client, which is different from traditional solutions that rely on OCR (Optical Character Recognition). This solution uses image recognition and relative coordinate calculation based on optical character recognition (OCR) to directly obtain the structural information of objects, recognizing target objects without relying on the screen display. This avoids the problem of being unable to operate when the window is obscured or minimized. The client determines the target object based on the object information and returns a confirmation command. The server registers a callback for the target object based on the confirmation command and records the operation events performed on the target object to generate event logs. This establishes a callback-based event capture and recording mechanism, effectively solving the technical difficulties of not being able to send events independently to specific windows and not being able to achieve background operations. The client generates playback decisions based on the event logs, and the server parses the event logs based on the playback decisions and sends corresponding operation commands to the target object, executing operation events on the target object. This achieves intelligent decision-making and accurate playback of operation events. The server can directly send operation commands to the target object without relying on the screen display, improving the efficiency and stability of batch operations on multiple windows. This allows the server to complete the entire background operation process from target recognition to accurate playback of event recording without relying on the screen display, improving the efficiency and stability of batch operations on multiple windows in remote collaborative control. Attached Figure Description

[0007] Figure 1 This is a flowchart of a remote collaborative control method for intelligent agents according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the callback generation process of a remote collaborative control method for intelligent agents according to an embodiment of the present invention. Figure 3 This is a window control flowchart of a remote collaborative control method for intelligent agents according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the image module adaptation and record file generation process of an intelligent agent remote collaborative control method according to an embodiment of the present invention. Figure 5 This is a schematic diagram of an intelligent agent remote collaborative control system according to an embodiment of the present invention. Detailed Implementation

[0008] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0009] In existing technologies, mainstream remote collaborative control solutions (such as RDP and Sunflower) all employ a combination of screen image transmission and global input simulation. These solutions require OCR recognition of screen images and calculation of relative coordinates to locate windows, making it impossible to directly obtain window structure information. This results in the inability to send events independently to specific windows or to perform background operations. Furthermore, operations cannot be executed properly when windows are obscured or minimized, hindering precise control of multiple windows and efficient batch operations.

[0010] To at least address the aforementioned issues, the server obtains the object information of the target object and sends it to the client. The client identifies the target object and returns a confirmation command. The server registers callbacks for the target object, records operation events to generate event logs, and the client generates a playback decision. The server parses the event logs to obtain the playback content, sends operation commands to the target object, and executes them. This approach allows for direct acquisition of the target object's structural information without relying on screen image recognition and relative coordinate calculations, achieving precise target object positioning. It also enables the independent sending of events to specific windows through the callback registration mechanism, allowing for background operations that can continue even if the window is obscured or minimized. Furthermore, the recording and playback mechanism enables intelligent decision-making and automatic execution of operation events, improving the efficiency and stability of batch operations across multiple windows. This effectively solves the technical challenges of traditional remote control solutions, such as the inability to acquire window characteristics, send events independently, and perform background operations.

[0011] The following describes in detail a remote collaborative control method for intelligent agents according to the present invention. Please refer to [link / reference]. Figure 1 The method 100 includes steps 110 to 150.

[0012] Step 110: The server obtains the object information of the object to be controlled in the target system and sends the object information to the client.

[0013] For example, the server iterates through the objects to be controlled in the target system, detects the window size and window type of each object, and sends the obtained window information as object information to the client over the network.

[0014] Step 120: The client determines the target object based on the object information and returns a confirmation command.

[0015] For example, after receiving object information sent by the server, the client selects the target object to be operated from multiple windows according to the preset control strategy and returns a confirmation instruction to the server.

[0016] Step 130: The server registers a callback for the target object according to the confirmation instruction, records the operation events performed on the target object, and generates an event log.

[0017] For example, after receiving a confirmation command, the server registers an event callback for the target window, captures user operation events on the target window through the callback, and generates event records associated with the target object.

[0018] Step 140: The client generates a replay decision based on the event log.

[0019] For example, the client receives event logs sent by the server, parses the sequence of operation events within them, and generates corresponding replay decisions.

[0020] Step 150: The server obtains the replay content based on the replay decision parsing event record, and sends operation instructions to the target object based on the replay content to execute the operation event on the target object.

[0021] For example, the server receives the playback decision sent by the client, parses the event records to obtain the playback content, and sends operation instructions to the target object based on the obtained playback content. The target window then executes the operation instructions through a callback.

[0022] As described above, by directly obtaining object information of the target object through the server, precise positioning of the target object is achieved without relying on screen image recognition and relative coordinate calculation. By registering a callback mechanism for the target object, events can be sent independently to specific target objects, enabling background operations that can be executed even if the window is obscured or minimized. Through a collaborative mechanism where the client generates playback decisions and the server parses event records and sends operation commands, intelligent decision-making and automatic execution of operation events are achieved. This effectively solves the technical challenges of traditional remote control solutions, such as the inability to obtain window characteristics, the inability to send events independently to specific windows, and the inability to achieve background operations, thus improving the accuracy, stability, and batch operation efficiency of remote collaborative control.

[0023] In one alternative implementation, step 110 includes steps 111 to 113 performed by the server.

[0024] Step 111: Traverse the currently active windows in the target system.

[0025] For example, the server calls the window enumeration interface provided by the operating system (such as the EnumWindows function in Windows) to obtain a list of all active and visible window handles on the current desktop, and then iterates through each currently active window in the order of its creation.

[0026] Step 112: Detect the window information of each active window.

[0027] For example, for each window that is traversed, the server obtains information such as the window's size, position, title, and class name by calling system interfaces, and determines the coordinate range of each window on the screen and the window's type characteristics.

[0028] Step 113: Send the window information as object information of the object to be controlled to the client.

[0029] For example, the server packages all detected window information into data packets and sends them to the client via network protocols. The client then determines which windows can be controlled based on the received window information.

[0030] As described above, by traversing the currently active windows in the target system and detecting their window information, it is possible to comprehensively obtain complete information on all operable windows in the target system and report it to the client. This provides a data foundation for the client to accurately locate and select the target object to be controlled, avoiding the shortcomings of traditional solutions that cannot achieve precise control due to the lack of window information.

[0031] In one alternative implementation, step 130 includes steps 131 to 133, which are performed by the server.

[0032] Step 131: Register a keyboard callback for the target object and capture keyboard operation events performed on the target object locally on the server side through the keyboard callback.

[0033] For example, the server calls the hook registration function provided by the operating system (such as SetWindowsHookEx) to register a keyboard hook callback for the target window. When the user presses or releases a keyboard key on the target window, the operating system triggers the callback and passes information such as the virtual key code and scan code of the key to the server.

[0034] Step 132: Register a mouse callback for the target object and capture mouse operation events performed on the target object locally on the server side through the mouse callback.

[0035] For example, the server registers a mouse hook callback for the target window. When the user moves the mouse, presses the mouse button, releases the mouse button, or double-clicks the mouse on the target window, the operating system triggers the callback, passing the mouse coordinates, button type, scroll wheel information, etc., to the server.

[0036] Step 133: The captured keyboard operation events and mouse operation events are used as operation events to generate the event record.

[0037] For example, the server will arrange the captured keyboard operations (such as key press and key release) and mouse operations (such as movement, click, and double-click) in chronological order, record the operation type, operation coordinates, operation key value and other parameters of each operation, package them to generate an event record associated with the target window, and store it in the server's local storage medium.

[0038] As described above, by registering keyboard and mouse callbacks for the target object locally on the server, keyboard and mouse operation events are captured directly at the location where the operation occurs. This avoids the indirect approach of relying on screen image recognition and coordinate calculation, as is the case in traditional solutions, ensuring the real-time performance and accuracy of event capture. At the same time, the captured operation events are generated into event records in chronological order, providing complete data support for subsequent playback decisions and automatic execution.

[0039] In one alternative implementation, step 133 includes steps 1331 to 1333 performed by the server.

[0040] Step 1331: Capture keyboard and mouse operation events performed on the target object.

[0041] For example, the server monitors all input operations on the target window in real time through registered keyboard and mouse callbacks. When the user presses a keyboard key on the target window, the keyboard callback is triggered and captures the key event; when the user moves the mouse or clicks a mouse button on the target window, the mouse callback is triggered and captures the mouse event.

[0042] Step 1332: Record the operation type and operation coordinates of keyboard operation events and mouse operation events.

[0043] For example, the server parses each captured operation event: for keyboard operation events, it records the operation type as "key pressed" or "key released" and records the corresponding virtual key code; for mouse operation events, it records the operation type as "mouse move", "left mouse button pressed", "left mouse button released", "right mouse button pressed", "right mouse button released" or "double mouse", and records the coordinate position of the mouse operation on the target window when it occurs.

[0044] Step 1333: Record the operation type and operation coordinates as event records associated with the target object.

[0045] For example, the server writes the recorded operation types and operation coordinates into the event log table in the order of the operation occurrence time. Each event log entry contains at least three fields: timestamp, operation type, and operation coordinates. The event log table is then associated with the unique identifier of the target object and stored.

[0046] As described above, by capturing and recording the operation type and coordinates of the operation event, the abstract input operation is transformed into a structured data record, providing accurate data basis for subsequent playback decisions. The recorded operation type determines the type of instruction to be executed during playback, and the recorded operation coordinates determine the location where the instruction needs to be applied during playback. Together, they constitute complete event description information, ensuring that the original operation can be accurately reproduced during playback.

[0047] In one alternative implementation, step 150 includes steps 151 to 153 performed by the server.

[0048] Step 151: The server extracts the operation type and operation coordinates that need to be replayed from the event log.

[0049] For example, after the server receives the playback decision sent by the client, it reads the event records associated with the target object, traverses each event entry in the order of the timestamps in the event records, and parses the operation type field (such as "key pressed" or "left mouse click") and operation coordinate field (such as X coordinate and Y coordinate) from each entry.

[0050] Step 152: Generate the corresponding operation instruction according to the operation type. The operation instruction carries the operation coordinates.

[0051] For example, the server maps instructions based on the extracted operation type: if the operation type is a keyboard operation, it generates operation instructions that simulate keyboard key presses and encapsulates the virtual key codes into the instructions; if the operation type is a mouse operation, it generates operation instructions that simulate mouse actions and encapsulates the operation coordinates into the instructions, while determining the action parameters of the instructions (such as pressing, releasing, moving, double-clicking, etc.) based on the specific value of the operation type.

[0052] Step 153: Send the operation instruction to the target object of the registered callback for execution.

[0053] For example, the server sends the generated operation command to the target window through the message sending mechanism provided by the operating system. After the registered keyboard or mouse callback on the target window receives the command, it simulates the corresponding operation on the server and completes the playback of the original operation event.

[0054] As described above, by extracting the operation type and coordinates from the event log, the stored event data is restored into executable operation instructions. These instructions are then sent to the target object for execution using registered callbacks, forming a complete closed loop from "recording" to "playback." This method does not rely on screen images; even if the target window is in the background or obscured, the operation instructions can still be delivered and executed normally through the callback mechanism, truly achieving precise background operation in remote collaborative control.

[0055] In an alternative implementation, steps 160 and 161 are further performed by the server.

[0056] Step 160: Create a file callback to monitor file operation events.

[0057] For example, the server calls the file system filter driver interface or file monitoring function (such as ReadDirectoryChangesW) provided by the operating system to create a file callback for the target file or directory. When the target file is opened, created, closed, read, written, or deleted, the operating system triggers the callback, passing information such as the type of file operation, file path, and operation process to the server.

[0058] Step 161: Generate window event logs based on the monitored file operation events.

[0059] For example, based on the monitored file operation events, the server records the execution time, operation type (open, create, close, read, write, delete) and file path involved in each file operation, generates window event records associated with the target file, and stores them in the server's local storage medium.

[0060] As described above, by creating file callbacks to monitor file operation events and generating window event records, the scope of remote collaborative control is expanded from window interface operations to file system operations. The server can track the complete operation trajectory of files in the target system in real time, providing a data foundation for subsequent file operation auditing, backtracking analysis, and automated file management, further enriching the business scenarios of intelligent agent remote collaborative control.

[0061] In an alternative implementation, steps 170 and 171 are further performed by the server.

[0062] Step 170: Record the screen during the operation event on the target object to generate a video recording file.

[0063] For example, while registering callbacks for the target object and starting to record operation events, the server simultaneously launches a screen recording module to capture the screen image of the area where the target object is located, recording the complete visual content during the execution of the operation event. When the operation event recording is completed or playback is executed, the screen recording module stops synchronously, and the recorded screen sequence is encoded into a video file format (such as MP4 or AVI) and stored locally on the server.

[0064] Step 171: Associate and store the video recording file with the event log.

[0065] For example, after generating a video recording file, the server extracts the time range of the video recording, establishes a mapping relationship between the video recording file and event logs, and stores this mapping relationship in an associated index table. When it is necessary to audit or verify operation events later, the corresponding video segment can be quickly located through the event log, or the corresponding operation record can be found through the video segment.

[0066] As described above, by recording screen footage and associating the video recordings with event logs, a dual recording mechanism for operational data and visual images is established. The video recordings can serve as an auxiliary verification method for event logs, used to monitor whether the agent's actions conform to execution specifications. When anomalies or disputes arise in the event logs, they can be traced and reviewed through the associated video footage, improving the reliability and auditability of remote collaborative control.

[0067] In one alternative implementation, step 140 includes steps 141 to 143 performed by the client.

[0068] Step 141: The client determines whether the operation events in the event log conform to the preset execution specifications.

[0069] For example, if the operation sequence of an operation event is consistent with the preset specification, the operation parameters are within the preset range, and the operation object meets the requirements, it is judged to conform to the preset execution specification; if there are missing items, incorrect order, or abnormal parameters in the operation event, it is judged to not conform to the preset execution specification.

[0070] Step 142: If the operation event in the event log conforms to the preset execution specifications, the client generates a replay decision that allows replay.

[0071] For example, after the client generates a replay decision that allows replay, it sends the replay decision to the server to allow the replay of the operation events in the event log.

[0072] Step 143: If the operation event in the event log does not conform to the preset execution specifications, the client generates a replay decision to refuse replay.

[0073] For example, after the client generates a replay rejection decision, it sends this decision to the server to prevent replay of the operation events in the event log. At this point, the server will not execute the originally recorded operation events but will instead trigger a correction mechanism to automatically re-execute the sequence of operation events that conforms to preset execution specifications, ensuring that the operation process is completed according to the correct specifications. In this way, even if the originally recorded operation events are abnormal, the system can automatically correct and execute the correct operations without manual intervention.

[0074] As described above, the client generates a replay decision to allow or deny replay by judging whether the operation events in the event log conform to the preset execution specifications. This ensures that only operation events that conform to the specifications can be replayed, thereby improving the security and reliability of operation replay.

[0075] In an alternative implementation, steps 180 and 181 are further performed collaboratively by the client and the server.

[0076] Step 180: After receiving the event record, the client adjusts the event record according to the preset task objective and generates an updated event record.

[0077] For example, if the preset task objective is to optimize the operation sequence, the client adjusts the execution order of each operation event in the event log so that the adjusted operation sequence conforms to the preset task objective; if the preset task objective is to streamline redundant operations, the client deletes operation events in the event log that do not conform to the preset task objective and generates an updated event log.

[0078] Step 181: The server executes the operation event based on the updated event log.

[0079] For example, the server receives the updated event logs sent by the client and executes each operation event sequentially according to the operation order and operation parameters in the updated event logs.

[0080] As described above, the client adjusts the event logs according to the preset task objectives to generate updated event logs, and the server executes operation events based on the updated event logs, thereby optimizing the event logs and adapting them to the task objectives.

[0081] Please refer to Figure 2 The present invention can apply the above-mentioned solution to the scenario of remote collaborative control of intelligent agents, especially the business scenario that requires precise control of multiple windows, event recording and intelligent playback. Taking the client (intelligent agent decision end) and the server (controlled end) to perform business operations collaboratively as an example, the entire interaction process involves the collaborative operation of the client, the server and the callback generation module inside the server, specifically including 201 to 207.

[0082] 201. Server-side device callback generation. The server calls the device monitoring interface provided by the operating system to generate device callbacks, which are used to monitor the access and removal events of hardware devices in the target system, such as USB flash drive insertion, new keyboard or mouse connection, PCI device changes, etc. The events captured by the device callbacks can be used to trigger subsequent control policy adjustments. This is equivalent to step 110 above.

[0083] 202. Server-side keyboard callback generation. The server registers keyboard callbacks for the target window and captures keyboard operation events performed on the target window locally on the server through the keyboard callbacks, including key press events and key release events. This is equivalent to capturing keyboard operation events in steps 131 and 1331 above.

[0084] 203. Server-side mouse callback generation. The server registers mouse callbacks for the target window and captures mouse operation events performed on the target window locally on the server through the mouse callbacks, including mouse movement events, selection events, press events, release events, and double-click events. This is equivalent to capturing mouse operation events in steps 132 and 1331 above.

[0085] 204. Server-side system callback generation. The server registers system callbacks to monitor system events at the target operating system level, such as file deletion, program movement, system logout, and other system behaviors. Events captured by system callbacks can be used to extend the management scope of remote collaborative control. This is equivalent to an extension of the object information acquisition in step 110 above.

[0086] 205. Server-side file callback generation. The server creates a file callback to monitor operation events on the target file, including file open, create, close, read, write, and delete events. This is equivalent to steps 160 and 161 above.

[0087] 206. Server-side event callback generation. The server registers custom event callbacks to monitor specific business events other than the device callbacks, keyboard callbacks, mouse callbacks, system callbacks, and file callbacks mentioned above. Event callbacks give the system good scalability, enabling it to flexibly capture and process custom events according to actual business needs. This is equivalent to step 130 above.

[0088] 207. Server-side event logging. The server organizes the operation events captured through keyboard callbacks, mouse callbacks, file callbacks, etc., in chronological order of occurrence, recording information such as the operation type, operation coordinates, and operation parameters for each operation event, generating an event log associated with the target object, and storing it in the server's local storage medium. This is equivalent to the event log generation process in steps 133 and 1331 to 1333 above.

[0089] Through the above steps, this invention constructs a complete callback generation system by coordinating device callbacks, keyboard callbacks, mouse callbacks, system callbacks, file callbacks, and event callbacks. Keyboard and mouse callbacks provide underlying event capture capabilities for precise control of window objects; file callbacks support file system operation monitoring; system and device callbacks expand the system's monitoring scope; and event callbacks ensure the system's scalability. All events captured by callbacks ultimately converge in the event log, providing a complete data foundation for subsequent intelligent playback decisions. This enables precise control and background operation of target objects, effectively solving the technical challenges of traditional remote control solutions that cannot acquire window characteristics, cannot independently send events to specific windows, and cannot achieve background operation.

[0090] Please refer to Figure 3 The window control process in this invention includes steps S301 to S304.

[0091] S301. Obtain Window Information. The server iterates through the currently active graphics modules (such as GDI or D3D) in the target system, detecting the window information of each active window, including window size, window type, window position, and window title. The server sends the obtained window information to the client. The client selects the target window to be controlled from multiple windows based on the window information and returns a confirmation command to the server. This is equivalent to the process of obtaining window information and determining the target object in steps 110 and 111 to 113 above.

[0092] S302. Register Window Callbacks. Based on the confirmation command returned by the client, the server registers keyboard and mouse callbacks for the target window. Keyboard callbacks capture keyboard operation events performed on the target window, including key presses and releases; mouse callbacks capture mouse operation events performed on the target window, including mouse movement, selection, pressing, releasing, and double-clicking. After registration, the server can monitor all input operations on the target window in real time through the callbacks. This is equivalent to the process of registering callbacks and capturing operation events in steps 130 and 131 to 133 above.

[0093] S303. Reset Window Characteristics. The server resets the characteristics of the target window during or after the registered window callback. Resetting window characteristics includes: setting the target window to the foreground (always on top), ensuring the target window remains in the foreground during operation execution and is not obscured by other windows; adjusting the text display attributes of the target window (such as font type and font size); setting the window to be movable or disabled, locking the window position or allowing users to move the window according to business needs. By resetting window characteristics, a stable and predictable window environment is provided for subsequent operation event recording and playback. This is equivalent to step 133 above.

[0094] S304. Register Window Events. After completing window callback registration and window property reset, the server registers window events. Window events refer to the specific events captured by the callbacks registered for the target window, including keyboard events (such as key press and key release) and mouse events (such as mouse click and mouse move). Based on the registered window events, the server captures and records the operation events occurring on the target window, generating event records associated with the target window. When specific conditions are met (such as window minimization or moving to a certain coordinate), the server can activate the corresponding event and perform the corresponding operation. This is equivalent to the process of capturing operation events and generating event records in steps 133 and 1331 to 1333 above.

[0095] Through the above steps, this invention first achieves precise positioning of the target window by acquiring window information; secondly, it establishes an underlying event capture channel by registering window callbacks; thirdly, it provides a stable window environment for operation execution by resetting window characteristics; and finally, it completes the capture and recording of operation events by registering window events. This process provides complete data support for subsequent playback decisions and automatic execution, realizing precise control of window objects and background operations. It effectively solves the technical problems of traditional remote control solutions that cannot acquire window characteristics, cannot independently send events to specific windows, and cannot achieve background operations.

[0096] Please refer to Figure 4 The graphic module adaptation and record file generation process in this invention includes steps S401 to S407.

[0097] S401. The server initiates the remote collaborative control process, preparing to acquire window information and adapt the graphics module to the target system. The server calls the window enumeration interface provided by the operating system to traverse all currently active windows in the target system and obtain the window information of each window, including basic information such as window size, window type, window position, and window title. This is equivalent to the process of traversing windows and detecting window information in steps 111 to 112 above.

[0098] S402. The server detects the types of graphics modules currently available in the target system. Graphics modules include GDI (Graphics Device Interface) and D3D (Direct3D), etc. The server determines the default graphics module type used by the current system and whether a hardware-accelerated D3D module exists by calling the system graphics interface or reading the system configuration. If a D3D module is detected as available, proceed to step S405; if the D3D module is not available, proceed to step S404. This is equivalent to the detection of graphics modules when obtaining object information in step 110 above.

[0099] S403. Detect Available Graphics Modules: The server detects the types of graphics modules currently available in the target system. Graphics modules include GDI (Graphics Device Interface) and D3D (Direct3D), etc. The server determines the default graphics module type used by the current system and whether a hardware-accelerated D3D module exists by calling the system graphics interface or reading the system configuration. If a D3D module is detected as available, proceed to S404 to create the D3D module; if the D3D module is unavailable, proceed to S405 to automatically adapt the GDI module. This is equivalent to the graphics module detection in step 110 above.

[0100] S404. When step S403 detects that the D3D module is available, the server creates the corresponding D3D module. The D3D module utilizes hardware acceleration capabilities to provide smoother graphics processing and higher frame rates, making it suitable for scenarios with high image quality requirements or frequent window refreshes.

[0101] S405. When S403 detects that the D3D module is unavailable, the server automatically downgrades and adapts, selecting the GDI module as the currently available graphics module. GDI is the traditional graphics device interface for Windows systems, offering good compatibility and ensuring that window image capture and processing can still be completed normally even in environments lacking hardware acceleration support. The server creates the corresponding GDI module. GDI modules are based on software rendering, have strong compatibility, and are suitable for scenarios with low graphics performance requirements or where D3D is unavailable. This is equivalent to the graphics module adaptation strategy in step 110 above.

[0102] S406. The server generates a log file based on the obtained window information and the selected graphics module type. The log file contains structured data about the window (such as window size, window type, window coordinates, etc.) and configuration information of the graphics module (such as module type, rendering method, etc.). The generated log file is associated with the server's event log and is used to restore the window state and graphics environment during subsequent playback. This is equivalent to an extension of the event log generation in step 133 above. After the log file is generated, the server ends the current graphics module adaptation and log file generation process and enters the subsequent operation event recording or playback stage.

[0103] Through the above steps, this invention achieves compatibility support for different graphics environments by detecting available graphics modules in the target system and automatically adapting to either D3D or GDI modules based on the detection results. Prioritizing the D3D module fully utilizes hardware acceleration capabilities to improve image processing smoothness; when D3D is unavailable, it automatically downgrades to the GDI module, ensuring the solution runs normally in various system environments. Simultaneously, by generating a log file to persistently store window information and graphics module configurations, a complete contextual environment is provided for subsequent operation event playback, effectively improving the compatibility and stability of remote collaborative control.

[0104] Please refer to Figure 5 The present invention also provides an intelligent agent remote collaborative control system 500, including a memory 501, a processor 502, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the various steps of the intelligent agent remote collaborative control method described above.

[0105] In summary, this invention provides a method and system for remote collaborative control of intelligent agents. First, the server obtains object information of the object to be controlled in the target system and sends it to the client. The client determines the target object based on the object information and returns a confirmation command, achieving precise positioning of the target object, unlike traditional solutions that rely on indirect methods such as OCR image recognition and relative coordinate calculation. Furthermore, the server registers keyboard and mouse callbacks for the target object, capturing operation events locally on the server through the callback mechanism. This allows for real-time acquisition of operation events without relying on screen display, solving the problem in traditional solutions where operations cannot be executed due to window obstruction or minimization.

[0106] The server records captured operation events according to operation type and operation coordinates, generating event records associated with the target object. When replay is required, the client determines whether the operation event conforms to the preset execution specifications. If it does, a replay decision is generated to allow replay. The server parses the event records based on the replay decision to obtain the replay content, extracts the operation type and operation coordinates, generates the corresponding operation instructions, and sends them to the registered callback target object for execution.

[0107] This invention effectively solves the technical problems of traditional remote control solutions, such as the inability to obtain window characteristics, the inability to send events to specific windows independently, and the inability to perform background operations, and significantly improves the accuracy, stability, and batch operation efficiency of remote collaborative control.

[0108] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for remote collaborative control of intelligent agents, characterized in that, include: The server obtains the object information of the object to be controlled in the target system and sends the object information to the client; The client determines the target object based on the object information and returns a confirmation command; The server registers a callback for the target object according to the confirmation instruction, records the operation events performed on the target object, and generates an event log. The client generates a replay decision based on the event log; The server parses the event record to obtain the replay content based on the replay decision, and sends an operation instruction to the target object based on the replay content to execute the operation event on the target object.

2. The method for remote collaborative control of an intelligent agent according to claim 1, characterized in that, The server obtains object information of the object to be controlled in the target system and sends the object information to the client, including: Iterate through the currently active windows in the target system; Detect the window information of each active window; The window information is sent to the client as the object information of the object to be controlled.

3. The method for remote collaborative control of intelligent agents according to claim 1, characterized in that, The server registers a callback for the target object based on the confirmation instruction, records the operation events performed on the target object, and generates an event log, including: Register a keyboard callback for the target object, and capture keyboard operation events performed on the target object locally on the server through the keyboard callback; Register a mouse callback for the target object, and capture mouse operation events performed on the target object locally on the server through the mouse callback; The captured keyboard and mouse operation events are used as the operation events to generate the event record.

4. The method for remote collaborative control of an intelligent agent according to claim 3, characterized in that, The step of generating the event record by using the captured keyboard operation events and mouse operation events as the operation events includes: Capture the keyboard operation events and mouse operation events performed on the target object; Record the operation type and operation coordinates of the keyboard operation events and the mouse operation events; The recorded operation type and operation coordinates are recorded as event records associated with the target object.

5. The method for remote collaborative control of an intelligent agent according to claim 1, characterized in that, The server parses the event record based on the playback decision to obtain the playback content, and sends an operation instruction to the target object based on the playback content to execute the operation event on the target object, including: The server extracts the operation type and operation coordinates that need to be replayed from the event records; The operation instruction is generated according to the operation type, and the operation instruction carries the operation coordinates. The operation instruction is sent to the target object that has been registered for callback execution.

6. The method for remote collaborative control of intelligent agents according to claim 1, characterized in that, Also includes: Create a file callback and monitor file operation events through the file callback; Window event logs are generated based on the monitored file operation events.

7. The method for remote collaborative control of intelligent agents according to claim 1, characterized in that, Also includes: Record the screen footage during the execution of the operation event on the target object to generate a video recording file; The video recording file is associated with and stored with the event log.

8. The method for remote collaborative control of an intelligent agent according to claim 1, characterized in that, The client generates a replay decision based on the event log, including: The client determines whether the operation event in the event log conforms to the preset execution specifications; If the operation event in the event log conforms to the preset execution specification, the client generates the replay decision that allows replay. If the operation event in the event log does not conform to the preset execution specification, the client generates a replay decision to refuse replay.

9. The method for remote collaborative control of an intelligent agent according to claim 1, characterized in that, Also includes: After receiving the event record, the client adjusts the event record according to the preset task objective and generates an updated event record. The server executes the operation event based on the updated event record.

10. A remote collaborative control system for intelligent agents, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements each step of the intelligent agent remote collaborative control method according to any one of claims 1 to 9.

Images