Method for carrying out universal non-intrusive automatic operation on equipment with man-machine interface by using multi-modal large model agent

Abstract

The invention provides a method for carrying out universal non-intrusive automatic operation on equipment with a man-machine interface by using a multi-mode large model agent. The method comprises the following steps: step 1, signal acquisition: acquiring an output signal of target equipment by edge equipment; step 2, data uploading: the edge computing device reads a video stream of the edge computing device or converts the original signal acquired in the step 1 into an analyzable digital signal, then divides the digital signal into independent screenshots and preprocesses the screenshots, and then uploads the processed screenshots and an identification result to a server; step 3, instruction generation and issuing: the server generates a subsequent operation instruction after analyzing the data, and returns the subsequent operation instruction to the edge computing device; 4, the edge computing device converts the operation instruction into a specific HID signal and sends the specific HID signal to the target device, and automatic operation closed loop is completed. The method has the beneficial effects that the target equipment can be automatically controlled in a mode which is completely the same as that of a human operator, and the method can be applied to any electronic equipment which can be operated by human beings.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for performing universal non-invasive automatic operation on a device having a human-machine interface using a multimodal large-model intelligent agent. Background Art

[0002] With the rapid development of information technology, most business processes have transitioned from paper-based operations to electronic operations based on personal computers, information platforms, and the internet. This type of business is characterized by complex and highly repetitive operations. Typical examples include: database entry operations after data classification and processing using Excel tools; large-scale file renaming and folder structure adjustment projects; and cyclical execution of software application functional testing. Because the work process involves some subjective judgment, there are significant technical barriers to achieving full automation, and it currently relies primarily on manual operations. However, it is worth noting that although this type of business has relatively limited professional and technical requirements, the scale of human resources it consumes is extremely large, which constitutes a significant efficiency paradox in the business operation dimension.

[0003] It's particularly important to note that the architectural complexity of the customized systems and dedicated platforms used by large and medium-sized enterprises often requires new employees to undergo one to two months of pre-job training before they can master basic tasks. This significant training cost is also incurred during job handovers. This positive correlation between system complexity and human resource investment makes the development of automated technology solutions that can replace complex business processes a critical and pressing issue in the current digital transformation process.

[0004] To address the aforementioned business transformation challenges, academic research has launched numerous technical research projects. A representative solution is Robotic Process Automation (RPA), which integrates digital simulation of human operations with system interactions through the deployment of software robots (bots). However, existing RPA systems have significant flaws. First, some of their script-based automation relies on precise pixel coordinates provided by image recognition to specify UI icons, making it difficult for the system to dynamically adapt to changes in interface layouts and limiting the scope of task applicability. Second, differences in target host software and hardware configurations, as well as changes in task parameters, directly impact the performance and versatility of automated execution based on API interfaces and automated software installation. More importantly, this traditional automation paradigm not only significantly raises the technical learning threshold for operators, but also, due to the need for open system permissions and the installation of additional software, creates significant cybersecurity risks, creating secondary governance challenges during enterprise digital transformation.

[0005] Traditional automation techniques are primarily implemented through the following methods. First, there are technologies that directly identify interface controls and elements by parsing the application's DOM (Document Object Model) structure. In web applications in particular, accessing the page's JavaScript code to locate the coordinates of on-screen interactive controls (such as drop-down menus and buttons) is a common method.

[0006] In terms of actual operation execution, traditional automation techniques mainly use two methods: one is to directly use the control interface exposed by the target application to efficiently control the program; the other is to perform operations indirectly through libraries such as pyautogui that simulate mouse and keyboard control. However, due to the characteristics of information acquisition and operation methods, traditional automation programs must run directly on the target device, leading to the following inevitable problems:

[0007] (1) Environmental adaptability requirements: Adaptation work needs to be carried out for different devices, applications, and systems;

[0008] (2) Hardware performance burden: high resource consumption, which may cause high load on the target device;

[0009] (3) Information security risks brought about by open permissions: the device needs to be granted extensive operating permissions;

[0010] (4) Operational difficulties for special applications: Unable to handle information platforms developed based on internal standards, systems that do not provide APIs, or applications that use non-standard controls.

[0011] To achieve specific operational procedures for specific tasks, typical Robotic Process Automation (RPA) requires users to record individual operations for each task or edit automated tasks through the modular programming GUI provided by the RPA software. Essentially, RPA can be considered the "automatic generation of intelligent automation scripts." Compared to traditional simple scripts, the generated intelligent process scripts include exception handling capabilities to handle certain changes and unexpected situations (such as error pop-ups). Furthermore, visual modular programming tools lower the barrier to entry, making them accessible to non-technical personnel. However, the following issues remain:

[0012] (1) Learning cost: RPA software operation still requires learning costs, and some scenarios require professional knowledge, which has even given rise to professional enterprise business automation service providers;

[0013] (2) Dealing with frequent UI updates: Frequent changes in interface elements will increase the script failure rate, requiring re-recording of operations or reconstruction of processes.

[0014] These issues reveal the fundamental challenges facing traditional RPA and automation methods, and also generate motivation to explore new solutions. Summary of the Invention

[0015] To solve the problems in the prior art, the present invention provides a method for performing universal non-invasive automatic operation of a device having a human-machine interface using a multimodal large-model intelligent agent, comprising the following steps:

[0016] Step 1, signal capture: The edge computing device directly acquires / identifies the output signal of the target device;

[0017] Step 2, data upload: The edge computing device directly reads its video stream or converts the original signal collected in step 1 into a parseable digital signal, then divides it into independent screenshots and pre-processes them, and then uploads the processed screenshots and recognition results to the server;

[0018] Step 3: Instruction generation and distribution: The server analyzes the data and generates subsequent operation instructions, which are then sent back to the edge computing device.

[0019] Step 4: Instruction execution feedback: The edge computing device converts the operation instruction into a specific HID signal and sends it to the target device, completing the automated operation closed loop;

[0020] The server is deployed with a large multimodal language model responsible for screen understanding and operation reasoning. It can integrate and process input text and images and provide corresponding feedback.

[0021] As a further improvement of the present invention, in step 1, the edge computing device acquires / identifies the output signal of the target device in the following manner:

[0022] Method 1: The edge computing device directly obtains / identifies the output signal of the target device to the human-machine interface. For computers, it is the video output signal (such as HDMI signal); for industrial instruments, it is the electrical signal output to the indicator light.

[0023] Method 2: Use a camera to shoot the human-computer interaction interface of the target device and directly identify its output and feedback visually.

[0024] As a further improvement of the present invention, in step 2, operation positioning is achieved by mapping the pixel coordinates of the interface elements in the screenshot to the actual physical coordinates. Specifically, when the edge computing device transmits the screenshot to the server, it is necessary to synchronously send the scaling coefficient after resolution adaptation processing. After the server obtains the operation coordinates based on the received scaled image analysis, it uses the scaling coefficient to reversely calculate the precise position of the target device in the actual screen coordinate system, thereby avoiding positioning deviations caused by differences in resolution between different devices.

[0025] As a further improvement of the present invention, this method needs to pre-define a basic action instruction set including all human-computer interaction interface operations on the edge computing device side, and by atomizing the underlying operations, the server can conveniently call the combined instructions.

[0026] As a further improvement of the present invention, the human-computer interaction interface includes a computer-type HID and a physical control panel. For the computer-type HID, the defined basic actions include: mouse operation: left button press / release, right button press / release, scroll wheel, cursor movement; keyboard operation: single key press / release; for other physical control panels, it can drive a dexterous hand robotic arm.

[0027] As a further improvement of the present invention, during the digital processing of video signals (such as HDMI), this method synchronously performs OCR processing of screenshots on the edge computing device. The recognition results are uploaded to the server together with the screenshot data as auxiliary prompt information during the inference of the large language model.

[0028] This method only requires edge computing devices to process and capture static screenshots of the target device screen status after the last operation to obtain all the information needed for decision-making. These screenshots contain visual information about the GUI interface layout, image elements, and user interaction components.

[0029] This method uses the OCR function of a large language model to extract text information embedded in images, including file names, control text, operation instructions, and read and summarize long documents.

[0030] As a further improvement of the present invention, this method uses the SoM method to add visual prompts to the large language input image on the basis of text prompts to enhance spatial cognition ability. The specific implementation process is as follows:

[0031] Step S1: Applying a segmentation model; dividing the screen elements into regions using the segmentation model, identifying all interactive objects as independent objects, and marking the pixel areas they occupy using masks;

[0032] Step S2: Generate a matrix; construct a proportional matrix based on the image resolution, where each pixel in the matrix is ​​marked as 1 if it belongs to the corresponding object area, and the remaining pixels are marked as 0;

[0033] Step S3: Create object bounding boxes and digital labels; calculate the average x / y coordinates of all pixels marked as 1 in the matrix, draw the object bounding box based on the coordinates, and assign a corresponding digital label to each element in the screenshot based on the pixel coordinates of each bounding box;

[0034] In step S1, the segmentation model used is the general segmentation model SAM proposed by Meta AI, and the segmentation model SAM is deployed on the server.

[0035] As a further improvement of the present invention, the method comprises:

[0036] Reflective memory step: The reflective memory module uses the large language model to evaluate whether historical operations were executed normally and whether the task objectives were successfully completed. The reflective memory module can fully record past operation records, screenshots of the target device, and corresponding operation instructions, and input this information into the large language model as prompts. The operation instructions field helps the large language model to smoothly recall the recent status of the target device, the executed operations, and their purpose during self-reflection. When the operation fails to achieve the task objectives, it can analyze the reasons for failure and implement strategy adjustments or operation corrections based on this.

[0037] As a further improvement of the present invention, the method further comprises:

[0038] Information memory step: The information memory module stores information obtained during the automated operation process, simulating the human behavior of recording key information and subsequently reviewing it. When performing tasks that require intelligent information collection, the skill function is called to guide the large language model to label and store the information extracted from the target device screen, which can then be quickly retrieved through the labels.

[0039] As a further improvement of the present invention, the method further comprises:

[0040] Steps for editing automatic operation tasks: Describe the tasks to be automatically executed on the target device using natural language and upload them to the server.

[0041] The beneficial effects of the present invention are: 1. The general edge computing device control (EGCC) framework described in the present invention is intended to automatically control the target device in exactly the same way as a human operator (human-computer interaction interface), so that it can be applied to any electronic device that can be operated by humans; 2. The present invention uses a wired connection between the edge computing device and the target controlled device, does not make any software or hardware requirements and modifications to the target device, and completes the acquisition of information and the construction and execution of automatic operations in a completely external connection manner. Moreover, since the framework runs completely on the cloud server, computing modules with high hardware performance requirements such as large models will not generate performance load on the target device or reduce the efficiency of automatic operations due to insufficient hardware performance; 3. The EGCC framework of the present invention constructs an optional action space for the multimodal large model and provides output examples to standardize the output behavior of the large model, and uses its high natural language ability and multimodal processing ability combined with image processing to make automatic operation decisions, thereby enabling users to complete automatic operation task editing and highly intelligent automatic operation task robustness through non-standardized natural language. and executable scope; 4. The reflective memory module of the present invention utilizes specific data structures and algorithms to help LLM smoothly recall the latest status of the target device, executed operations and their purposes during self-reflection. When the operation fails to achieve the task goal, the system can analyze the cause of the failure and implement strategy adjustments or operational corrections. The information memory algorithm enables the EGCC framework to simulate the human behavior pattern of recording key information and subsequently reviewing it. The EGCC framework will call specially designed skill functions and has the ability to complete complex warning tasks; 5. The present invention significantly improves the image understanding and spatial cognition capabilities of LLM. Through the digital labels attached by the EGCC framework, LLM can express the output content more clearly and specifically; 6. The EGCC framework only receives the output of the target device to the human-computer interaction interface, and only simulates and inputs the input that can be generated by the human-computer interaction interface of the device. It communicates with the target device in a way that is completely the same as that of a human operator for the target device, thereby eliminating the need for the target device to open any ports, set application system APIs, and grant background permissions, thereby minimizing the impact of deployment costs and information security concerns. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of the overall structure of the EGCC framework of the method of the present invention;

[0043] Figure 2 This is the connection diagram between Raspberry Pi 4B and PIKVM-A3 according to the method of the present invention;

[0044] Figure 3 This is a schematic diagram of the overall structure of the EGCC framework of the method of the present invention;

[0045] Figure 4 is a schematic diagram illustrating an example of the image understanding and spatial perception capabilities of a large model when the method of the present invention directly inputs the original screenshot into the large model;

[0046] FIG5 is an example of an image used in verification by the method of the present invention. Figure 5a Original picture, Figure 5b Attach a picture of the visual prompt;

[0047] Figure 6 It is a flow chart of visual prompt generation of the method of the present invention;

[0048] Figure 7 The method of the present invention adds a visual prompt to the original picture and then inputs a large model inspection test picture. DETAILED DESCRIPTION

[0049] This invention is dedicated to building an innovative automated operation framework to break through the bottleneck of existing technologies. The core research objectives focus on the following four dimensions:

[0050] (1) Operation control is completely based on natural language, completely eliminating learning costs and ensuring that operators can achieve intuitive operation without professional technical training.

[0051] (2) It uses the same operating mode as a human operator, that is, it only uses the human-machine interaction interface (HMI) such as the screen, keyboard, and mouse to complete the information acquisition and operation execution of the target device, without requesting permissions for the target device, opening ports, and without any configuration modification or program installation.

[0052] (3) It does not impose any software or hardware requirements on the target device, and does not require adaptation to different deployment environments. It can effectively deal with the lack of non-standardized UI components and operation APIs. Therefore, it can be applied to automatic operation scenarios of electronic devices other than computers (such as intelligent distribution box control, automatic operation of large industrial machinery, intelligent monitoring and problem response of dashboard indicator lights, etc.).

[0053] (4) It has a robust fault-tolerant mechanism that can prevent operational failures in automated business processes even if the target application UI is frequently updated.

[0054] To achieve this goal, this paper innovatively proposes a theoretical model for "non-intrusive manipulation" based on edge computing devices and a multimodal large-scale model. This model adheres to the principles of General Computer Control (GCC) and, while ensuring the integrity and security of the target system, constructs a universal automated manipulation framework based solely on visual cognition and manipulation behavior simulation.

[0055] Building on this theoretical foundation, this invention essentially provides an automated AI agent solution for general-purpose devices with human-computer interaction interfaces, based on multimodal large-scale model technology. If the existing edge computing devices, such as keyboards and mice, are replaced with robotic arms, dexterous hands, cameras, and other devices, the AI ​​agent described in this invention can, based on the same principles, replace human operators of physical buttons and switches, with a wide range of applications.

[0056] The present invention targets business process automation tasks with different specific processes, system configurations, hardware conditions and even information security requirements, and is committed to realizing an automated operation framework with low application, deployment and maintenance costs for all human-operable electronic devices.

[0057] Against the backdrop of accelerating global digital transformation, business process automation has become a core strategic tool for enterprises to improve operational efficiency and reduce labor costs. This technological trend is driven by multiple drivers: the urgent need for efficiency optimization within enterprises, the mature application of cloud computing and artificial intelligence technologies, and the rigid demand for digital processes in remote work. The current technology ecosystem exhibits three generations of evolution: early single-application automation based on scripting tools such as VBA (the first generation); enterprise-level RPA systems that rely on UI element positioning (the second generation, including UiPath and Automation Anywhere); and Intelligent Process Automation (IPA), which integrates natural language processing (NLP), optical character recognition (OCR), and machine learning. The third generation significantly enhances the ability to handle complex tasks.

[0058] The generational evolution of business process automation technology has shown clear characteristics. The first generation of automation systems based on scripting tools simulated operations by precisely controlling the timing of input devices. Essentially, they translated manual operations into sequences of coordinate instructions (e.g., moving the mouse to a specified coordinate, executing a click, or entering a preset keystroke combination).

[0059] The second-generation enterprise-level RPA system achieved breakthroughs through technological abstraction. Its core innovations are reflected in three aspects: first, it uses a visual programming interface (such as a drag-and-drop flowchart designer) to encapsulate the underlying code into reusable modules, lowering the threshold for use and editing; second, it introduces a multimodal element positioning mechanism, combined with document object model (DOM) structure parsing, image feature matching, and OCR technology, so that the system can adapt to moderate changes in interface elements (such as button position offset); finally, it constructs conditional branching logic and presets exception handling paths (such as a pop-up capture retry mechanism).

[0060] The third-generation intelligent process automation (IPA) achieves a cognitive leap by integrating AI technology. Its technical architecture includes three core components: an environmental perception module (obtaining system status through API / logs), a decision engine (parsing unstructured instructions based on NLP models), and an execution unit (using a mixture of GUI operations and API calls).

[0061] To achieve truly intelligent automated operation, environmental awareness, logical decision-making, and operational execution are essential. Just like a human operator, they need to observe the target device's status on a computer screen, determine the multiple operations required to achieve their goals through deliberation, and then use a keyboard and mouse to execute these operations on the target device. Depending on whether environmental awareness, logical decision-making, and operational execution are implemented, and how they are implemented, different automated operation methods will result in different deployment and performance limitations.

[0062] General Computer Control (GCC) refers to the basic intelligence that can learn and perform any computer operation task that can be achieved by humans by talking to computers only through a standard human-computer interaction interface. Specifically, GCC only uses the image output of the target device to the human-computer interaction interface (such as the computer display screen) and the sound output in some cases as the input of the automatic operating system, and only uses the keyboard and mouse operations as the output of the automatic operating system to perform automatic operations of business processes, thereby providing a general automatic operation interface that can be applied to any computer automatic operation task. With its versatility, GCC is regarded as an important milestone in the realization of general artificial intelligence (AGI) that begins in the digital world, and it can further adapt to the physical environment by combining with robotics technology. To realize GCC, several key challenges need to be solved:

[0063] First, the observation space for GCC tasks is multimodal, including images (such as video frames and screenshots), text (such as command-line output and text embedded in images), and audio (such as narration, sound effects, and background music). For example, in a digital game scenario, the agent needs to parse screen images to understand the game environment while simultaneously following text and voice commands to complete interactive tasks. This requires the system to possess advanced multimodal processing capabilities to integrate and reason about diverse input sources.

[0064] Secondly, GCC requires precise interaction with the computer through standard input devices such as the keyboard and mouse. Mouse operations, due to their dynamic position and velocity, have continuous action space characteristics. While keyboard operations belong to a discrete action space, their operational dimensions are still very large due to additional attributes such as key combinations and timing. Furthermore, the syntactic constraints faced by the agent when generating code further increase operational complexity.

[0065] Furthermore, GCC tasks often have partial observability. For example, when performing a document editing task, the screen may only display a portion of the document's contents. This requires the agent to have long-term memory, capable of storing historical action results and experiential knowledge, and making fast and accurate decisions based on this information.

[0066] By overcoming these challenges, GCC will be able to fully realize its potential as a general-purpose framework and realize application possibilities in a wider range of fields.

[0067] To address the numerous challenges inherent in traditional automation, this invention proposes a non-invasive operation concept. Its core principle is to maintain operational authority within the reach of a human operator, completing the automation process solely through a human-computer interface. Specifically, the system acquires the required data based on information displayed on the screen, and controls the target device using input devices such as a mouse and keyboard.

[0068] The resulting non-intrusive manipulation framework appears to the target device as if it were a monitor, mouse, or other peripheral. The framework is fully external to the target system, enabling direct manipulation of any computer device without requiring additional adaptation. Based on this non-intrusive approach, this technique requires no software or driver installation and imposes no performance or system requirements beyond those imposed by the task.

[0069] This invention discloses a method for performing universal, non-invasive, automatic operation of devices with human-machine interfaces using a multimodal, large-scale intelligent agent, including: Step 1, signal capture: An edge computing device directly acquires / identifies the output signal of a target device; this includes not only computers, but also not limited to HDMI signals, and strictly speaking, includes all outputs of target devices through human-machine interfaces, such as dashboards and indicator lights of industrial equipment. The specific example provided in this invention uses computers and HDMI signals as the objects. In actual application, there are two ways to obtain the corresponding signals:

[0070] Method 1: Use edge computing devices to directly obtain / identify the output signal of the target device to the human-machine interface (for computers, it is an HDMI signal; for industrial instruments, it may be an electrical signal output to the indicator light).

[0071] Method 2: Use a camera to shoot the human interface of the target device that cannot be connected through method 1, and directly [see and recognize] the output and feedback of the target device in the same way as humans.

[0072] Step 2, data upload: The edge computing device directly reads its video stream or converts the raw signal collected in step 1 into a parseable digital signal, then splits it into independent screenshots and pre-processes them. The processed screenshots and recognition results are then uploaded to multiple servers.

[0073] Step 3: Instructions are generated and sent: The server analyzes the data and generates subsequent operation instructions, which are then transmitted back to the edge computing device. The signal is automatically uploaded without the need for instructions, and the specific pre-processing method (splitting into screenshots or directly reading the video stream) is not fixed and varies depending on the scenario.

[0074] Step 4: Instruction execution feedback: The edge computing device converts the operation instruction into a specific HID signal and sends it to the target device, completing the automated operation closed loop;

[0075] The server is deployed with a large multimodal language model responsible for screen understanding and operation reasoning. It can integrate and process input text and images and provide corresponding feedback.

[0076] This paper designs an automation framework, EGCC (Edge-General Computer Control), based on non-invasive operations and the concept of GCC, combined with edge computing technology. The EGCC framework provides a feasible solution for achieving the ultimate goal of GCC from the perspective of edge computing technology.

[0077] 1.EGCC Framework

[0078] In order to master all computer tasks through a universal human-computer interaction interface and build a basic intelligent agent for automated tasks suitable for human-operable devices, the present invention proposes the EGCC framework. EGCC is an automated operation framework with high intelligence and flexibility, based on the large language model (LLM) as the core of logical judgment and edge computing devices as the information processing and transmission medium, which can meet the requirements related to GCC tasks.

[0079] The EGCC framework must possess the following core capabilities: understanding the current state of a target device solely from the information displayed on the computer screen, extracting and reasoning about execution steps from task objectives described in natural language by the user, and integrating text and image information for comprehensive reasoning. To achieve these capabilities, a model with strong multimodal processing capabilities and excellent performance in both image and text understanding is essential.

[0080] Figure 1 The overall architecture of the EGCC framework is shown. The framework consists of three parts: an edge computing device connected to the target device via a wired connection, a server that undertakes core intelligent processing, and a large language model (LLM) responsible for screen understanding and operation reasoning. Users can describe automation tasks in the form of text files and upload them to the server. This framework specifically uses ChatGPT-4o as the LLM core because its excellent performance in image and text fusion understanding can fully meet the GCC task requirements. Similar large language models with multimodal interaction capabilities can also be used as the LLM core, including domestic large models with multimodal capabilities such as Wenxin 4.5.

[0081] 2. Editing of automatic operation tasks

[0082] Users can describe the tasks to be automatically executed on the target device in natural language and upload them to the server. Table 1 shows an example of a task instruction: if a currently open document file is to be summarized, in addition to clarifying the core goal of generating a summary, four specific steps must be specified: scrolling through the document content, closing the document file using a shortcut key, opening a file for recording the summary, and writing the summary results to the file. Similarly, when it comes to posting a tweet on a social network (such as X, Weibo, etc.), the instruction must not only include the tweet content but also detail the entire operational process from launching the social network client to completing the post.

[0083] In terms of task instruction design, a notable characteristic of business process tasks is their determinism—non-random tasks with a clear correspondence between actions and outcomes predominate. This means that the general steps required to complete complex, long tasks are highly predictable, fundamentally different from game agent tasks that require rapid, real-time responses. Based on this characteristic, the EGCC framework effectively implements indirect planning of task execution paths by breaking down long-term tasks into multiple phased sub-goals.

[0084] Table 1 Example of automatic operation task instruction text

[0085]

[0086]

[0087] 3. Framework Processing and Edge Computing

[0088] The EGCC framework uses only the target device's HDMI signal as input, using it to monitor the device's current status in real time and evaluate the results of previous operations. The system's output interacts with the target device by generating HID (Human Interface Device) signals that simulate mouse and keyboard operations. To achieve this two-way communication, the system uses edge computing devices as the connection hub between the server and the target device.

[0089] The edge computing device connects to the target device's video output port via an HDMI cable and connects to control terminals such as a mouse and keyboard using a HID cable, establishing a wired connection with the target device. Furthermore, the device maintains communication with a remote server via the internet. The core workflow is as follows: the edge computing device transmits the captured HDMI video stream to the server via the WebSocket protocol. The server makes operational decisions based on the received image data and transmits the instructions back to the edge computing device. The edge computing device then converts the decision instructions into corresponding HID control signals, driving the target device to perform specific operations. As an example, the EGCC framework uses a Raspberry Pi 4B single-board computer combined with the PIKVM-A3 hardware module to build an edge computing device, ensuring efficient video capture and precise HID signal control capabilities.

[0090] PiKVM is a hardware kit based on Raspberry Pi that has virtual power / reset button control, LED status reading and graphics output capture functions. It can also send HID signals through the USB 3.0 interface to simulate mouse and keyboard operations to control the target device. Figure 3 The system operation flow chart integrating the above principles is shown, and the steps are described as follows:

[0091] Step y1, HDMI signal capture: use an HDMI capture card to obtain the video output signal of the target device in real time;

[0092] Step y2, signal conversion and transmission: The HDMI capture card converts the original signal collected in step 1 into a resolvable digital signal and transmits it to the edge computer;

[0093] Step y3, data processing and uploading: The edge computer divides the digital signal from step 2 into independent screenshots and pre-processes them, and then uploads the processed screenshots and recognition results to the server via the WebSocket protocol;

[0094] Step y4, instruction generation and issuance: the server analyzes the data and generates subsequent operation instructions, which are then transmitted back to the edge computer;

[0095] Step y5, instruction execution feedback: The edge computing device converts the operation instruction into a specific HID signal and sends it to the target device to complete the automated operation closed loop.

[0096] This process implements a low-latency perception-decision-execution cycle through edge computing architecture, ensuring that the system can achieve precise control without intruding into the target device.

[0097] To reduce the computational load on servers and large language models, the EGCC framework offloads some basic processing tasks to edge computing devices. Because the framework only observes the target device's status through HDMI signals and is limited by its non-invasive nature, it cannot run control capture programs on the target device. Consequently, it is impossible to directly obtain the number, function, and location of interactive controls through methods such as the DOM.

[0098] Taking clicking the START button as an example, the system cannot directly obtain the precise coordinates of the button. To address this issue, EGCC adopts a graphical GUI parsing strategy: operation positioning is achieved by mapping the pixel coordinates of interface elements in the screenshot to actual physical coordinates. Specifically, when the edge computing device transmits the screenshot to the server, it needs to simultaneously send the scaling factor after resolution adaptation. After the server analyzes the received scaled image to obtain the operation coordinates, it uses the scaling factor to reversely calculate the precise position of the target device in the actual screen coordinate system, thereby avoiding positioning deviations caused by differences in resolution between different devices.

[0099] In the automated operation scenario for computer equipment, in addition to the coordinate conversion mechanism, a basic action instruction set containing all mouse operations must be pre-defined on the edge computing device side. By atomizing the underlying operations, the server can easily call combined instructions. Since edge computing devices achieve computer control by simulating the underlying HID signal, it is only necessary to combine several basic keyboard and mouse operations (such as the drag operation can be decomposed into three steps of "press the left mouse button → move the cursor → release the left button") to cover various complex interaction needs. The basic actions specifically defined in this framework include but are not limited to:

[0100] Mouse operations: left button press / release, right button press / release, scroll wheel, cursor movement;

[0101] Keyboard operation: Single key press / release.

[0102] Furthermore, by fully leveraging the computing power of edge computing devices, some low-computational-load image processing tasks can be completed locally. For example, when performing intelligent operations such as keyword search or text parsing, large language models (LLMs) may be limited by insufficient text processing capabilities when faced with text-intensive interfaces. To this end, EGCC simultaneously performs OCR (optical character recognition) on screenshots on edge computing devices during the digitization of HDMI signals.

[0103] The recognition results are uploaded to the server along with the screenshot data as auxiliary prompts during LLM reasoning. This mechanism, through edge-cloud collaborative processing, effectively offloads server computing pressure and enhances model understanding capabilities through structured text, significantly improving the accuracy and efficiency of automated operations.

[0104] 4. Information acquisition based on screenshots

[0105] Unlike dynamic and unpredictable tasks like playing video games or watching videos, most business processes are static, have low randomness, and predictable outcomes. This eliminates the need for dynamic inputs like video streams, allowing the EGCC framework to acquire all the information needed for decision-making simply by processing and capturing static screenshots of the target device's screen state after the last action, via edge computing devices. These screenshots contain visual information such as GUI layout, graphical elements, and user interaction components.

[0106] By leveraging LLM's OCR capabilities, the system can also extract text information embedded in images. This text information includes file names, control text, operation instructions, and full-text content involved in high-intelligence tasks such as reading and summarizing long documents.

[0107] While screenshots already contain the necessary information for static task analysis and judgment, relying solely on images for information parsing still presents significant challenges compared to traditional automated methods that directly access information through APIs or DOM. This places extremely high demands on the LLM model's spatial cognition and visual understanding capabilities. Furthermore, the LLM model's "hallucination" issues and output instability remain key challenges to overcome in the pursuit of stable and efficient automation tasks.

[0108] To address these issues, the present invention introduces a variety of methods to improve the stability of the LLM model and enhance its capabilities, including providing visual cues, dynamically adjusting the cuing strategy, and combining OCR recognition support with edge computing devices.

[0109] The GPT-4o model (or other large multimodal models, hereinafter referred to) used in this framework serves as a large-scale multimodal language model. It can comprehensively understand and process input text and images, and provide relatively reasonable feedback. However, several issues remain in image recognition: while it can generally identify the information contained in an image, errors can occur at the level of detail. This manifests itself in misjudgments of concepts such as the number of objects, their precise location, and their relative relationships, and it is difficult to accurately convert this information into textual descriptions.

[0110] like Figure 4 As shown in the figure, even when GPT-4o is fed screenshots of relatively low-complexity GUIs, it often fails to respond. The LLM's basic image recognition and OCR functions are not only unable to directly return the precise location coordinates of the corresponding icon, but also have difficulty accurately indicating the specific location of a specific icon.

[0111] The authors speculate that the root cause of this problem lies in the composition of the labeled image datasets used for LLM training—these datasets are primarily based on real-world photographs and lack sufficient labeled data related to computer screen GUIs. This results in the LLM model's image recognition capabilities for computer interface spaces being significantly lower than those for real-world scenes. In this situation, directly applying LLM to automated computer tasks can lead to severe instability. For example, even a simple action like "double-clicking an icon to open an application" can require dozens of attempts before success.

[0112] To address the above issues, the EGCC framework proposes the use of a SoM approach, which adds visual cues to the LLM input image based on textual cues to enhance spatial cognition. The specific implementation process is as follows:

[0113] Step S1: Applying a segmentation model; using the segmentation model to divide the screen elements into regions, identifying all interactive objects such as icons, buttons, and input boxes as independent objects, and marking the pixel areas they occupy with masks;

[0114] Step S2: Generate a matrix; construct a proportional matrix based on the image resolution, where each pixel in the matrix is ​​marked as 1 if it belongs to the corresponding object area, and the remaining pixels are marked as 0;

[0115] Step S3: Create object bounding boxes and digital labels; calculate the average x / y coordinates of all pixels marked as 1 in the matrix, draw the object bounding box based on the coordinates, and assign a corresponding digital label to each element in the screenshot based on the pixel coordinates of each bounding box;

[0116] In this way, the EGCC framework can present various elements on the screen to the LLM in a clearly legible form, thereby enhancing the LLM's spatial cognition ability and achieving efficient and accurate processing of computer tasks.

[0117] The EGCC framework uses the Segment Anything Model (SAM), a general segmentation model proposed by Meta AI. Designed to address the segmentation of arbitrary objects in an image, SAM offers exceptional flexibility and adaptability, adapting to a wide range of images and objects without requiring pre-training for specific tasks or objects.

[0118] SAM has extremely fast inference speed and supports real-time segmentation. It can complete processing within seconds even when faced with complex and large images. Therefore, it is very suitable as an auxiliary module for LLM model image recognition in enhanced automation tasks.

[0119] A limitation of SAM is that its inference performance is highly dependent on high-performance hardware. In particular, when using the Transformer model, it consumes a large amount of video memory (VRAM). Traditional automated methods impose a heavy hardware load on the target device, which can significantly reduce SAM's image processing speed due to the performance limitations of the target device.

[0120] The EGCC framework deploys SAM on a high-performance server adapted to the environment. This architecture allows SAM to maintain the highest image processing speed without placing any burden on the target device. Through this design, the EGCC framework leverages SAM's high performance while effectively avoiding resource constraints or performance degradation on the target device.

[0121] This type of visual enhancement technology not only significantly improves the LLM model's image understanding and spatial cognition capabilities, but also enables the model to express output more clearly and specifically through the digital labels added by the EGCC framework. This capability is an essential core element of the EGCC automation framework.

[0122] like Figure 7 The presented example demonstrates that, in a computer-based image recognition task, LLM significantly improves performance on images augmented with visual labels from the EGCC framework when processing the same screenshots. Furthermore, LLM is able to clearly establish the association between the additional numerical labels and the corresponding icons. When presented with the original image on the left and asked, "What is the position of the Google Chrome icon in the left menu bar?", LLM incorrectly answers, "In the screenshot, the Google Chrome icon is the second icon from the top in the column on the left." However, when presented with the visually cued image on the right, LLM not only correctly answers but also clearly states, "In the column of icons on the left, the Google Chrome icon is the third icon from the top (marked as number 20)."

[0123] By utilizing this feature, the EGCC framework enables the LLM model to accurately and strictly describe the operation target and its specific location, thereby greatly improving the accuracy of target recognition and operation execution.

[0124] 5. Frame Memory and Self-Reflection

[0125] In highly intelligent automated tasks, memory modules and self-reflection capabilities within the framework are essential to prevent scripted operations from stalling or seriously deviating from the primary task objectives, and to effectively utilize information collected in early stages for long-term tasks. The EGCC framework has two core components: the reflective memory module and the information memory module.

[0126] The Reflective Memory module uses the LLM to evaluate whether historical operations were performed normally and whether task objectives were successfully achieved. This module fully records past operations, screenshots of the target device, and corresponding operation instructions, and inputs this information into the LLM model as prompts. The operation instructions field helps the LLM model to smoothly recall the target device's recent status, the operations performed, and their purpose during self-reflection. If an operation fails to achieve the task objective, the system can analyze the failure cause and implement policy adjustments or operational corrections.

[0127] The information memory module is responsible for storing information obtained during automated operations (such as screenshots during scrolling operations), allowing the EGCC framework to simulate the human behavior of recording key information and reviewing it later. In particular, when performing tasks that require intelligent information collection, the framework will call a specially designed skill function to guide the LLM to store the information extracted from the target device screen in a labeled manner, which can then be used to quickly retrieve the information. With the support of this module, the EGCC framework can complete high-level intelligent tasks such as "reading long texts and generating summary emails to specified addresses."

[0128] Key points of the present invention:

[0129] 1. The Generic Edge Computing Device Control (EGCC) framework described in this paper aims to automatically control target devices in exactly the same way as a human operator (human-computer interface), and can be applied to any human-operable electronic device.

[0130] 2. This invention uses a wired connection between the edge computing device and the target controlled device, without requiring or modifying any software or hardware requirements on the target device. Information acquisition and the construction and execution of automatic operations are accomplished in a completely external connection manner. Furthermore, because the framework runs entirely on cloud servers, computing modules with high hardware performance requirements, such as large models, do not impose performance loads on the target device or reduce automatic operation efficiency due to insufficient hardware performance.

[0131] 3. The EGCC framework of the present invention constructs an optional action space for the multimodal large model and provides output examples to standardize the output behavior of the large model. It uses its advanced natural language capabilities and multimodal processing capabilities combined with image processing to make automatic operation decisions, thereby enabling users to complete automatic operation task editing through non-standardized natural language and highly intelligent automatic operation task robustness and executable scope;

[0132] 4. The reflective memory module of the present invention utilizes specific data structures and algorithms to help the LLM smoothly recall the recent status of the target device, executed operations, and their purpose during self-reflection. When an operation fails to achieve the task objective, the system can analyze the failure cause and implement strategy adjustments or operational corrections. The information memory algorithm enables the EGCC framework to simulate the human behavior of recording key information and subsequently reviewing it. The EGCC framework will call specially designed skill functions and has the ability to complete complex warning tasks.

[0133] 5. This type of visual enhancement technology not only significantly improves the LLM's image understanding and spatial cognition capabilities, but also enables the LLM to express output content more clearly and specifically through the digital labels added by the EGCC framework. This capability is an indispensable core element of the EGCC automation framework.

[0134] 6. The EGCC framework only receives the output of the target device to the human-computer interaction interface, and only simulates and inputs the input that the human-computer interaction interface of the device can generate. It communicates with the target device in a way that is completely the same as a human operator for the target device, thereby eliminating the need for the target device to open any ports, set application system APIs, and grant background permissions, thereby minimizing the impact of deployment costs and information security concerns.

[0135] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for performing universal non-invasive automatic operation of a device with a human-machine interface using a multimodal large-scale intelligent agent, characterized in that: The following steps are involved: Step 1, signal capture: The edge computing device directly acquires / identifies the output signal of the target device; Step 2, data upload: The edge computing device directly reads its video stream or converts the original signal collected in step 1 into a parseable digital signal, then divides it into independent screenshots and pre-processes them, and then uploads the processed screenshots and recognition results to the server; Step 3: Instruction generation and distribution: The server analyzes the data and generates subsequent operation instructions, which are then sent back to the edge computing device. Step 4: Instruction execution feedback: The edge computing device converts the operation instruction into a specific HID signal and sends it to the target device, completing the automated operation closed loop; The server is deployed with a large multimodal language model responsible for screen understanding and operation reasoning. It can integrate and process input text and images and provide corresponding feedback.

2. The method according to claim 1, characterized in that In step 1, the edge computing device acquires / identifies the output signal of the target device in the following manner: Method 1: The edge computing device directly obtains / identifies the output signal of the target device to the human-machine interface. For computers, it is a video output signal, and for industrial instruments, it is an electrical signal output to the indicator light. Method 2: Use a camera to shoot the human-computer interaction interface of the target device and directly identify its output and feedback visually.

3. The method according to claim 1, characterized in that In step 2, operation positioning is achieved by mapping the pixel coordinates of the interface elements in the screenshot to the actual physical coordinates. Specifically, when the edge computing device transmits the screenshot to the server, it needs to simultaneously send the scaling coefficient after resolution adaptation processing. After the server obtains the operation coordinates based on the received scaled image analysis, it uses the scaling coefficient to reversely calculate the precise position of the target device in the actual screen coordinate system, thereby avoiding positioning deviations caused by differences in resolution between different devices.

4. The method according to claim 2, characterized in that This method requires pre-defining a basic action instruction set containing all human-computer interaction interface operations on the edge computing device side. By atomizing the underlying operations, the server can easily call combined instructions.

5. The method according to claim 4, characterized in that The human-computer interaction interface includes a computer-type HID and a physical control panel. For the computer-type HID, the defined basic actions include: mouse operation: left button press / release, right button press / release, scroll wheel, cursor movement; keyboard operation: single key press / release. For the physical control panel, it can be used to drive a dexterous hand robotic arm.

6. The method according to claim 2, characterized in that During the digitization of video signals, this method simultaneously performs OCR processing of screenshots on the edge computing device. The recognition results are uploaded to the server together with the screenshot data as auxiliary prompt information for large language model inference. This method only requires the edge computing device to process and capture static screenshots reflecting the status of the target device screen after the last operation to obtain all the information required for decision-making. These screenshots contain visual information about the GUI interface layout, image elements, and user interaction components. This method uses the OCR function of a large language model to extract text information embedded in images, including file names, control text, operation instructions, and read and summarize long documents.

7. The method according to claim 1, characterized in that This method uses the SoM method to add visual cues to the large language input image based on text cues to enhance spatial cognition ability. The specific implementation process is as follows: Step S1: Applying a segmentation model; dividing the screen elements into regions using the segmentation model, identifying all interactive objects as independent objects, and marking the pixel areas they occupy using masks; Step S2: Generate a matrix; construct a proportional matrix based on the image resolution, where each pixel in the matrix is ​​marked as 1 if it belongs to the corresponding object area, and the remaining pixels are marked as 0; Step S3: Create object bounding boxes and digital labels; calculate the average x / y coordinates of all pixels marked as 1 in the matrix, draw the object bounding box based on the coordinates, and assign a corresponding digital label to each element in the screenshot based on the pixel coordinates of each bounding box; In step S1, the segmentation model used is the general segmentation model SAM proposed by Meta AI, and the segmentation model SAM is deployed on the server.

8. The method according to claim 1, characterized in that The method includes: Reflective memory step: The reflective memory module uses the large language model to evaluate whether historical operations were executed normally and whether the task objectives were successfully completed. The reflective memory module can fully record past operation records, screenshots of the target device, and corresponding operation instructions, and input this information into the large language model as prompts. The operation instructions field helps the large language model to smoothly recall the recent status of the target device, the executed operations, and their purpose during self-reflection. When the operation fails to achieve the task objectives, it can analyze the reasons for failure and implement strategy adjustments or operation corrections based on this.

9. The method according to claim 1 or 8, characterized in that The method further includes: Information memory step: The information memory module stores information obtained during the automated operation process, simulating the human behavior of recording key information and subsequently reviewing it. When performing tasks that require intelligent information collection, the skill function is called to guide the large language model to label and store the information extracted from the target device screen, which can then be quickly retrieved through the labels.

10. The method according to claim 1, characterized in that The method further includes: Steps for editing automatic operation tasks: Describe the tasks to be automatically executed on the target device using natural language and upload them to the server.

Images