Ai control device and method

Abstract

[SUMMARY] [PROBLEM]To enable existing systems placed in inviolable domains to utilize autonomous AI without modifying the existing systems that do not have AI. [SOLUTION]An AI control device 10 that physically communicates with a target system 20, comprising: a display information acquisition I / F 1 for acquiring display information from the target system 20; an input information transmission I / F 2 connected to an input device connection portion of the target system 20; an analysis unit 6 for analyzing the display information and extracting display elements contained in the display information; an AI processing unit 9 for executing AI processing based on the extracted information and external information; and a result information input unit 13 for converting result information obtained by the AI processing into physical operation signals and outputting them from the input information transmission I / F 2 to the existing system 20. [SELECTED DRAWING]Figure1

Description

AI CONTROL DEVICE AND METHODTechnical Fieid

[0001] The present invention relates to an AI control device and method.

[0002] Electronic medical record systems and similar systems used in medical environments are designed to remain inviolable from external access. Such systems can be considered as being placed within an inviolable domain. For existing systems within such an inviolable domain, maintaining security is of paramount importance. On the other hand, users desire to introduce AI (Artificial Intelligence) into such existing systems so as to handle their data in a smarter and more efficient manner.

[0003] However, in order to introduce AI into existing systems, it is often necessary to reconstruct or reconfigure such systems. In this case, reviewing or updating the system entails significant costs. Furthermore, the security of existing systems may become compromised if the introduction of AI involves communication with external entities.

[0004] Meanwhile, prior art documents related to these background technologies have been disclosed (for example, Patent Document 1 and Patent Document 2).

[0005] US Patent Application Publication No. 2021 / 0109777US Patent No. 9555544

[0006] The technique disclosed in Patent Document 1 involves a first computer system receiving display signals and audio signals output by a second computer system, determining whether those signals satisfy predetermined conditions, and executing predetermined actions (such as blocking operations, transmitting control signals, generating alerts, and the like) accordingly. This method is configured to trigger actions based on rule-based conditional judgments, and is assumed to cover a wide range of applications including KVM environments and the management of virtual machines.

[0007] In addition, the technique disclosed in Patent Document 2 records images (or videos) of a user manually operating a computer, detects scene changes using computer vision, saves the series of operations as a process definition, and later enables a robot to physically interact with the computer and automatically execute the same operations. Based on the recorded operation and image information, the robot determines click positions by matching with the target screen, and so forth.

[0008] However, the technique disclosed in Patent Document 1 does not disclose a mechanism in which AI analyzes the semantic content of the GUI displayed on the screen or dynamically determines optimal operations according to operational objectives. The determinations are made solely based on pre-defined conditions or rules (such as if-statements), and do not include intelligent contextual understanding or planning for achieving particular purposes.

[0009] Furthermore, the technique disclosed in Patent Document 2 is essentially an automation technology of the "record and playback" type, intended to reproduce a user's operation history, rather than to understand the meaning of the operations and make autonomous decisions. The processing of screen changes is also based on image comparison using threshold values, and does not involve AI analyzing the structure or intent of the GUI. Moreover, the robot's actions follow pre-defined processes and do not assume dynamic decision-making or adaptation to changing environments.

[0010] Accordingly, the present invention aims to address and improve the disadvantages of the above conventional examples, particularly by enabling existing systems placed in inviolable domains, which do not have AI capabilities, to utilize autonomous AI without requiring any modification to those existing systems. Specifically, it is an object of the present invention to enable existing systems to be autonomously operated by AI, by means of AI-based screen analysis and autonomous determination of operation contents.

[0011] The AI control device of the present invention physically communicates with a target system. The AI control device is provided with a display information acquisition interface (I / F) for acquiring display information from the target system, and an input information transmission I / F connected to an input device connection portion of the target system. Furthermore, the AI control device is equipped with an input device connection I / F for connecting input devices, and an external information I / F for acquiring information from external sources. In addition, the AI control device comprises a display information storage unit for storing display information, and an analysis unit for analyzing the display information stored in the display information storage unit and extracting display elements (such as text and / or images) contained in the display information.

[0012] The present invention further comprises an extracted information storage unit for storing the extracted display elements, and an external information storage unit for storing external information acquired via the external information I / F. Additionally, the present invention includes an AI processing unit that executes AI processing based on the information stored in the extracted information storage unit and the external information storage unit, and a result information storage unit for storing result information obtained by the AI processing. Moreover, the present invention is provided with a result information input unit that converts the result information into physical operation signals and outputs them via the input information transmission I / F.

[0013] In the present invention, AI processing is performed based on display elements acquired from the display information obtained from the existing system. In addition, the result information (output information) of the AI processing is converted into physical operation signals, such as keyboard or mouse signals, and input to the existing system.

[0014] Therefore, according to the present invention, existing systems placed in inviolable domains, which do not have AI capabilities, can utilize autonomous AI without requiring any modification to the existing systems.

[0015] Figure 1 is a block diagram showing a first embodiment.Figure 2 is a block diagram showing a first embodiment.Figure 3 is a block diagram showing a modified example of the first embodiment.Figure 4 is a block diagram showing a modified example of the first embodiment.Figure 5 is a block diagram showing a modified example of the first embodiment.Figure 6 is a block diagram showing a modified example of the first embodiment.Figure 7 is a block diagram showing a second embodiment.Figure 8 is a block diagram showing a second embodiment.Figure 9 is a block diagram of the AI processing unit shown in Figure 8.Figure 10 is a flowchart showing the operation of the second embodiment.

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0017] In Figure 1, the AI control device 10 is connected to an existing system 20 placed in an inviolable domain.

[0018] The existing system 20 includes a central system 21, a terminal device 22, and a display device 23. Originally, the existing system 20 also included an input device 24. The input device 24 was connected to an input device connection portion 241 of the terminal device 22. In Figure 1, however, the input device 24 is removed from the terminal device 22 and connected to the AI control device 10.

[0019] The central system 21 provides predetermined services to the terminal device 22 operated by a user. The central system 21 includes servers and databases for providing services to the terminal device 22. In the medical field, for example, the central system 21 may be an electronic medical record system, a medical image management system (PACS), or a combination system linking these components.

[0020] The terminal device 22 is a device operated by a user who has authority to operate the central system 21. The terminal device 22 may be any computer equipment capable of utilizing the services of the central system 21. A display device 23, such as a display monitor, is connected to the terminal device 22. Additionally, the terminal device 22 is equipped with an input device connection portion 241 to which the input device 24 can be connected.

[0021] The input device connection portion 241 is an input interface compatible with the standards of the input device 24. The input device 24 includes a keyboard, mouse, and microphone. The input device connection portion 241 is a serial interface such as a USB interface compatible with these input devices 24. In the medical field, the terminal device 22 may also be diagnostic equipment for image diagnosis or similar purposes.

[0022] In this embodiment, the display device 23 is provided with an output interface, such as HDMI (a registered trademark), for distributing display information shown on the screen to external devices.

[0023] In contrast, the AI control device 10 is equipped with a display information acquisition interface (I / F) 1, an input information transmission I / F 2, an input device connection I / F 3, an external information I / F 4, a display information storage unit 5, and an analysis unit 6. The AI control device 10 further includes an extracted information storage unit 7, an external information storage unit 8, an AI processing unit 9, a result information storage unit 11, an input information transmission unit 12, a result information input unit 13, and a history information storage unit 14.

[0024] These elements are provided within the housing of the AI control device 10. Each of the interfaces 1, 2, 3, and 4 is a hardware resource for wired or wireless connection provided in the housing. Each storage unit 5, 7, 8, 11, and 14 is provided in a storage area of a large-capacity memory. The other units 6, 9, 12, and 13 are implemented by a processor executing predetermined programs.

[0025] The display information acquisition I / F 1 acquires display information displayed on the display device 23. In the present embodiment, the display information acquisition I / F 1 is an HDMI (High-Definition Multimedia Interface) terminal, and receives distributed display information from the display device 23 via an HDMI cable. Alternatively, the display information acquisition I / F 1 may be a connection interface such as USB for a camera that captures the display screen of the display device 23. In this case, the display information acquisition I / F 1 acquires the display information from the camera.

[0026] The input information transmission I / F 2 is connected to the input device connection portion 241 of the existing system 20, where the input device 24 had been connected. The input interface of the input device connection portion 241 is compatible in advance with the output interface of the input device 24. The keyboard or mouse, as the input device 24, supports serial interfaces such as USB, and the microphone, as the input device 24, supports audio interfaces. Similarly, the input information transmission I / F 2 includes serial output interfaces for the keyboard or mouse and audio output interfaces for the microphone.

[0027] The input device 24 is connected to the input device connection I / F 3. As described above, the input device 24 has a serial output interface and / or an audio output interface. The input device connection I / F 3 corresponds to these and includes a serial input interface and an audio input interface.

[0028] The external information I / F 4 acquires information from external sources. The external information I / F 4 may include various types of interfaces in order to support various external devices. For example, the external information I / F 4 may include interfaces such as USB, Wi-Fi, and Bluetooth (all registered trademarks). External devices that can be connected to the external information I / F 4 include sensors, mobile terminals, other AI control devices 10, and servers that provide application programs or modules.

[0029] The display information storage unit 5 stores the display information of the existing system 20 acquired by the display information acquisition I / F 1. As a storage method, it is possible to store a screenshot of the display information according to instructions from the user or the AI processing unit 9.

[0030] The analysis unit 6 analyzes the display information stored in the display information storage unit 5 and extracts text and images contained in the display information. The analysis unit 6 interprets the contents of the display information by AI image analysis and extracts images (photographs) and text contained in the display information. Since the extracted images and text were displayed on the display device 23 of the existing system 20, this means that the data in the existing system 20, which is in an inviolable domain, is read into the AI control device 10.

[0031] The extracted information storage unit 7 stores the text and images extracted by the analysis unit 6.

[0032] The external information storage unit 8 stores external information acquired from the external information I / F 4. External information includes data values from sensors and documents such as academic papers. The external information also includes applications or modules for AI processing used by the AI processing unit 9.

[0033] The AI processing unit 9 performs processing such as generative AI based on information stored in the extracted information storage unit 7, the external information storage unit 8, and the history information storage unit 14. The AI processing unit 9 is equipped with AI processing functions such as AI image analysis and LLM (Large Language Model).

[0034] The result information storage unit 11 stores the result information obtained by the processing of generative AI. The result information includes text or voice generated by the AI processing unit 9.

[0035] The input information transmission unit 12 outputs input information, input from the input device 24, via the input information transmission I / F 2, and transmits it to the existing system 20. The terminal device 22 recognizes the input device 24 as if it were directly connected to the terminal device 22. The terminal device 22 performs processing according to the input information from the input device 24 as usual.

[0036] The result information input unit 13 converts the result information into input information and inputs it to the input information transmission unit 12. Here, the input information has the same specifications as the input information output by input devices 24 such as a keyboard, mouse, or microphone. The result information input unit 13 is equipped with a keyboard emulator. The keyboard emulator converts text as result information into key codes recognized by the OS and applications running on the terminal device 22, and inputs them to the input information transmission unit 12. The result information input unit 13 enables the AI processing unit 9 to input result information to the terminal device 22 in the same way that a user would input information using the input device 24. The input information transmission unit 12 outputs input information, whether it is input from the input device 24 or the result information input unit 13, via the input information transmission I / F 2 without distinction.

[0037] The history information storage unit 14 stores the history of operations performed by the user, the history of program modules used by the AI processing unit 9, past analysis results performed by the analysis unit 6, and process history data executed by the AI processing unit 9.

[0038] Next, the operation of the present embodiment will be described based on Figure 1. The user is authenticated as an authorized user of the existing system 20 through keyboard input or voice input from the input device 24 (hereinafter simply referred to as "operation"), and then operates the terminal device 22 of the existing system 20 as usual. The display device 23 of the terminal device 22 displays images and text based on data managed within the inviolable domain.

[0039] Subsequently, the user operates the input device 24 to request the AI control device 10 to perform predetermined processing. The AI processing unit 9 authenticates and identifies the user prior to handling this request. Applications, program modules, and methods stored in the external information storage unit 8 are managed for each user.

[0040] Upon receiving the request, the AI processing unit 9 sets objectives according to the request, constructs a process for achieving the set objectives, and performs data analysis and output of the analysis results according to the process. The AI processing unit 9 performs analysis using modules linked to the user and data to which access is permitted, as described below.

[0041] ** Examples of data available to the AI ** - Text or image data stored in the extracted information storage unit 7 - Information stored in the history information storage unit 14 (such as past analysis results, process history data, and result values processed by applications) - Sensor data values and documents such as academic papers stored in the external information storage unit 8

[0042] The AI processing unit 9 generates a scenario for the above process and also generates combinations of data and applications to be used in the process. The information and prompts required for this generation include those embedded in the operation program of the AI processing unit 9 and those used as modules as needed. The operation program and the content of the modules of the AI processing unit 9 are updated as necessary. Here, modules available on the Internet may also be used. However, data obtained from the existing system 20 is never stored externally from the AI control device 10.

[0043] Upon receiving the above request, the AI processing unit 9 retrieves the corresponding method from the external information storage unit 8. The AI processing unit 9 also creates an argument container and a checklist in the data cache area. The AI processing unit 9 collects data according to the checklist. At this time, it may acquire screenshots of the display information or obtain audio data from the microphone. Once all items in the argument container are filled by these processes, the final processing result is output.

[0044] The AI processing unit 9 asks the user for confirmation on whether the final processing result may be output to the existing system 20. The user operates the input device to input the confirmation. Upon receiving the confirmation input, the AI processing unit 9 stores the final processing result in the result information storage unit 11 as result information.

[0045] The result information input unit 13 inputs the result information stored in the result information storage unit 11 to the input information transmission unit 12. If the result information is text, the result information input unit 13 uses a keyboard emulator to convert the text as result information into key codes that can be recognized by the terminal device 22 and passes them to the input information transmission unit 12.

[0046] The input information transmission unit 12 inputs the result information, converted into predetermined key codes, to the terminal device 22 via the input information transmission I / F 2. The AI control device 10 automatically inputs the key codes of the result information to the terminal device 22 in place of the user, acting as a robot.

[0047] According to the above embodiment, the AI control device acquires data managed in the inviolable domain of the existing system 20 based on display information obtained from the display device 23 of the existing system 20, and uses it for AI processing. The result information of the AI processing is then input to the existing system 20 and used in its processing. Therefore, according to this embodiment, data of the existing system 20 placed in the inviolable domain can be processed by AI without modifying the existing system 20 that does not have AI.

[0048] Next, Figure 2 illustrates a configuration in which multiple AI control devices 10 are connected to expand AI functions. In the existing system 20 placed in an inviolable domain, the central system 21 is connected to multiple terminal devices 22 via a private network. The multiple terminal devices 22 are configured to utilize the services of the central system 21.

[0049] Each terminal device 22 is connected to an AI control device 10 as described above. Each AI control device 10 can share information and coordinate processing via a computer network connected through the external information I / F 4. For example, one AI control device 10 can access applications and data possessed by another AI control device 10. Furthermore, it is possible to realize online conferences among users utilizing each AI control device 10. In such cases, translation functions, minutes summarization functions, screen sharing functions, and data analysis functions individually possessed by each AI control device 10 can be shared and used collaboratively.

[0050] ** Other utility and modified examples ** Consider utilization in the medical field. Data such as image diagnosis, genetic information, clinical trial information, the latest medical techniques, and the proliferation of measurement terminals for home use, all accumulate as resources for planning and decision-making in treatment. There are aspects where this vast amount of data is not being fully utilized. In addition, medical practice depends on human resources such as the abilities, working hours, and efforts of physicians. In order to preserve human resources while effectively utilizing vast data, it is necessary to have AI that can support more accurate and efficient medical care.

[0051] However, installing applications on the existing system 20 or adding terminals to connect with AI by changing various settings, including security, is difficult. In particular, for hospital electronic medical record systems, modification or alteration of the system is ethically difficult. Upgrading such systems is a challenging situation.

[0052] The AI control device 10 provides an environment where AI can perform the same tasks as humans. The AI control device 10 also enables collaborative work between AI and humans. Furthermore, the AI control device 10 allows for effective utilization of data and improved work efficiency within the scope of user privileges of the inviolable system.

[0053] The AI control device 10 is a system that allows users with viewing and editing privileges for the existing system 20 to handle data in highly secure systems (on-premises and networked) in a smarter manner, without installing applications on the original system, modifying programs, or reconstructing the system.

[0054] The AI control device 10 extends the existing system 20 while maintaining its security. The monitor, keyboard, and various peripheral devices that constitute the existing system are connected to the AI control device 10, thereby configuring and extending a sandbox within the system environment. Users with viewing and editing privileges are able to freely handle the data of the existing system 20 by AI processing. At this time, no modification of programs within the existing system 20 in the inviolable domain or addition of applications for adaptation to a new system occurs. The AI control device 10 does not interfere with modification of the existing system 20 itself.

[0055] Regardless of individual ability, human information processing is limited by time and capacity. Although there is a large amount of useful data, its descriptions are often inconsistent and may not be properly recorded. As a result, it is sometimes not possible to sufficiently consider the necessary data when needed, which can lead to disadvantages. The AI control device 10 provides improvements for the effective utilization of systems and data by subjecting data in the existing system 20 in the inviolable domain to AI processing.

[0056] When a single user tries to assimilate the content of books or academic papers as knowledge, it is a considerable burden to pick out only the appropriate content from vast records while also performing other duties. It is necessary to extract the relevant parts from data recorded in the existing system 20 that a person can view, and to efficiently describe the data on the assumption that it will be extracted. To support these tasks, the AI control device 10 browses and records data in the same way as a human.

[0057] If all data are managed and stored according to predetermined settings or programs and answers are sought by a specific program, then expansion of the existing system 20 would require reconstructing the entire system. In addition, scalable operation and legal confidentiality and security ethics issues must be addressed. Furthermore, whether expanding the existing system 20 itself or migrating to a new system, the costs for development and maintenance become a significant burden.

[0058] The AI control device 10 acts as eyes that "see" through monitors and cameras, as ears that "hear" through microphones, and as a mouth that "communicates" through speakers and keyboards. By installing the AI control device 10 in front of the terminal device 22 of the existing system 20, the AI control device 10 can pseudo-use means of obtaining data in the same way as the user. In this way, the user is supported in handling data in a smarter manner.

[0059] In the existing system 20, whether all the information in the database and big data can be fully utilized and reflected in outcomes depends on the individual's processing capability. The AI control device 10, within security privileges, converts all information that the user can learn (display information) into useful knowledge and decision-making materials (by AI processing), supporting the user to execute jobs (requests) more accurately and efficiently.

[0060] In the above embodiment, the AI control device 10 is configured as an independent housing, but it may also be incorporated into a peripheral device of the existing system 20.

[0061] The display device 23 has been described as an output device of the existing system 20. However, the existing system 20 may also be provided with a speaker that outputs audio. In such a case, an audio acquisition I / F may be provided in place of the display information acquisition I / F 1, and a microphone connected to the audio acquisition I / F may collect sound emitted from the speaker. At this time, audio information may be sampled in place of display information and stored in an audio information storage unit instead of the display information storage unit 5.

[0062] The connection between the AI control device 10 and the existing system 20 may be made via cables such as USB or HDMI. Only physical connection is required between the two, and it is not necessary to install program files or applications on the inviolable system.

[0063] The AI control device 10 may acquire display information of CT medical images displayed on the display device 23 by taking a screenshot, analyze the display information, extract text included in the display information, compare multiple acquired CT medical images, confirm the progress of treatment, and perform other such processes.

[0064] The result information input unit 13 may store multiple sets of key codes in advance for use by the keyboard emulator, and may switch the set of key codes used according to the application or OS interface of the terminal device 22.

[0065] Peripheral devices connected to the external information I / F 4 may include an electroencephalograph.

[0066] Various settings of the AI control device 10 may be managed from an external mobile terminal or tablet terminal via Wi-Fi or Bluetooth included in the external information I / F 4. The same applies to the settings for sensors or other peripheral devices connected to the external information I / F 4.

[0067] The external information storage unit 8 may store encryption keys for communication, user information for user authentication, module information that the AI control device 10 is permitted to use for the user, part of the user data, and configuration data for sensor devices.

[0068] Consider extending an in-hospital system with the AI control device 10. Patient information, for security reasons, is placed in an inviolable system with no external access. Although servers and databases may be located outside the hospital, the system is configured as a closed system. Only authorized users can access it from terminals within the hospital. For more efficient and effective use of data, it is preferable to install AI that can perform the same operations as humans viewing and editing data from in-hospital electronic medical record systems and medical image management systems (PACS) from in-hospital terminals.

[0069] The AI control device 10 basically supports medical practice by grasping analysis content from what is displayed on the screen or from audio input of the existing system 20, acquiring and processing data in accordance with relevant topics, outputting the obtained results, and entering them by voice or into electronic medical records.

[0070] Various examples of the use of the AI control device 10 can be considered. For example, consider upgrading an outpatient clinic. An audio information acquisition I / F (as previously described) in place of the display information acquisition I / F acquires conversation during outpatient consultation. The AI processing unit 9 automatically generates a summary of the conversation based on the audio information and automatically inputs it into the electronic medical record system.

[0071] Next, consider the use as a second-opinion generator. The AI processing unit 9 creates a draft of content to be written in a referral letter, based on the contents of the electronic medical record displayed on the screen of the display device 23.

[0072] Next, consider status and condition monitoring using sensors. The AI processing unit 9 generates a report on progress and status within the treatment plan, based on the content of the electronic medical record displayed on the screen of the display device 23 and various values sent from sensors.

[0073] Next, consider the use in treatment strategy. When the user requests “interpret the CT,” the AI processing unit 9 writes findings in text based on the CT images displayed on the display device 23. Furthermore, the AI processing unit 9 generates a treatment plan in report form based on the findings and test results.

[0074] Next, consider handing over a patient to another doctor. The AI processing unit 9 generates a summary report of the treatment course to date, based on the display information of the electronic medical record shown on the screen of the display device 23. When creating a second opinion, physicians typically write referral letters by hand or on the computer while viewing the patient’s electronic medical record. At such times, since the medical record screen is open, the AI control device 10 can perform data analysis or output a summary of the contents.

[0075] It is also possible to cross-reference sensor data of blood pressure measured in real time with blood test data from the electronic medical record and analyze their correlation.

[0076] Items to be automatically executed by the AI control device 10 and items to be visually confirmed by humans may be set in advance, and the history of who (AI or user) used the AI control device 10 and with what authority may be recorded in the history information storage unit 14. In addition, the history of what images the AI has read may also be recorded in the history information storage unit 14.

[0077] As shown in Figure 2, when multiple AI control devices 10 are connected, for example, analysis modules, sensors, or applications owned by the cardiology department can be used by other departments through the AI control devices 10. In that case, authorization settings for use with other AI control devices 10 and records of how they are used may be managed by a particular AI control device 10.

[0078] The AI control device 10 can perform the same operations as a human, such as visually recognizing the screen and inputting via an input device. For example, the AI control device 10 (i.e., the robot) may be equipped with a virtual keyboard, allowing the AI to use the existing system 20 via Bluetooth communication. At that time, file transfer from the robot can be used to exchange data with the existing system 20. However, in that case, the existing system 20 would require a program to link human input with file input. With an AI control device 10 capable of outputting key codes from a virtual keyboard, data can be exchanged without adding a program to the existing system 20.

[0079] With regard to security management, data handled by the AI control device 10 may be managed using security technologies such as blockchain. Also, user confirmation before inputting result information into the existing system 20 may be performed using a security chip of a device such as a smartphone. Management may also be performed by linking to a unique telephone number.

[0080] The AI control device 10 records the operation history by the user. This is recorded for purposes such as user management and monitoring in collaboration between AI and humans. However, it is also intended to enable the reproduction of human actions, machine actions, and the results thereof. For example, it is possible to create a movie of mouse operations or convert an operation flow into program code based on the history.

[0081] This makes it possible to teach users or other AI control devices what should be done to achieve intended results as “experience.”

[0082] ** Modified Example 1 ** Next, the first modified example of the above embodiment will be described with reference to Figure 3. The same reference numerals are given to the same components as those in the above embodiment, and redundant description is omitted.

[0083] As in the above embodiment, the AI control device 10 exchanges information with the existing system 20, including the central system 21 and terminal device 22, via the display information acquisition I / F 1 and the input information transmission I / F 2. This information exchange is carried out by wired or wireless connection. The AI control device 10 also exchanges information with external devices 31, tablet terminals, or mobile terminals 32 (hereinafter collectively referred to as “devices 31, 32” as appropriate) via the external information I / F 4. In Figure 3, these information lines are indicated by solid lines.

[0084] On the other hand, the exchange of information between the AI control device 10 and the existing system 20 may also be performed via a server 34 and a computer network. Similarly, the exchange of information between the AI control device 10 and external devices may also be performed via a server 35 and a computer network (in Figure 3, information lines for exchanges performed via a computer network are shown as dotted lines).

[0085] In Figure 3, the server 34 connected between the AI system 10 and the existing system 20 is equipped with KVM over IP (a KVM switch via IP connection, or remote KVM) functionality. KVM stands for Keyboard, Video (Visual unit), Mouse. In this case, remote KVM software is installed on the server 34 and operates using the resources (CPU, memory, network) of the server 34. The AI control device 10 connects via a network to the remote KVM software on the server 34 and controls the KVM switch connected to the server 34 to operate the existing system 20 at a remote location.

[0086] The server-integrated terminal 35 is a small computer that serves some or all server functions. The server-integrated terminal 35 is a device equipped with functions such as file server, print server, and web server. The server-integrated terminal 35 may also collect and process data from IoT devices acting as device 31.

[0087] The server-integrated terminal 35 transmits information received from devices 31, 32 to the AI control device 10 via a computer network. The server-integrated terminal 35 also transmits information received from the AI control device 10 to devices 31, 32 via a computer network.

[0088] According to this modified example 1, in addition to the same effects as the above embodiment, the following advantageous effect is achieved. That is, the AI control device 10 can exchange information with the existing system 10 and devices 31, 32 located remotely via a server. Therefore, the AI control device 10 can also use remote devices 31, 32 as input and output devices. Specifically, as input devices, the AI control device 10 can use the microphone or touch panel input of a tablet terminal or mobile terminal 32 located remotely. As output devices, the AI control device 10 can use the display or speaker output of a tablet terminal or mobile terminal 32 located remotely.

[0089] ** Modified Example 2 ** Next, a second modified example of the above embodiment will be described with reference to Figure 4. The same reference numerals are assigned to the same parts as those in the above embodiment and modified examples, and redundant description is omitted.

[0090] Figure 4 shows two remote facilities, A and B. The following describes an example in which each of facilities A and B is a medical facility. However, facilities A and B may also be other types of facilities.

[0091] Facility A includes an AI control device 10, an existing system 20, and an input device 24 operated by physician a for inputting into the existing system 20. On the other hand, facility B includes a server 41 with KVM over IP functionality, a computer terminal 42, and an input device 44 operated by physician b. The existing system 20 includes, for example, an EMR system (Electronic Medical Record system) as the central system 21, which electronically records and manages patients' medical information.

[0092] The server 41 has KVM over IP functionality. As a result, the server 41 receives screen display information of the existing system 20 sent from the AI control device 10 via a computer network and provides it to the computer terminal 42. The server 41 also transmits keyboard and mouse input information input from the input device 44 to the AI control device 10 via the computer network.

[0093] The AI control device 10 in facility A and the server 41 in remote facility B are connected via a computer network. The connection between these two is a secure connection, protected for information confidentiality, integrity, and availability by connection management functions provided by the AI control device 10 (such as encryption, authentication, access control, digital signature, VPN, etc.).

[0094] Physician a in facility A can input necessary information from the input device 24 while viewing the screen of the existing system 20 and also utilizing support functions of the AI control device 10, as described in the above embodiment. Meanwhile, physician b in remote facility B can also view the screen of the existing system 20 and support information generated by the AI control device 10, on the screen of his or her own computer terminal 42 via the KVM over IP function of server 41. Physician b then makes necessary inputs from the local input device 44 into the computer terminal 42. This input information is input into the AI control device 10 via the KVM over IP function of server 41.

[0095] As a result, physician b in remote facility B can obtain screen display information from the existing system 20 and the AI control device 10 in facility A, and make necessary inputs remotely into the AI control device 10 and the existing system 20. Therefore, physician b can utilize the functions of the existing system 20 from a remote location. Furthermore, physician b can also utilize medical devices connected to the AI control device 10, application functions provided by the AI control device 10, and the expertise of physician a in facility A. Thus, during medical care, physician b can not only use facility B’s system but also obtain necessary advice and support from facility A’s system and physician a.

[0096] In some regions, there are few specialists, and a single physician may serve as a general practitioner. Even in such cases, the system of this modified example allows navigation of medical care tailored to the patient's condition based on specialized knowledge and expertise, enabling regional physicians to obtain professional support. Similarly, the system configuration of this modified example supports collaboration among multiple facilities and home care.

[0097] In addition, with the connection management function provided by the AI control device 10, calculations and information processing using AI can be performed securely even among multiple remote systems. By appropriately setting the security level of the connection management function of the AI control device 10 according to the connected remote systems, it becomes possible to connect simultaneously to systems with different security levels.

[0098] ** Modified Example 3 ** Next, a third modified example of the above embodiment will be described with reference to Figure 5. The same reference numerals are assigned to the same parts as those in the above embodiment and modified examples, and redundant description is omitted.

[0099] In the above embodiment, the result information input unit 13 of the AI control device 10 is configured with a keyboard emulator. However, the configuration is not limited to this; as shown in Figure 5, the keyboard emulator 51 may also be provided outside the AI control device 10. The USB keyboard emulator 51 is a device or software recognized as a USB keyboard by the computer terminal of the existing system 20. The USB keyboard emulator 51 converts signals output by the AI control device 10 to the existing system 20 into keyboard input and sends them to the terminal device 22 of the existing system 20.

[0100] In this modified example, the AI control device 10 has a built-in server function and exchanges information with a tablet terminal or mobile terminal 32 as a client via a computer network.

[0101] In the above embodiment, the AI control device 10 directly acquires screen display information from the display device 23 of the terminal device 22 included in the existing system 20. However, the configuration is not limited to this; a tablet terminal or mobile terminal 32 may acquire the screen display information from the display device 23, and then the AI control device 10 may acquire the same screen display information from the tablet terminal or mobile terminal 32 using its server function. In this case, an HDMI capture device may capture the screen display information from the display device 23 and input it to a dedicated app on the tablet terminal or mobile terminal 32. Even in such a configuration, the same effects as the above embodiment and modified examples can be achieved.

[0102] ** Modified Example 4 ** Next, a fourth modified example of the above embodiment will be described with reference to Figure 6. The same reference numerals are given to the same parts as those in the above embodiment and modified examples, and redundant description is omitted.

[0103] When the AI control device 10 and the existing system 20 are connected via a LAN (local area network), as shown in Figures 6(A) and 6(B), a hardware-based KVM over IP or software-based KVM over IP (hereinafter simply referred to as “remote KVM 61”) operating on the LAN may be introduced between the AI control device 10 and the existing system 20.

[0104] In the example of Figure 6(A), the AI control device 10 acquires screen display information from the existing system 20 via the remote KVM 61 and, using its built-in server function, provides the same screen display information to the tablet terminal or mobile terminal 32 via a computer network. In addition, the AI control device 10 receives operation information input by keyboard or pointer to the tablet terminal or mobile terminal 32, via its built-in server function and computer network, and inputs the same operation information to the existing system 20 via the remote KVM 61.

[0105] In the example of Figure 6(B), the AI control device 10 acquires screen display information from the existing system 20 via the remote KVM 61. The AI control device 10 also inputs keyboard and mouse operation information to the existing system 20 via the KVM device 61. The exchange of this information may be performed via a dedicated application 62 that supports the API (Application Programming Interface) provided by the KVM device 61. Even in such a configuration, the same effects as the above embodiment and modified examples can be achieved.

[0106] ** Overview of the Embodiment ** Next, the second embodiment of the present invention will be described. This embodiment is directed to an AI control device 10 and an AI control method that utilize a physical KVM (Keyboard, Video, Mouse) interface. The overall configuration of this embodiment is fundamentally based on the first embodiment, as illustrated in Figures 1 to 6. Components that are identical to those of the first embodiment are denoted by the same reference numerals, and redundant explanation is omitted for clarity.

[0107] In this embodiment, as shown in Figure 7, the AI control device 10 is physically connected to the user interface of an existing (target) system 20 via a KVM interface 61. The KVM interface integrates signals for keyboard, monitor, and mouse and is configured to allow direct input and output to the target system 20. Notably, the KVM interface serves as a physical pathway, enabling GUI operations on the target system 20 to be performed autonomously, without human intervention.

[0108] The AI control device 10 is configured to acquire, in real time, screen information (video signals) from the target system 20 via the KVM interface 61. The acquired screen data is subjected to analysis by the AI processing unit. Based on the analysis results, the AI control device 10 generates physical operation signals (keyboard and / or mouse signals) and transmits them to the target system 20 through the KVM interface 61. In this manner, the AI control device 10 enables autonomous external control without making any changes to the internal software of the target system 20.

[0109] The aforementioned AI processing-including analysis, decision-making, and planning-may be executed locally within the AI control device 10, remotely via an AI cloud 71 or AI network server 72, or in a distributed (hybrid) configuration utilizing both resources. The device is configured to seamlessly switch between these processing modes in response to system load and operational requirements, thereby maintaining optimal performance at all times. Furthermore, distributed processing may be realized by networked cooperation among multiple AI control devices 10, enabling resource sharing, load balancing, and enhanced redundancy.

[0110] In highly regulated environments-such as medical, industrial, and infrastructure systems-legacy systems are often restricted from the installation of new software or agents, owing to security, reliability, and compliance considerations. Accordingly, the application of software-based RPA or API-integrated automation solutions is generally infeasible in such circumstances.

[0111] It is to be understood that conventional KVM-over-IP devices and remote control devices (e.g., PiKVM, TinyPilot) are premised on remote human operation and do not support fully autonomous AI control. While recent proposals (such as US20210109777A1 and US9555544B2) enable partial screen monitoring and control via KVM, they fail to offer a robust agentless automation framework with flexible AI resource allocation, a non-intrusive architecture, secure and reliable logging, fallback mechanisms, and distributed multi-system support.

[0112] Accordingly, it is an object of the present embodiment to provide an AI control device and method capable of achieving fully autonomous, agentless, and non-intrusive control over existing business systems and control devices (target system 20), without any modification to software or network configuration, and independently of the OS boot state or network presence.

[0113] In particular, in fields such as medicine, industry, and infrastructure, there are numerous cases where agent installation, API integration, software modification, or even network access to the target system 20 is expressly prohibited. Even in such environments, it is essential to realize fully autonomous external operation and monitoring, by means of a method that is entirely non-intrusive, both physically and logically, with respect to the target system 20.

[0114] The present embodiment enables the AI control device 10 to acquire and control both the screen output and input signals of the target system 20 bidirectionally via the KVM interface 61. The device is configured to perform automatic emulation of GUI operations, based on AI-driven screen recognition, state assessment, and operation decision-making. As a result, fully autonomous operation on the existing GUI is accomplished, without the need for human intervention.

[0115] In addition, the AI control device 10 is configured to dynamically switch between local (AI processing unit 9) and remote (AI cloud 71 or AI network server 72) AI processing modes, in accordance with the network status and operational requirements. The device is further provided with a fallback mechanism, such that, in the event of network failure or AI processing error, processing is automatically and seamlessly switched to local AI mode, thereby ensuring uninterrupted, highly reliable operation.

[0116] Furthermore, the AI control device 10 is provided with secure logging capabilities, anomaly detection and fallback processing, multi-layer authentication (including passwords, tokens, TPM, biometrics), and a scheduling function for simultaneous or sequential control of multiple target systems. Accordingly, the device can be operated reliably and efficiently even in highly regulated and large-scale environments.

[0117] Thus, the present embodiment comprehensively fulfills the requirements for next-generation AI control of legacy systems, including non-intrusiveness, agentlessness, autonomous operation, high reliability, support for multiple systems, and network independence.

[0118] ** Configuration of the Present Embodiment ** The AI control device 10 according to the present embodiment is configured as follows. It enables fully autonomous and non-intrusive operation for target systems 20 where software modification or agent installation is not feasible. In particular, the AI control device 10 is physically connected to the target system 20 via a KVM interface 61 and completes all operations-from screen acquisition to the output of physical operation signals-within the external device itself. The structural configuration is shown in Figures 1 and 8.

[0119] (1)Display Information Acquisition I / F 1 The AI control device 10 is configured to capture video output from the target system 20 via physical interfaces such as HDMI, DisplayPort, or VGA. Thus, the device is able to acquire GUI-based screen information in real time, without any intervention on the software side. The device is operable even in offline or pre-boot states, such as the BIOS setup screen.

[0120] (2)Display Information Storage Unit 5 (Storage Device) The acquired screen information is stored, either temporarily or permanently, in a display information storage unit 5 (e.g., RAM, SSD). The stored images are used for AI analysis, operational decision-making, logging, and subsequent verification, ensuring both high-speed processing and record integrity.

[0121] (3)AI Processing Unit 9 (AI Processing Module) The AI processing unit 9 is the core processing module of the AI control device 10 and, as shown in Figure 9, includes the following functions and elements.

[0122] * Configured to perform standalone AI analysis and operational decision-making via the device’s built-in processor.

[0123] * Configured to utilize distributed computational resources on an external AI cloud 71 or AI network server 72 via network connection, thereby enabling high-performance decision-making.

[0124] * Configured to distribute AI processing between the device’s internal processor and external cloud / network servers.

[0125] * Analysis Unit 6 (Analysis Module): Primarily responsible for extracting data such as text and images from display information and the like. The analysis unit 6 includes, for example, the following functions: - Object detection by CNN or the like - Text extraction by OCR (Optical Character Recognition) - Recognition of button states and transition states in GUIs

[0126] * Analysis Logic 94: Implements algorithms for data analysis, decision-making, and planning, including planning logic for operational task sequencing, based on extracted data and external information.

[0127] (4)External Information I / F 4 (Communication Module) The external information interface 4 is arranged to provide secure data communication with an AI cloud 71 or AI network server 72, employing protocols such as TLS or VPN. It also supports distributed AI processing among multiple AI control devices 10, ensuring the confidentiality and authentication of all communication channels.

[0128] (5)Result Information Input Unit (Operation Signal Generator) The result information input unit 13 is configured to generate physical operation signals (e.g., USB-HID signals) simulating human keyboard / mouse operations, based on the processing results of the AI module. These signals are output to the target system 20 via USB. The device maintains non-intrusive and agentless control, even during OS boot or network outages.

[0129] (6)Logging Unit 81 The logging unit 81 is configured to record, in encrypted form and with timestamps, all processing results from the AI unit and physical operation signals generated by the result information input unit. Logs are protected against tampering and serve as audit trails, evidence records, and operational traceability, particularly suited to regulated industries such as healthcare and finance.

[0130] (7)Authentication Unit 82 The authentication unit 82 provides multi-factor authentication (passwords, tokens, biometrics such as fingerprint or facial recognition, hardware-based authentication including TPM) to ensure the legitimacy of users or cooperating AI modules.

[0131] (8)Fallback Module 83 The fallback module 83 is configured to automatically switch to local AI processing upon detection of network failure, cloud AI response failure, or abnormal states, thus ensuring uninterrupted operation and high system reliability even under adverse conditions.

[0132] (9)Scheduler 84 (Optional Feature) The scheduler 84 centrally manages the order, priority, and timing of control for multiple target systems 20, whether operated simultaneously or sequentially, thereby optimizing control efficiency and resource allocation in multi-system environments.

[0133] ** System Implementation ** The AI control device 10 is physically connected to the HDMI and USB ports of the target system 20, and is configured to non-intrusively acquire screen output and emulate user operations. The device may be implemented using a general-purpose PC, single-board computer (e.g., Raspberry Pi), custom embedded board, or a dedicated edge gateway, depending on operational requirements.

[0134] It is to be understood that the AI control device 10 may be realized on general-purpose hardware, dedicated hardware (FPGA, ASIC, custom circuits), or any suitable combination thereof. Dedicated hardware is particularly suitable for use in medical or safety-critical systems.

[0135] The AI control device 10 can be configured by installing AI software on a PC, Raspberry Pi, or other single-board computer. For example, a configuration in which an HDMI capture module is connected to a Raspberry Pi to acquire the screen of the target system, and a USB-HID emulator is used to output keyboard and mouse input signals, is one example.

[0136] Dedicated hardware, including FPGA, ASIC, and custom boards, is intended to address requirements such as high reliability, processing speed, security, power management, and compliance with medical standards, all of which may be difficult to meet with PC-based systems. This may include the use of dedicated SoCs, security chips, real-time operating systems, and hardware encryption technologies.

[0137] In any configuration, the core functions are common-namely, screen acquisition via physical interfaces, AI processing, and generation of physical inputs-serving as a form of the hardware platform for the AI control device 10.

[0138] The AI processing unit 9 and its software modules (such as object detection, OCR, planning, etc.) can be executed in local, remote, or distributed / hybrid configurations. This enables the system to flexibly respond to the presence or absence of a network and to processing capacity constraints, achieving secure and autonomous operation.

[0139] During operation, the AI processing unit 9 determines, in real time and based on user policy, network conditions, and task content, whether to process AI tasks locally or to transfer data to a remote AI cloud 71 or network server 72. In the event that the remote AI environment becomes temporarily unavailable, processing is automatically and seamlessly switched to local AI, without any interruption of system control.

[0140] Accordingly, the present embodiment provides a robust, non-intrusive, and agentless AI control device, capable of stable and secure operation even in highly regulated or mission-critical environments.

[0141] ** Operational Example of the Present Embodiment ** An operational example of the present embodiment will be described with reference to the flowchart in Figure 10.

[0142] Step S1: User / Device Authentication The authentication unit 82 performs authentication at startup or when an operation request is received, using passwords, tokens, biometrics, or hardware security modules, thereby preventing unauthorized access or operation. Mutual authentication with the AI cloud 71 or AI network server 72 may also be performed. Required authentication information is input through the external information interface 4 or the input device 24.

[0143] Step S2: Video Frame Acquisition The display information acquisition interface 1 captures the screen output signals of the target system 20 in real time and stores them as video frames in the display information storage unit 5. Screen output signals can be acquired even when the OS is not running or during the BIOS screen.

[0144] Step S3: Selection of AI Processing Route When the network is available, the AI processing unit 9 activates the remote AI mode 92, encrypts the acquired video frame data, and transmits it to the AI cloud 71 or AI network server 72 via the external information I / F 4 (secure communication channel).

[0145] On the other hand, when the network is disconnected or immediate responsiveness is required, the AI processing unit 9 automatically selects the local AI mode 91 and performs AI analysis internally.

[0146] The AI processing unit 9 may also, as needed, activate the hybrid AI mode 93 to divide analysis processing, for example by performing object detection and OCR locally, while delegating complex decision-making and planning to the cloud.

[0147] Steps S4-S6: Execution of AI Analysis The AI processing unit 9, AI cloud 71, or AI network server 72 execute, either sequentially or in parallel, object detection (CNN), text recognition (OCR), GUI element state recognition, and planning for action selection, thereby generating operation commands (action commands) for the target system 20. In local or hybrid modes, these tasks may be executed by the analysis unit 6 and analysis logic 94, using generative AI 9a as appropriate.

[0148] The AI processing unit 9 receives, as input, elements extracted from display information (such as images, text, and GUI structures), and recognizes screen elements using object detection models such as CNNs and OCR engines. Subsequently, based on the recognition results, user goals, operation history, and the like, a planning AI (for example, a reinforcement learning model or an LLM-based task instruction generation model) automatically generates action commands. These AI models are constructed and optimized by training on actual business screen data and operation history data. Furthermore, the AI models can be executed either on a cloud implementation, a local implementation, or both, and support remote and automatic updating of model parameters and inference logic.

[0149] Step S7: Acquisition and Generation of Action Commands - In remote AI mode 92, the device receives action commands generated on the cloud side. - In local AI mode 91, action commands are generated internally. - In hybrid AI mode 93, the device generates action commands based on both internal and cloud analysis results.

[0150] Step S8: Generation and Transmission of Physical Operation Signals The result information input unit 13 emulates USB-HID-compliant physical operation signals (keyboard input, mouse movement, clicks, etc.) based on the action command, and physically sends them to the target system 20. In this way, GUI operation is performed fully autonomously without human intervention.

[0151] Examples of action commands include: - Recognizing a specific button on the screen and performing a click operation - Detecting a text input field and entering a predetermined string - Generating and executing a macro sequence (multiple window operations, screen transitions, automated report creation) based on AI planning

[0152] During operation, the logging unit 81 records all analyses, decisions, operations, and results in an encrypted log format, ensuring tamper resistance and complete auditability. Error events and processing transitions are also logged to support root cause analysis in the event of system failures.

[0153] Furthermore, the fallback module 83 is configured to detect communication failures, AI processing errors, or abnormal system responses and to automatically trigger fallback mechanisms such as switching to local AI mode or retrying operations. This ensures uninterrupted and continuous operation.

[0154] The AI control device 10 may also perform control processing for multiple target systems 20 sequentially or in parallel, or may distribute AI processing among multiple AI control devices 10 for cooperative control and load balancing within a distributed AI network.

[0155] Additionally, Where permitted by policy, the device is capable of securely updating internal AI models, control logic, and security certificates via encrypted channels, thereby supporting ongoing performance and functionality improvements throughout long-term operation.

[0156] [[Industrial Applicability] The AI control device 10 is widely applicable across industries, with the flexibility to allocate and switch processing between local and cloud environments as needed. Representative use cases are provided below.

[0157] * Healthcare - EMR Terminal Automation In hospitals and clinics, EMR terminals are subject to strict limitations on internal modification for security and patient data protection. The AI control device 10 can be connected non-intrusively at the HDMI / USB level, performing screen recognition and automatic input without the need for agents. Lightweight processing such as OCR and GUI element analysis is executed locally, while heavy reasoning and decision support may be delegated to cloud resources, ensuring both regulatory compliance and robust operation even in environments with limited connectivity.

[0158] * Factory Automation - HMI Control Industrial HMI and control panels often require continuous, around-the-clock operation. The device enables non-intrusive automation of touch panel operations and screen change recognition. Advanced optimization and anomaly detection may be performed on the cloud in normal operation, with control automatically switching to local AI mode 91 during network failure, thus minimizing downtime and maintaining productivity.

[0159] * ATM / Infrastructure: Terminal maintenance and security support In domains such as ATMs, government, and transportation infrastructure, where high physical security and tamper resistance are required, the device provides non-intrusive control at the USB-HID and video signal level, supporting screen state monitoring and autonomous maintenance operations without internal modification or software installation. Remote AI mode 92 performs anomaly detection and risk assessment, while local AI mode 91 executes emergency response and initial recovery, ensuring both security and operational reliability.

[0160] The present embodiment described above adopts a novel configuration that is fundamentally distinct from conventional KVM-over-IP devices, RPA (Robotic Process Automation) technology, and existing USB / HDMI-connected remote operation devices (e.g., PiKVM, TinyPilot, etc.), and achieves the following innovative effects.

[0161] 1.Distributed AI processing and fallback mechanism The device is configured to process AI tasks for GUI screens acquired via the KVM interface by means of either local or remote modules, with dynamic switching between modes as required.

[0162] Based on factors such as network bandwidth, latency, security policy, the nature of AI tasks, and processing load, the AI control device 10 is configured to dynamically select and switch to the optimal processing mode, and in the event of communication disruption, it autonomously falls back to local AI mode 91 to ensure continuous processing.

[0163] 2.True Non-intrusive and Agentless control By combining physical video signal capture with USB-HID keyboard and mouse input emulation, the device provides fully autonomous control of target systems from outside, regardless of OS state or network connectivity.

[0164] For example, AI-based screen analysis and autonomous operation can be achieved even in a UI that is in an unbooted OS state, such as the BIOS setup screen or a recovery UI after a crash. This eliminates the need for any API access, software modification, or agent introduction to the target system 20, enabling completely external and non-intrusive control.

[0165] 3.Security, Logging, and Update Integration The fallback module 83 provides failover functionality in the event of AI or network failure. The logging unit 81 securely records all operations and AI decisions in an encrypted, tamper-resistant format, supporting both auditing and reproduction of system behavior.

[0166] Furthermore, the AI control device 10 supports remote updating of firmware and AI modules, enabling the device to maintain security and reliability over extended operational periods.

[0167] 4.Scalable Multi-System Distributed Control Using the scheduler 84, a single AI control device can control multiple target systems in parallel or sequence, or multiple devices can be networked for distributed AI control, thereby supporting high-availability environments with dynamic resource allocation and redundancy.

[0168] Examples of such configurations include a single AI control device 10 controlling multiple target systems 20, multiple AI control devices 10 redundantly controlling a single target system 20, or delegation / sharing of AI processing. This excels at dynamic resource allocation and redundancy assurance, enabling compatibility with high-availability environments such as factories, hospitals, and data centers.

[0169] 5.Adaptability to Highly Regulated Fields The device’s agentless and non-intrusive design enables safe automation of legacy systems in domains such as healthcare, finance, infrastructure, and manufacturing, serving also as a bridge for future system upgrades and integration.

[0170] 6.Clear Distinction from Prior Art It is to be understood that conventional remote operation devices presuppose human operation, and RPA-like approaches are limited in practice due to requirements for software installation or API connections. The present invention provides a fundamentally new approach for agentless, fully autonomous external control.

[0171] Moreover, prior art such as US20210109777A1 and US9555544B2 is limited to monitoring and fixed control, and does not provide comprehensive GUI-based AI analysis, autonomous operation, distributed control, or multi-system support. The present embodiment overcomes these limitations and enables practical deployment of next-generation AI autonomous control infrastructure.

[0172] It is to be understood that the invention is not limited to the embodiments and examples described herein. Various modifications, changes, and substitutions may be made without departing from the spirit and scope of the invention as defined in the appended claims.

[0173] For example, the described embodiments and examples are provided for clarity and completeness of explanation, and do not limit the invention to only the structures and features described. Other configurations, additions, deletions, or substitutions may be made as appropriate.

[0174] In addition, the control and information lines depicted in the drawings are shown only as necessary for the explanation, and are not intended to represent every line present in the actual product. In practical implementation, it is to be understood that nearly all components may be considered interconnected.

[0175] 1: Display information acquisition I / F, 2: Input information transmission I / F, 3: Input device connection I / F, 4: External information I / F, 5: Display information storage unit, 6: Analysis unit, 7: Extracted information storage unit, 8: External information storage unit, 9: AI processing unit, 9a: Generative AI, 10: AI control device, 11: Result information storage unit, 12: Input information transmission unit, 13: Result information input unit, 14: History information storage unit, 20: Existing system, 21: Central system, 22: Terminal device, 23: Display device, 24: Input device, 31: Device, 32: Tablet terminal or mobile terminal, 34: Server, 35: Server-integrated terminal, 41: Server, 42: Computer terminal, 44: Input device, 51: USB keyboard emulator, 61: Remote KVM (KVM device), 62: Dedicated application, 71: AI cloud, 72: AI network server, 81: Logging unit, 82: Authentication unit, 83: Fallback module, 84: Scheduler, 91: Local AI mode, 92: Remote AI mode, 93: Hybrid AI mode, 94: Analysis logic, 241: Input device connection portion

Claims

An AI control device that physically communicates with a target system, comprising:a display information acquisition I / F for acquiring display information from the target system;an input information transmission I / F connected to an input device connection portion of the target system;an input device connection I / F for connecting an input device;an external information I / F for acquiring information from outside;a display information storage unit for storing the display information;an analysis unit for analyzing the display information stored in the display information storage unit and extracting display elements contained in the display information;an extracted information storage unit for storing the extracted display elements;an external information storage unit for storing external information acquired via the external information I / F;an AI processing unit for executing AI processing based on information stored in the extracted information storage unit and the external information storage unit;a result information storage unit for storing result information obtained by the AI processing; anda result information input unit for converting the result information into physical operation signals and outputting them from the input information transmission I / F.The AI control device according to claim 1, whereinthe result information input unit outputs the result information from the input information transmission I / F when confirmation is received from the user via the input device.The AI control device according to claim 1,wherein the external information interface includes an interface for connecting sensor devices not provided in the target system.The AI control device according to claim 1,wherein the AI processing unit determines the processing content based on a request input by a user through the input device.The AI control device according to claim 1,wherein the external information interface includes a communication interface for communicating with another AI control device.The AI control device according to claim 1,wherein the external information interface includesa display information transmission function for transmitting display information of the target system to a remote computer terminal via a server, andan input information reception function for receiving input information input into the computer terminal from the input device via the server.An AI control device comprising:(a) a display information acquisition I / F configured to acquire display information from an external target system;(b) a display information storage device for storing the display information;(c) an AI processing unit that performs analysis and decision-making by AI processing, said AI processing unit being implemented locally, remotely on a cloud / network server, or in a distributed manner, and supporting dynamic switching between local processing and remote / cloud processing;(d) a communication module configured to perform secure, authenticated data transfer with a cloud or network server; and(e) an operation signal generation unit configured to generate keyboard signals and / or mouse signals (physical operation signals) corresponding to the processing results of the AI processing unit and transmit them to the target system via a physical interface;wherein the AI control device enables fully autonomous, agentless, and non-intrusive operation for the target system, and supports dynamic local and / or remote (cloud) AI processing.The AI control device according to claim 7, further comprising:(f) a logging unit configured to record secure and tamper-resistant audit logs.The AI control device according to claim 7, further comprising:(g) an authentication unit for authenticating a user and / or a device that executes the remote AI processing.The AI control device according to claim 7, further comprising:(h) a fallback module for performing error detection and fallback processing to a local AI mode.The AI control device according to claim 7, further comprising:(i) a scheduler for managing operations for multiple target systems.The AI control device according to claim 7,wherein the physical interface includes a USB-HID interface.The AI control device according to claim 9,wherein the authentication unit supports password authentication, token authentication, or biometric authentication.The AI control device according to claim 7,wherein the AI control device is configured to allow secure remote updating of internal AI models or control logic.The AI control device according to claim 7, further comprising:(f) a logging unit configured to record secure and tamper-resistant audit logs;(g) an authentication unit for authenticating a user and / or a device that executes the remote AI processing; and(h) a fallback module for performing error detection and fallback processing to a local AI mode.An AI control method comprising the following steps:(k) authenticating a user of the AI control device and / or a cloud / network server that executes AI processing in cooperation with the AI control device;(l) after authentication, acquiring display information from the target system;(m) selecting an AI processing route from at least local or remote;(n) executing AI processing by the selected route and obtaining an action command corresponding to the display information; and(o) converting the action command into a KVM physical operation signal and inputting it into the target system via a physical interface.

Images