IPKVM-based remote automatic control system and method, and electronic device

Abstract

The application provides a remote automatic control system and method based on IP KVM and electronic equipment. The system comprises a local client, an IP KVM and a server. The local client is in communication connection with the IP KVM. The IP KVM is in communication connection with the server. The IP KVM is also in communication connection with multiple remote servers. The server is used for acquiring first running data in real time. The first running data is running data of the remote server. The server identifies a first key feature contained in the first running data. The server judges whether the first key feature is the same as a second key feature. The second key feature is a key feature marked in advance from second running data of the remote server. If the first key feature is the same as the second key feature, the operation instruction for the second key feature is retrieved. The server converts the operation instruction into a keyboard and mouse control signal and sends the keyboard and mouse control signal to the remote server. The application has the effect of improving the remote control efficiency of the remote server.

Description

Technical Field

[0001] This application relates to the technical field of remote control, specifically to a remote automatic control system, method, and electronic device based on IP KVM. Background Technology

[0002] IP KVM, also known as KVM over IP, is a KVM switch with remote management capabilities. KVM stands for Keyboard, Video, and Mouse, used to control multiple computers using a single keyboard, monitor, and mouse. Typically, due to signal attenuation during transmission from the computer to peripheral devices, analog KVM switches are usually located relatively close to the controlled computers. IP KVM, as a digital KVM switch, can transmit signals from a remote server to the local client via the internet or a dedicated network, then convert the signals into video data for display. Simultaneously, IP KVM can convert keyboard and mouse commands from the remote control terminal into control signals and transmit them to the remote server, enabling the local client to control multiple remote servers.

[0003] Currently, in the traditional KVM industry, the process of remotely controlling a remote server requires administrators to continuously monitor the video feed of the remote server's interface and issue commands to different interfaces. Over time, administrator fatigue can occur, leading to decreased efficiency in real-time control of the remote server. Therefore, a method is needed to improve the efficiency of remote server control. Summary of the Invention

[0004] This application provides a remote automatic control system, method, and electronic device based on IP KVM, which improves the efficiency of remote control of remote servers.

[0005] The first aspect of this application provides a remote automatic control system based on IP KVM, the system comprising a local client, an IP KVM, and a server, wherein:

[0006] The local client communicates with the IP KVM;

[0007] The IP KVM communicates with the server.

[0008] The IP KVM also communicates with multiple remote servers;

[0009] The remote server is used to send first running data to the local client in real time via the IP KVM, wherein the first running data is the running data of the remote server;

[0010] The local client is used to send the first running data to the server via the IP KVM;

[0011] The server includes an acquisition module, an identification module, a processing module, and an output module, wherein:

[0012] The acquisition module is used to acquire the first running data in real time;

[0013] The identification module identifies a first key feature contained in the first running data;

[0014] The processing module is configured to determine whether the first key feature and the second key feature are the same, wherein the second key feature is a key feature pre-marked from second runtime data from a remote server; and...

[0015] If the first key feature is the same as the second key feature, then retrieve the pre-stored operation instructions for the second key feature;

[0016] The output module is used to convert the operation instructions into keyboard and mouse control signals and send the keyboard and mouse control signals to the remote server.

[0017] By adopting the above technical solution, a local client can communicate with multiple remote servers via IP KVM, enabling control of these servers. During the process of the local client controlling the remote server via IP KVM, the remote server sends initial operational data to the local client in real time, and the local client simultaneously sends the same data back to the server. The server identifies a first key feature of the initial operational data and compares it with a second key feature. If they match, it indicates that the server has pre-stored operation instructions corresponding to the first key feature. The server retrieves the pre-stored operation instructions corresponding to the second key feature, converts them into keyboard and mouse control signals, and sends them to the remote server. Finally, the remote server converts the keyboard and mouse control signals back into operation instructions to execute the corresponding operations, thus completing the server's automatic control of the remote server. Since all the above operations are performed automatically by the computer, the amount of manual operation is greatly reduced, thereby improving the efficiency of remote server control.

[0018] A second aspect of this application provides a remote automatic control method based on IP KVM, the method being applied to a server of the aforementioned system, the method comprising:

[0019] The first running data is acquired in real time, and the first running data is the running data of the remote server;

[0020] Identify the first key feature contained in the first running data;

[0021] Determine whether the first key feature and the second key feature are the same, wherein the second key feature is a key feature pre-marked from the second running data of the remote server;

[0022] If the first key feature is the same as the second key feature, then retrieve the pre-stored operation instructions for the second key feature;

[0023] The operation instructions are converted into keyboard and mouse control signals, and the keyboard and mouse control signals are sent to the remote server.

[0024] By employing the above technical solution, the server identifies a first key feature of the first running data and compares it with a second key feature. If they match, it indicates that the server has pre-stored operation instructions for the first key feature. The server retrieves the pre-stored operation instructions corresponding to the second key feature and converts them into keyboard and mouse control signals, sending them to the remote server. Finally, the remote server converts the keyboard and mouse control signals back into operation instructions to execute the corresponding operation, thereby completing the automatic control of the remote server. Since all the above operations are completed automatically by the computer, the amount of manual operation is greatly reduced, thus improving the efficiency of remote control of the remote server.

[0025] Optionally, before acquiring the first running data in real time, the method further includes:

[0026] Obtain the second running data, and obtain the second key feature marked in the second running data;

[0027] Obtain the operation instruction and establish a mapping relationship between the operation instruction and the second key feature.

[0028] By adopting the above technical solution, the server can pre-acquire the second operating data and the second key features marked in the second operating data, and establish a mapping relationship between the operation instructions and the second key features, so that the server can send operation instructions to the remote server according to the key features and complete the automatic control of the remote server.

[0029] Optionally, after determining whether the first key feature and the second key feature are the same, the method further includes:

[0030] If it is determined that the first key feature is different from the second key feature, then the similarity between the first key feature and multiple second key features is calculated.

[0031] Determine whether the similarity value exceeds a preset first threshold;

[0032] If none of the multiple similarity values ​​exceed the first threshold, then the operation instruction sent by the local client for the first key feature is received.

[0033] The number of times the operation command is received is counted, and it is calculated whether the number of times the operation command is received exceeds a preset second threshold.

[0034] If the number of times the operation command is received exceeds a preset second threshold, a mapping relationship between the first key feature and the operation command is established.

[0035] By employing the above technical solution, when the server determines that the first key feature and the second key feature are different, it indicates that the server has not pre-stored any operation instructions for the first key feature. If the similarity between the first and second key features calculated by the server does not exceed a first threshold, it indicates that the server has not pre-stored any second running data similar to the first running data. The server counts the number of times the same operation instruction for the first running data is subsequently sent by the local client. If the number exceeds a preset second threshold, it indicates that the local client has repeatedly used the same operation instruction for the first key feature. The server then establishes a mapping relationship between the first key feature and the operation instructions, thereby completing the learning of the operation instructions for the first key feature, facilitating the automatic sending of the corresponding operation instructions when the server subsequently recognizes the first key feature.

[0036] Optionally, after determining whether the similarity value exceeds a preset first threshold, the method further includes:

[0037] If it is determined that the values ​​of multiple similarities all exceed the first threshold, then the multiple similarities are sorted according to their numerical values, the first similarity with the largest value is selected, and the third key feature corresponding to the first similarity is retrieved.

[0038] Retrieve pre-stored operation instructions targeting the third key feature;

[0039] The operation instructions are converted into keyboard and mouse control signals, and the keyboard and mouse control signals are sent to the remote server.

[0040] By adopting the above technical solution, if the server determines that multiple similarity values ​​all exceed the first threshold, it indicates that there are multiple second key features similar to the first key feature. The server selects the first similarity value that is higher than the other similarities, and retrieves the third key feature corresponding to the first similarity value and the operation instructions for the third key feature. Thus, the server completes the selection of operation instructions for the first key feature through fuzzy matching, thereby completing the automatic control of the remote server.

[0041] Optionally, after sending the keyboard and mouse control signal to the remote server, the method further includes:

[0042] Obtain the execution result after the remote server executes the operation command;

[0043] The execution results are converted into a video signal and sent to the local client.

[0044] Receive the evaluation result of the operation result sent by the local client, and determine whether the evaluation result is correct or incorrect;

[0045] If the evaluation result is determined to be correct, then a mapping relationship between the first key feature and the operation instruction is established.

[0046] By employing the above technical solution, after the server selects the operation command based on the first key feature through fuzzy matching, it sends keyboard and mouse control signals to the remote server. After executing the operation command, the remote server sends a video signal converted from the execution result to the local client. If the local client determines that the operation command is correct, indicating that the operation command selected based on the first key feature is correct, the server establishes a mapping relationship between the first key feature and the operation command. This facilitates the server directly selecting the corresponding operation command when it subsequently identifies the first key feature again.

[0047] Optionally, after sending the keyboard and mouse control signal to the remote server, the method further includes:

[0048] Real-time detection of any anomalies in data downloaded from the remote server;

[0049] If an anomaly is detected in the data, the data is sent to the local client's isolation zone and deleted from the remote server.

[0050] By adopting the above technical solution, the server monitors the data downloaded from the remote server in real time for anomalies and takes security measures for abnormal data, thus protecting the security of the remote server's data and preventing abnormal data from affecting the operation of the server's automated operation commands. Abnormal data is sent to the local client's isolation zone, facilitating subsequent understanding and analysis of the anomaly by the local client.

[0051] Optionally, the type of the first running data may also include:

[0052] Video data, text data, image data, and code data;

[0053] The key features corresponding to the first running data of the video data type include combinations of key characters and key pixels.

[0054] The key features corresponding to the first running data of the text data type include key characters;

[0055] The key features corresponding to the first running data of the image data type include key pixel combinations;

[0056] The key features corresponding to the first running data of the code data type include key feature codes.

[0057] By adopting the above technical solutions, the server can process various types of operational data and identify the key features contained in different types of operational data, thereby increasing the application scenarios of automatic control methods.

[0058] A third aspect of this application provides an electronic device including a processor, a memory, a user interface, and a network interface, wherein the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any of the foregoing.

[0059] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0060] 1. A local client can communicate with multiple remote servers via IP KVM, enabling control of these servers. During the process of controlling the remote servers via IP KVM, the remote servers send initial operational data to the local client in real time, and the local client simultaneously sends the same data back to the server. The server identifies a first key feature of the initial operational data and compares it with a second key feature. If they match, it indicates that the server has pre-stored operation instructions corresponding to the first key feature. The server retrieves the pre-stored operation instructions corresponding to the second key feature, converts them into keyboard and mouse control signals, and sends them to the remote server. Finally, the remote server converts the keyboard and mouse control signals back into operation instructions to execute the corresponding operations, thus completing the server's automatic control of the remote servers. Because all of the above operations are performed automatically by the computer, the amount of manual operation is greatly reduced, thereby improving the efficiency of remote server control.

[0061] 2. Without pre-stored operation instructions for the first key feature, the server can automatically count the number of times the local client sends the same operation instruction for the first running data. If the number exceeds a preset threshold, it indicates that the local client has repeatedly used the same operation instruction for the first key feature. The server then establishes a mapping relationship between the first key feature and the operation instructions, thereby learning the operation instructions for the first key feature. This facilitates the server automatically sending the corresponding operation instruction when it subsequently recognizes the first key feature.

[0062] 3. If the server determines that multiple similarity values ​​exceed the first threshold, it indicates the existence of multiple second key features similar to the first key feature. The server selects the first similarity value that is higher than the others, and retrieves the third key feature corresponding to the first similarity value and the operation instructions for the third key feature. Thus, through fuzzy matching, the server selects the operation instructions for the first key feature, thereby completing the automatic control of the remote server. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the structure of a remote automatic control system based on IP KVM disclosed in an embodiment of this application.

[0064] Figure 2 This is a schematic diagram of the structure of a server disclosed in an embodiment of this application.

[0065] Figure 3 This is a flowchart illustrating a remote automatic control method based on IP KVM disclosed in an embodiment of this application.

[0066] Figure 4 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application.

[0067] Explanation of reference numerals in the attached diagram: 1. Local client; 2. IP KVM; 3. Server; 4. Acquisition module; 5. Identification module; 6. Processing module; 7. Output module; 8. Processor; 9. Communication bus; 10. User interface; 11. Network interface; 12. Memory. Detailed Implementation

[0068] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0069] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0070] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0071] This embodiment discloses a remote automatic control system based on IP KVM, referring to... Figure 1 The system includes a local client 1, an IP KVM 2, and a server 3, wherein:

[0072] Local client 1 communicates with IP KVM2. IP KVM2 communicates with server 3. IP KVM2 also communicates with multiple remote servers. The remote servers are used to send initial runtime data to local client 1 in real time via IP KVM2; this initial runtime data is the runtime data from the remote servers. Local client 1 is used to send this initial runtime data to server 3 via IP KVM2. (See reference...) Figure 2 Server 3 includes an acquisition module 4, an identification module 5, a processing module 6, and an output module 7, wherein: the acquisition module 4 is used to acquire first running data in real time; the identification module 5 identifies a first key feature contained in the first running data; the processing module 6 is used to determine whether the first key feature is the same as a second key feature, the second key feature being a key feature pre-marked from the second running data of the remote server; and, if the first key feature is the same as the second key feature, then a pre-stored operation instruction for the second key feature is retrieved; and the output module 7 is used to convert the operation instruction into keyboard and mouse control signals and send the keyboard and mouse control signals to the remote server.

[0073] Specifically, in the traditional "client-KVM-server" architecture, the client connects directly to multiple servers via KVM. This architecture limits the client to receiving user input commands and sending keyboard and mouse control signals to the server to control it. This application, building upon the existing architecture, connects server 3 to IP KVM2, enabling a remote automatic control method based on IP KVM. Server 3 can perform automated control of remote servers, thus improving the efficiency of remote server control. Local client 1 communicates with IP KVM2, which in turn communicates with multiple remote servers. Simultaneously, IP KVM2 communicates with server 3. Local client 1 is connected to a monitor, keyboard, and mouse. During remote server operation, the video data of the current interface is converted into a video signal and transmitted via IP KVM2 to the monitor of local client 1 for display. The user on local client 1 issues operation commands via keyboard and mouse based on the video content. These commands are then converted into keyboard and mouse control signals by IP KVM2 and transmitted to the remote server, thus controlling it. Furthermore, since the remote server and IPKVM2, IPKVM2 and local client 1, and IPKVM2 and server 3 all communicate via the Internet, various file types such as text, images, and videos can be transferred between each other via the Internet, and the transmission distance will no longer be limited.

[0074] This embodiment also discloses a remote automatic control method based on IP KVM, referring to... Figure 2 It includes the following steps:

[0075] S110, real-time acquisition of the first running data, which is the running data of the remote server.

[0076] Specifically, during the process of local client 1 controlling the operation of the remote server through IP KVM2, the remote server sends the first running data to local client 1 in real time, and local client 1 sends the first running data to server 3.

[0077] S120, identify the first key feature contained in the first running data.

[0078] Specifically, the first key feature is determined based on the type of the first running data. For example, if the first running data is video-type running data, then the first key feature is a combination of key characters and key pixels.

[0079] S130, determine whether the first key feature and the second key feature are the same, the second key feature is a key feature that is marked in advance from the second running data of the remote server.

[0080] Specifically, server 3 identifies a first key feature contained in the first running data. Then, server 3 retrieves multiple pre-stored second running data and obtains the second key feature marked in the second running data. Server 3 then compares the first key feature with the second key feature to determine whether they are the same.

[0081] S140, if the first key feature is the same as the second key feature, then retrieve the pre-stored operation instructions for the second key feature.

[0082] Specifically, if server 3 determines that the first key feature and the second key feature are the same, it indicates that server 3 has pre-stored operation instructions for the first key feature. Then server 3 retrieves the pre-stored operation instructions corresponding to the second key feature.

[0083] S150 converts operation commands into keyboard and mouse control signals and sends the keyboard and mouse control signals to the remote server.

[0084] Specifically, server 3 then converts the operation commands into keyboard and mouse control signals and sends these signals to the remote server. Finally, the remote server converts the keyboard and mouse control signals back into operation commands to execute the corresponding operations, thus completing server 3's automatic control of the remote server. Since all of the above operations are performed automatically by the computer, the amount of manual operation is greatly reduced, thereby improving the efficiency of remote control of the remote server.

[0085] For example, when the remote server is processing temporary files, the second running data is the pre-stored video data displayed on the local client 1's monitor. In this second running data, if the video displays a file named "XXX-Expired File.docx", the corresponding key characters marked in the second key feature are "expired file" and "docx". For this second key feature, the corresponding operation instruction is "Delete the file containing the key characters 'expired file' and 'docx'". When the local client 1 remotely controls the remote server to process files in real time, the remote server sends the video signal of the file processing interface to the local client 1 in real time. Simultaneously, the server 3 acquires the video displayed on the remote client's monitor, i.e., acquires the first running data. If the server 3 identifies a file named "Transmission Record-Expired File.docx" in a certain video frame, the server 3 automatically sends an operation instruction to the remote server to "Delete this file".

[0086] In one possible implementation, before acquiring the first operational data in real time, the method further includes: acquiring second operational data and acquiring a second key feature marked in the second operational data; acquiring operation instructions and establishing a mapping relationship between the operation instructions and the second key feature.

[0087] Specifically, in order to facilitate the subsequent sending of operation instructions to the remote server based on the key features and to complete the automatic control of the remote server, it is necessary to obtain the second key feature and the operation instructions corresponding to the second key feature in advance.

[0088] For example, when local client 1 controls a remote server via IP KVM2, the remote server transmits video data from its interface to local client 1's monitor via IP KVM2 for display. Simultaneously, server 3 acquires the video from the displayed interface in real time. Local client 1 sends keyboard and mouse operation signals to the remote server via IP KVM2, achieving remote control of the server's operation. During operation, the user selects video frames for which automation is desired via local client 1 and marks key characters on the frames. Server 3 automatically sets these key characters as secondary key features. Then, operation commands are recorded, with local client 1 continuously inputting the required commands to server 3 until completion. Server 3 then automatically establishes a mapping between the operation commands and the secondary key features.

[0089] In one possible implementation, after determining whether the first key feature and the second key feature are the same, the method further includes: if the first key feature and the second key feature are determined to be different, then calculating the similarity between the first key feature and multiple second key features. Determining whether the similarity value exceeds a preset first threshold. If none of the multiple similarity values ​​exceed the first threshold, then receiving an operation instruction for the first key feature sent by the local client 1. Counting the number of times the operation instruction is received; if the number of times the operation instruction is received exceeds a preset second threshold, then establishing a mapping relationship between the first key feature and the operation instruction.

[0090] Specifically, if server 3 determines that there is no second key feature identical to the first key feature, then during the process of local client 1 controlling the remote server operation via IP KVM2, server 3 acquires the first key feature of the first running data in real time, calculates the similarity between the first key feature and the second key feature, and obtains the similarity score. The similarity calculation is a conventional technique in the relevant technical field and will not be elaborated further here. Server 3 then determines whether the similarity score exceeds a preset first threshold. The specific value of the first threshold is not specifically limited in this embodiment, but can be adjusted in other embodiments according to actual conditions.

[0091] If none of the multiple similarity values ​​exceed the first threshold, meaning no second key feature similar to the first key feature is pre-stored, then server 3 receives the operation command for the first running data sent by local client 1. Subsequently, if server 3 repeatedly identifies the first key feature and repeatedly receives the same operation command for the first running data sent by local client 1, server 3 calculates the number of received operation commands for the first key feature. If the calculated number of received operation commands exceeds a preset second threshold, it indicates that local client 1 has sent the same operation command for the first key feature multiple times. The specific value of the second threshold is not specifically limited in this embodiment; other embodiments can adjust it according to actual conditions. Server 3 establishes a mapping relationship between the second key feature and the operation command, thereby completing the learning of the operation command for the first running data. For the similarity calculation between the first and second key features, different algorithms can be used to calculate the similarity based on different types of key features. For example, character-type key features can be similar based on a character matching algorithm.

[0092] In one possible implementation, after determining whether the similarity value exceeds a preset first threshold, the method further includes: if it is determined that multiple similarity values ​​all exceed the first threshold, sorting the multiple similarities by value, selecting the first similarity with the largest value, and retrieving the third key feature corresponding to the first similarity. Then, retrieving pre-stored operation instructions for the third key feature, converting the operation instructions into keyboard and mouse control signals, and sending the keyboard and mouse control signals to a remote server.

[0093] Specifically, if server 3 determines that multiple similarity values ​​all exceed the first threshold, it indicates the existence of multiple second key features similar to the first key feature. Then, server 3 selects the first similarity value that is higher than the others, and retrieves the third key feature corresponding to the first similarity value and the operation instructions for the third key feature. In this way, the operation instructions for the first key feature are selected through fuzzy matching, thereby completing the automatic control of the remote server.

[0094] In one possible implementation, after sending the keyboard and mouse control signals to the remote server, the process further includes: obtaining the execution result of the operation command executed by the remote server; converting the execution result into a video signal and sending it to the local client 1; receiving the evaluation result of the execution result sent by the local client 1, and determining whether the evaluation result is correct or incorrect; if the evaluation result is correct, then establishing a mapping relationship between the first key feature and the operation command.

[0095] Specifically, after selecting the operation command based on the first key feature through fuzzy matching, server 3 sends keyboard and mouse control signals to the remote server. Server 3 needs to obtain feedback from local client 1 to determine if the matching result is correct. After executing the operation command, the remote server sends the running result to local client 1, that is, it sends the video of the running interface after executing the operation command to local client 1. If local client 1 judges the operation command to be correct, it indicates that the operation command selected by server 3 for the first key feature is the correct operation command. Then, server 3 establishes a mapping relationship between the first key feature and the operation command, so that server 3 can directly select the operation command corresponding to the first key feature when the first key feature reappears on the remote server.

[0096] For example, when the remote server inputs text, the first running data is the video data of the remote server's current running interface displayed on the local client 1's monitor. If the user enters the text "The current engine needs to be disassembled" on the remote server using the keyboard of the local client 1, the server 3 recognizes the first key feature "The current engine needs to be disassembled," and then the server 3 retrieves the pre-stored second running data and the second key feature of the second running data. When the remote server inputs text, the second running data is the pre-stored video data displayed on the local client 1's monitor.

[0097] If server 3 finds that the database does not pre-store a second key feature identical to the first key feature, it calculates the similarity between the first key feature and multiple second key features. If the third key feature with the highest similarity is the input "Need to disassemble the current engine," and the corresponding operation instruction is to enter "The following steps are included," server 3 converts the operation instruction into keyboard and mouse control signals and sends them to the remote server. The remote server enters "The following steps are included" after "Need to disassemble the current engine" and converts the execution result into a video signal and sends it to local client 1. If the user determines that "The following steps are included" should be entered after "Need to disassemble the current engine," the local client 1 sends a "correct" judgment result to server 3. If the user determines that other content should be entered after "Need to disassemble the current engine," the local client 1 sends an "incorrect" judgment result to server 3. If server 3 receives a "correct" judgment result, it establishes a mapping relationship between the first key feature "Need to disassemble the current engine" and the operation instruction "The following steps are included."

[0098] In one possible implementation, after sending keyboard and mouse control signals to the remote server, the process further includes: real-time detection of any anomalies in the data downloaded by the remote server. If anomalies are detected, the data is sent to the isolation zone of local client 1, and the data is deleted from the remote server.

[0099] Specifically, to prevent abnormal data from the remote server from affecting the execution of automated operation instructions on server 3, server 3 will monitor in real time whether the data downloaded from the remote server is abnormal. If abnormal data is detected, server 3 will send the data to the isolation zone of local client 1 and delete the data from the remote server, so that local client 1 can understand and analyze the abnormal situation later.

[0100] In one possible implementation, the types of the first running data may also include: video data, text data, image data, and code data.

[0101] Specifically, the key features corresponding to the first running data of video data type include key characters and key pixel combinations. The key features corresponding to the first running data of text data type include key characters. The key features corresponding to the first running data of image data type include key pixel combinations. The key features corresponding to the first running data of code data type include key feature codes. In this embodiment, the extraction and recognition of the key features corresponding to the first running data of video data type are preferably performed using a text detection algorithm. In this embodiment, the extraction and recognition of the key features corresponding to the first running data of text data type are preferably performed using an image recognition algorithm. In this embodiment, the extraction and recognition of the key features corresponding to the first running data of code data type are preferably performed using a character matching algorithm. The techniques used for the extraction and recognition of the above-mentioned key features are merely conventional techniques in the relevant technical field and will not be further elaborated here.

[0102] In one possible implementation, server 3 is used to calculate the similarity between the first key feature and multiple second key features if it is determined that the first key feature is different from the second key feature.

[0103] Determine whether the similarity value exceeds a preset first threshold;

[0104] If none of the multiple similarity values ​​exceed the first threshold, then the operation instruction for the first key feature sent by the local client 1 is received.

[0105] The number of times an operation command is received is counted. If the number of times an operation command is received exceeds a preset second threshold, a mapping relationship between the first key feature and the operation command is established.

[0106] In one possible implementation, server 3 is used to sort the multiple similarities by numerical value if it is determined that the values ​​of multiple similarities all exceed the first threshold, select the first similarity with the largest value, and retrieve the third key feature corresponding to the first similarity.

[0107] Retrieve pre-stored operation instructions targeting the third key feature;

[0108] The operation commands are converted into keyboard and mouse control signals, and then the keyboard and mouse control signals are sent to the remote server.

[0109] In one possible implementation, server 3 is used to obtain the running results after the remote server executes the operation instructions;

[0110] The execution results are converted into a video signal and sent to local client 1;

[0111] Receive the evaluation result of the running result sent by local client 1, and determine whether the evaluation result is correct or incorrect;

[0112] If the evaluation result is correct, then a mapping relationship between the first key feature and the operation command is established.

[0113] In one possible implementation, server 3 is used to detect in real time whether the data downloaded from the remote server is abnormal;

[0114] If an anomaly is detected in the data, the data is sent to the isolation zone of local client 1 and deleted from the remote server.

[0115] In one possible implementation, server 3 is used to acquire video data, text data, image data, and code data;

[0116] The key features corresponding to the first running data of video data types include combinations of key characters and key pixels;

[0117] The key features of the first running data of the text data type include key characters;

[0118] The key features corresponding to the first running data of the image data type include key pixel combinations;

[0119] The key features corresponding to the first running data of the code data type include the key feature code.

[0120] This application also provides an electronic device, with reference to... Figure 4 The electronic device may include: at least one processor 8, at least one communication bus 9, user interface 10, network interface 11, and at least one memory 12.

[0121] Among them, the communication bus 9 is used to realize the connection and communication between these components.

[0122] The user interface 10 may include a display screen and a camera. Optionally, the user interface 10 may also include a standard wired interface and a wireless interface.

[0123] The network interface 11 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0124] The processor 8 may include one or more processing cores. The processor 8 connects to various parts of the server 3 via various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 12, and by calling data stored in the memory 12. Optionally, the processor 8 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 8 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 8.

[0125] The memory 12 may include random access memory (RAM) or read-only memory. Optionally, the memory 12 may include a non-transitory computer-readable storage medium. The memory 12 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 12 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. Optionally, the memory 12 may also be at least one storage device located remotely from the aforementioned processor 8. As shown in the figure, the memory 12, as a computer storage medium, may include an operating system, a network communication module, a user interface 10 module, and an application program for a remote automatic control method based on IP KVM.

[0126] In the electronic device shown in the figure, the user interface 10 is mainly used to provide an input interface for the user and to obtain the user input data; while the processor 8 can be used to call an application program stored in the memory 12 for a remote automatic control method based on IP KVM. When executed by one or more processors 8, the electronic device performs one or more methods as described in the above embodiments.

[0127] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0128] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some service interfaces; indirect couplings or communication connections between apparatuses or units may be electrical or other forms.

[0130] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0132] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device 12. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage device 12 and includes several instructions to cause a computer device (which may be a personal computer, server 3, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage device 12 includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a magnetic disk, or an optical disk.

[0133] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of other embodiments of this disclosure upon considering the specification and the disclosure of practical truth. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A remote automatic control system based on IP KVM, characterized in that, The system includes a local client (1), an IP KVM (2), and a server (3), wherein: The local client (1) is connected to the IP KVM (2) in a communication connection; The IP KVM (2) is connected to the server (3) in communication. The IP KVM (2) also communicates with multiple remote servers; The remote server is used to send first running data to the local client (1) in real time through the IP KVM (2), wherein the first running data is the running data of the remote server; The local client (1) is used to send the first running data to the server (3) through the IP KVM (2); The server (3) includes an acquisition module (4), an identification module (5), a processing module (6), and an output module (7), wherein: The acquisition module (4) is used to acquire the first running data in real time. Before acquiring the first running data in real time, in the case that the second running data is a video frame, key characters are marked on the video frame, the key characters are used as the second key feature, and the recorded operation instructions are acquired. The operation instructions are instructions that are continuously input to the server (3). The mapping relationship between the second key feature and the operation instructions is established so as to enable automatic control of the remote server. The identification module (5) identifies the first key feature contained in the first running data; The processing module (6) is used to determine whether the first key feature and the second key feature are the same, wherein the second key feature is a key feature pre-marked from the second running data of the remote server; and, If the first key feature is the same as the second key feature, then retrieve the pre-stored operation instructions for the second key feature; And, if it is determined that the first key feature is different from the second key feature, then the similarity between the first key feature and other second key features is calculated; it is determined whether the values ​​of multiple similarities exceed a preset first threshold; if the values ​​of multiple similarities do not exceed the first threshold, then the operation instruction for the first key feature sent by the local client (1) is received; the number of times the operation instruction is received is counted, and it is calculated whether the number of times the operation instruction is received exceeds a preset second threshold; if it is calculated that the number of times the operation instruction is received exceeds the preset second threshold, then a mapping relationship between the first key feature and the operation instruction is established to realize the learning of the operation instruction corresponding to the first key feature; The output module (7) is used to convert the operation instructions into keyboard and mouse control signals and send the keyboard and mouse control signals to the remote server.

2. A remote automatic control method based on IP KVM, characterized in that, The method is applied to the server (3) as described in claim 1, and the method includes: The first running data is obtained in real time. The first running data is the running data of the remote server. Before obtaining the first running data in real time, in the case that the second running data is a video frame, key characters are marked on the video frame. The key characters are used as the second key feature. Recorded operation instructions are obtained. The operation instructions are instructions that are continuously input to the server (3). The mapping relationship between the second key feature and the operation instructions is established so as to enable automatic control of the remote server. Identify the first key feature contained in the first running data; Determine whether the first key feature and the second key feature are the same, wherein the second key feature is a key feature pre-marked from the second running data of the remote server; If the first key feature is the same as the second key feature, then retrieve the pre-stored operation instructions for the second key feature; If it is determined that the first key feature is different from the second key feature, then the similarity between the first key feature and other second key features is calculated. Determine whether the values ​​of multiple similarities exceed a preset first threshold; If none of the multiple similarity values ​​exceed the first threshold, then the operation instruction for the first key feature sent by the local client (1) is received; The number of times the operation command is received is counted, and it is calculated whether the number of times the operation command is received exceeds a preset second threshold. If the number of times the operation instruction is received exceeds a preset second threshold, a mapping relationship between the first key feature and the operation instruction is established to achieve the learning of the operation instruction corresponding to the first key feature. The operation instructions are converted into keyboard and mouse control signals, and the keyboard and mouse control signals are sent to the remote server.

3. The remote automatic control method based on IP KVM according to claim 2, characterized in that, After determining whether the similarity value exceeds a preset first threshold, the method further includes: If it is determined that the values ​​of multiple similarities all exceed the first threshold, then the multiple similarities are sorted according to their numerical values, the first similarity with the largest value is selected, and the third key feature corresponding to the first similarity is retrieved. Retrieve pre-stored operation instructions targeting the third key feature; The operation instructions are converted into keyboard and mouse control signals, and the keyboard and mouse control signals are sent to the remote server.

4. The remote automatic control method based on IP KVM according to claim 3, characterized in that, After sending the keyboard and mouse control signals to the remote server, the method further includes: Obtain the execution result after the remote server executes the operation command; The running results are converted into video signals and sent to the local client (1); Receive the evaluation result of the operation result sent by the local client (1), and determine whether the evaluation result is correct or incorrect; If the evaluation result is determined to be correct, then a mapping relationship between the first key feature and the operation instruction is established.

5. The remote automatic control method based on IP KVM according to claim 2, characterized in that, After sending the keyboard and mouse control signals to the remote server, the method further includes: Real-time detection of any anomalies in data downloaded from the remote server; If an anomaly is detected in the data, the data is sent to the isolation zone of the local client (1) and the data is deleted from the remote server.

6. The remote automatic control method based on IP KVM according to claim 2, characterized in that, The types of the first running data also include: Video data, text data, image data, and code data; The key features corresponding to the first running data of the video data type include combinations of key characters and key pixels. The key features corresponding to the first running data of the text data type include key characters; The key features corresponding to the first running data of the image data type include key pixel combinations; The key features corresponding to the first running data of the code data type include key feature codes.

7. An electronic device, characterized in that, The device includes a processor (8), a memory (12), a user interface (10), and a network interface (11). The memory (12) is used to store instructions. The user interface (10) and the network interface (11) are both used to communicate with other devices. The processor (8) is used to execute the instructions stored in the memory (12) to cause the electronic device to perform the method as described in any one of claims 2-6.

Images