Control device, device control method, and electronic device

Abstract

The invention provides a control device, a device control method and an electronic device. The control device comprises a USB controller, an information acquisition module and a decision module. The USB controller is used for connecting the control equipment and the controlled equipment; the information acquisition module is used for acquiring output information of controlled equipment and sending the output information to the decision module; the decision module is used for analyzing the output information, generating a control instruction and sending the control instruction to the USB controller; the USB controller is used for sending the control instruction to the controlled device, and the control instruction is used for controlling the controlled device. Therefore, automatic control of the controlled equipment can be realized on a hardware level through a hardware control mode, and compared with a mode of controlling the controlled equipment based on a remote control protocol of software, the mode is not limited by a software system state or a network environment.

Description

Technical Field

[0001] This application relates to the field of equipment control, and more particularly to a control device, equipment control method, and electronic device. Background Technology

[0002] In scenarios involving multiple personal computers (PCs) or multiple servers, remote control is a common management method. In related technologies, remote control of these devices is typically based on software-based remote control protocols. Examples include the Remote Desktop Protocol (RDP) and the Secure Shell (SSH) command-line interface remote management protocol. The client program of the software can be installed on the controlling device, and the server program can be installed on the controlled device.

[0003] However, in practical applications, these remote communication protocols are easily limited by the device's software system status or network environment, making it impossible to remotely control the device. For example, if the software in the controlled device malfunctions, the control terminal will be unable to connect to the server-side application of the controlled device, thus preventing remote control. Similarly, if the controlled device is restricted by a firewall, the control terminal will also be unable to connect to the server-side application of the controlled device, resulting in the inability to remotely control the device. Summary of the Invention

[0004] This application provides a control device, a device control method, and an electronic device to solve the problem that current solutions for controlling devices via remote control protocols are easily limited by the device's network environment or software environment, resulting in the inability to remotely control the device.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows:

[0006] In a first aspect, a control device is provided, comprising a USB controller, an information acquisition module, a decision-making module, and a processing module, wherein:

[0007] The USB controller is used to connect the control device and the controlled device;

[0008] The information acquisition module is used to acquire the output information of the controlled device and send the output information to the decision module;

[0009] The decision module is used to analyze the output information and generate control commands, and send the control commands to the USB controller;

[0010] The USB controller is used to send the control commands to the controlled device, and the control commands are used to control the controlled device.

[0011] Secondly, a device control method is provided, applied to a control device, wherein the control device is connected to a controlled device via a USB controller, and the control method includes:

[0012] Obtain the output information of the controlled device;

[0013] The output information is analyzed and control commands are generated;

[0014] The control command is sent to the controlled device, and the control command is used to control the controlled device.

[0015] Thirdly, an electronic device is provided, comprising:

[0016] processor;

[0017] Memory used to store the processor's executable instructions;

[0018] The processor is configured to execute the instructions to implement the method as described in the second aspect.

[0019] In this embodiment, when controlling the controlled device, the control device can connect to the controlled device via a USB controller, acquire the output information of the controlled device, analyze the output information, generate control commands for controlling the controlled device, and send the control commands to the controlled device via the USB controller. Thus, automatic control of the controlled device can be achieved at the hardware level through hardware control, which, compared to software-based remote control protocols, is not limited by software system status or network environment. Furthermore, since the control commands are generated by analyzing the output information of the controlled device, the control commands are more closely matched to the current state of the controlled device. Controlling the controlled device through these commands better meets the control requirements, thereby achieving accurate control. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1This is a schematic diagram of the structure of a control device according to an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of a screen display screen according to an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of a screen display screen according to another embodiment of this application;

[0024] Figure 4 This is a schematic diagram of a screen display screen according to another embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the structure of a control device according to another embodiment of this application;

[0026] Figure 6 This is a schematic diagram of the structure of the decision module deployed on the control device side in one embodiment of this application;

[0027] Figure 7 This is a schematic diagram of the structure of the decision module deployed on the control device side in another embodiment of this application;

[0028] Figure 8 This is a schematic diagram illustrating the deployment of a decision module in the cloud, as shown in one embodiment of this application.

[0029] Figure 9 This is a schematic diagram illustrating the control of multiple PCs according to an embodiment of this application;

[0030] Figure 10 This is a schematic diagram of a control device designed inside a PC according to an embodiment of this application;

[0031] Figure 11 This is a schematic diagram illustrating the connection between a control device and a PC according to an embodiment of this application;

[0032] Figure 12 This is a schematic flowchart of a device control method according to an embodiment of this application;

[0033] Figure 13 This is a schematic flowchart of another embodiment of the device control method of this application;

[0034] Figure 14 This is a schematic flowchart of another embodiment of the device control method of this application;

[0035] Figure 15 This is a schematic flowchart of another embodiment of the device control method of this application;

[0036] Figure 16 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0037] In related technologies, remote control of controlled devices (such as PCs) can be achieved based on software-based remote control protocols. However, this control method has certain limitations in practical applications. First, remote control protocols lack universality, requiring the installation of a client program on the controlling device and a corresponding server program on the controlled device to enable remote control. Second, current remote control protocols are easily limited by the device's software system state. For example, the software needs to be operating normally to achieve remote control; if the controlled device enters an abnormal state, such as a crash, the controlling device will be unable to connect to the server application on the controlled device, thus preventing remote control. Finally, current remote control protocols are easily limited by the controlled device's network environment. For instance, if the controlling device is blocked by a firewall, it will be unable to connect to the server application on the controlled device, thus preventing remote control.

[0038] It is evident that current solutions for controlling devices via remote control protocols are easily limited by the device's network or software environment, leading to the inability to remotely control the device.

[0039] This application provides a control device, a device control method, and an electronic device. When controlling a controlled device, the control device can connect to the controlled device via a USB controller, acquire the output information of the controlled device, analyze the output information, generate control commands for controlling the controlled device, and send the control commands to the controlled device via the USB controller. Therefore, automatic control of the controlled device can be achieved at the hardware level through hardware control, which is not limited by software system status or network environment compared to software-based remote control protocols. Furthermore, since the control commands are generated by analyzing the output information of the controlled device, the control commands are more closely matched to the current state of the controlled device. Controlling the controlled device through these commands better meets the control requirements, thereby achieving accurate control of the controlled device.

[0040] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in this application will be clearly and completely described below with reference to the accompanying drawings of one or more embodiments. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this application.

[0041] The terms "first," "second," etc., used in this application and the claims are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that this application can be implemented in orders other than those illustrated or described herein. Furthermore, in this application and the claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0042] It should be noted that the controlled device in each embodiment of this application can be a device that needs to be controlled in a PC or other device control scenario, and no specific limitation is made here.

[0043] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0044] Figure 1 This is a schematic diagram of the structure of a control device 10 according to an embodiment of this application. Figure 1 The control device 10 shown includes a USB controller 11, an information acquisition module 12, and a decision module 13. The USB controller 11 connects the control device 10 and the controlled device 20. The information acquisition module 12 acquires the output information of the controlled device 20 and sends it to the decision module 13. The decision module 12 analyzes the output information, generates control commands, and sends the control commands to the USB controller 11. The USB controller 11 sends the control commands to the controlled device 20, and the control commands are used to control the controlled device 20.

[0045] In this way, when controlling the controlled device, the controlling device connects to the controlled device via a USB controller. The information acquisition module in the controlling device acquires the output information of the controlled device, the decision module analyzes the output information and generates control commands for controlling the controlled device, and the USB controller sends the control commands to the controlled device. Therefore, automatic control of the controlled device can be achieved at the hardware level through hardware control, which is not limited by software system status or network environment compared to software-based remote control protocols. Furthermore, since the control commands are generated after analyzing the output information of the controlled device, the control commands are more closely matched to the current state of the controlled device. Controlling the controlled device through these commands better meets the control requirements, thus achieving accurate control.

[0046] In some implementations, the output information of the controlled device may include the screen display of the controlled device. Correspondingly, when the information acquisition module acquires the output information of the controlled device, it can acquire the screen display of the controlled device and send the screen display to the decision module. The decision module analyzes the screen display and generates control commands for controlling the controlled device.

[0047] In this way, since the screen display of the controlled device can usually intuitively reflect the current status of the controlled device, accurate control of the controlled device can be achieved by acquiring the screen display and analyzing the screen display to generate control commands.

[0048] It should be noted that in other possible implementations, the output information of the controlled device may also include other information, such as the logs of the controlled device, etc., which are not specifically limited here.

[0049] In some implementations, the decision-making module may include an information recognition module and a large language model. The information recognition module is used to recognize the output information of the controlled device. The large language model is used to analyze the recognition results from the information recognition module and generate control commands based on the analysis results. Optionally, when the output information of the controlled device includes screen displays, the information recognition module may specifically be an image recognition module. This image recognition module may include a text recognition module (such as an Optical Character Recognition (OCR) module), an icon recognition module, and an image understanding module, thereby improving the accuracy of the recognition results and better recognizing the output information of the controlled device. The large language model is a language model trained on a large-scale text corpus containing billions (or more) of parameters. It is a series of artificial intelligence models. By using the large language model to analyze the recognition results of the information recognition module, the accuracy of the analysis results can be improved, thereby generating more accurate control commands based on the analysis results and achieving accurate control of the controlled device.

[0050] In some implementations, the control device may further include a receiving module. The receiving module receives user instructions and sends them to a decision module. These user instructions are used to control the controlled device. The decision module generates control instructions based on the analysis of the controlled device's output information and the user instructions. In other words, when the control device includes a receiving module, the user can issue user instructions to control the controlled device through the receiving module. These user instructions can be voice instructions or text instructions, etc., without specific limitations. After receiving the user instructions, the receiving module can send them to the decision module. The decision module, when generating control instructions, can combine the user instructions with the analysis of the controlled device's output information to generate the control instructions. Thus, because control instructions for the controlled device can be generated by combining user instructions, not only can accurate control of the controlled device be achieved, but the user's control needs can also be met. For the user, this allows for control of the controlled device via voice or text.

[0051] It should be noted that in some potential application scenarios, if the control device includes a display screen, after the control device is connected to the controlled device via a USB controller, it can also display a prompt message on the screen. This prompt message indicates to the user whether to enter assistant mode. In assistant mode, the user can issue user commands to the control device to control the controlled device via voice or text. If the user confirms entering assistant mode, they can send a confirmation message to the control device (for example, a confirmation option can be provided below the prompt message, which the user can click to send confirmation). After receiving the confirmation message, the control device can enter assistant mode. At this time, the user can issue user commands to the control device via voice or text. After receiving the user command, the receiving module of the control device can send the user command to the decision module, which then combines the user command to generate control commands for the controlled device.

[0052] To facilitate understanding how the decision-making module generates control commands based on the output information of the controlled device, or based on the output information of the controlled device and user commands, the following will use output information as an example. Figures 2 to 4 The following is an example of a screen display. Taking a decision-making module that includes an image recognition module and a large language model, and an image recognition module that includes an OCR module, an icon recognition module, and an image understanding module as an example:

[0053] If the screen display received by the decision module is as follows Figure 2 As shown, the decision module may include the following steps when generating control commands:

[0054] Step 1: The image acquisition module will... Figure 2 The screen display shown is sent to the image understanding module.

[0055] Optionally, the image acquisition module sends data to the image understanding module. Figure 2 When displaying the screen as shown, the following information can be sent to the image understanding module:

[0056] {

[0057] "screenshot": "Screenshot in base64 encoding"

[0058] }

[0059] Step 2: The image understanding module interprets this as the device needing to press a special key, such as (F2), which typically occurs when the motherboard battery is low. At this time, the image understanding module can send the following data to the large language model:

[0060] {

[0061] "question": "This is a computer's BIOS setup interface. In this interface, the system displays the message 'CMOS Data / Time Not Set,' indicating that the date and time information in the computer's BIOS has not been set correctly. The user needs to press the specified key ('F2') to perform the corresponding operation. Please generate the keyboard operation steps."

[0062] }

[0063] Step 3: After understanding the large language model, the following control instructions are generated: [

[0065] {

[0066] "step":1,

[0067] "keyboard":"F2"

[0068] } ]

[0070] Step 4: The decision module sends control commands to the PC.

[0071] If the screen display received by the decision module is as follows Figure 3 As shown, the decision module may include the following steps when generating control commands:

[0072] Step 1: The image acquisition module will... Figure 3 The screen display shown is sent to the image understanding module.

[0073] Step 2: The image understanding module interprets the Linux system as unable to boot, and can then send the following data to the large language model:

[0074] {

[0075] "question": "This means there was a problem during startup, and the system failed to mount the root file system successfully. This could be due to various reasons, such as disk failure, incorrect boot parameters, or the storage device where the root file system resides not being detected. How can I proceed to enter the operating system?"

[0076] }

[0077] Step 3: After understanding the large language model, the following control instructions are generated: [

[0079] {

[0080] "step":1,

[0081] "description":"Restart PC"

[0082] },

[0083] {

[0084] "step":2,

[0085] "description": "Enumerate large-capacity storage devices and load the Ubuntu installation disc."

[0086] },

[0087] {

[0088] "step":3,

[0089] "description": "Boot from CD"

[0090] },

[0091] {

[0092] "step":4,

[0093] "description":"Entering the Ubuntu LiveCD system"

[0094] },

[0095] {

[0096] "step":5,

[0097] "description":"Install boot repair tool"

[0098] },

[0099] {

[0100] "step":6,

[0101] "description":"Repair bootloader"

[0102] } ]

[0104] Step 4: The decision module sends control commands to the PC.

[0105] If the screen display received by the decision module is as follows Figure 4 As shown, the decision module may include the following steps when generating control commands:

[0106] Step 1: The image acquisition module will... Figure 4 The screen display shown is sent to the image understanding module.

[0107] Step 2: The image understanding module interprets the problem as a Windows system boot failure. The image understanding module can then send the following data to the large language model:

[0108] {

[0109] The question states: "This image shows the Windows Startup Manager interface, which contains an error message about Windows failing to boot. The prompts indicate that you need to insert the Windows installation disc and restart your computer, then follow the on-screen instructions. When you see the language selection page, click 'Repair your computer.'"

[0110] }

[0111] Step 3: After understanding the large language model, the following control instructions are generated: [

[0113] {

[0114] "step":1,

[0115] "description":"Restart PC"

[0116] },

[0117] {

[0118] "step":2,

[0119] "description": "Enumerate mass storage devices and load the Windows installation disc."

[0120] },

[0121] {

[0122] "step":3,

[0123] "description": "Boot from CD"

[0124] },

[0125] {

[0126] "step":4,

[0127] "description": "Repair your computer"

[0128] } ]

[0130] Step 4: After the PC is plugged into the terminal, the user actively issues an operation command, such as: reinstall the operating system. At this time, the large language model can generate the following operation command: [

[0132] {

[0133] "step":1,

[0134] "description":"Restart PC"

[0135] },

[0136] {

[0137] "step":2,

[0138] "description": "Enumerate mass storage devices and load the Windows 11 installation disc."

[0139] },

[0140] {

[0141] "step":3,

[0142] "description": "Boot from CD"

[0143] },

[0144] {

[0145] "step":4,

[0146] "description": "Install operating system"

[0147] } ]

[0149] Step 5: The decision module sends control commands to the PC.

[0150] It should be noted that each set of instructions generates sub-instructions as needed, such as pressing a specific key or displaying a specific image on the screen. The large language model can determine whether each instruction has been executed completely based on the results of the sub-instructions.

[0151] In this embodiment, the control instructions generated by the decision module may include hardware operation instructions, which are operation instructions that need to be executed by hardware modules in the controlled device. For example, the control instruction is a mouse click operation, which needs to be executed by the mouse in the controlled device. In this case, in order to achieve control of the controlled device, in some embodiments, the USB controller is also used to enumerate the hardware modules corresponding to the controlled device according to the control instructions, and simulate the hardware modules to perform the operation corresponding to the control instructions. That is, when the control instructions require the hardware modules in the controlled device to perform corresponding operations, the USB controller can enumerate and simulate the hardware modules in the controlled device to perform the corresponding operations, thereby achieving control of the controlled device at the hardware level.

[0152] Taking a mouse click as an example, after receiving a control command, the USB controller can simulate the mouse of the controlled device to perform a click operation (without the mouse of the controlled device performing the click operation). At this time, the USB controller (or control device) can be equivalent to the mouse of the controlled device. Thus, the hardware module that simulates the hardware of the controlled device can be used to control the controlled device at the hardware level.

[0153] It should be noted that the hardware modules of the controlled devices mentioned above include, but are not limited to, mice, keyboards, and large-capacity storage devices, etc., without specific limitations here.

[0154] In some implementations, the USB controller is also used to enumerate Remote Network Driver Interface Specification (RNDIS) ports to communicate with the controlled device. This communication can be network communication; when the USB controller communicates with the controlled device by enumerating RNIDS ports, it can be considered that the USB controller is attempting to establish a network connection with the controlled device.

[0155] After the USB controller communicates with the controlled device, if the communication is successful, it indicates that the control device has successfully established a network connection with the controlled device. In this case, the control device can control the controlled device using software control methods in related technologies, or by combining the hardware control method provided in this application embodiment with the software control method in related technologies. If the USB controller fails to communicate with the controlled device, it indicates that the control device cannot establish a network connection with the controlled device. Consequently, the control device cannot control the controlled device using software. In this case, the control device can control the controlled device using the hardware control method provided in this application embodiment. That is, the information acquisition module in the controlled device can acquire the output information of the controlled device and send the output information to the decision module. The decision module analyzes the output information to generate control commands for controlling the controlled device and sends the control commands to the USB controller. The USB controller then sends the control commands to the controlled device to achieve control of the controlled device.

[0156] Optionally, in a more specific implementation, after the control device connects to the controlled device via a USB controller, the USB controller can first enumerate the RNIDs ports and communicate with the controlled device. If the communication is successful, the controlled device can be controlled using software control methods in related technologies. If the communication fails, the controlled device can be controlled using hardware control methods provided in the embodiments of this application. Therefore, different control methods can be used to control the controlled device based on the communication status between the control device and the controlled device, thereby improving the flexibility of controlling the controlled device and meeting control requirements.

[0157] In some implementations, the USB controller is also used to re-enumerate the Rndis port and re-communicate with the controlled device after sending control commands to the controlled device. That is, after controlling the controlled device via hardware, the controlling device can re-attempt communication with it. If communication is successful, the controlling device can control the controlled device using software control methods in related technologies; if communication fails, it can continue to control the controlled device using the hardware control methods provided in this application. Thus, by continuously attempting to communicate with the controlled device and using software control methods in related technologies when communication is successful, flexible switching of control methods can be achieved to meet control requirements.

[0158] Optionally, in some embodiments, during the process of successfully communicating with the controlled device and controlling the controlled device through software control methods in related technologies, the control device can obtain the status of the controlled device, such as network status and operating status. If the status of the controlled device is abnormal, causing the control device to be unable to control the controlled device through software control methods, the control device can control the controlled device through the hardware control methods provided in the embodiments of this application to meet the control requirements of the controlled device.

[0159] In some implementations, the control device may further include a processing module. The processing module can receive control commands from the decision module and forward them to the USB controller. That is, after generating control commands, the decision module can first send the control commands to the processing module, which then sends them to the USB controller. The processing module may simply forward the control commands, or it may process the commands before sending them to the USB controller; no specific limitation is made here.

[0160] Optionally, when the control instructions include hardware operation instructions, the processing module can also be used to enumerate the hardware modules corresponding to the controlled device according to the control instructions, and simulate the hardware modules to perform the operations corresponding to the control instructions, so as to realize automatic control of the controlled device. For example, if the control instruction is a mouse click operation, the processing module can simulate the mouse of the controlled device to perform a click operation after receiving the control instruction.

[0161] Compared to the above method where the hardware module emulates the controlled device via USB to perform the corresponding operations, having the processing module emulate the hardware module to perform the corresponding operations results in higher execution efficiency.

[0162] The aforementioned processing module can be a microcontroller unit (MCU) or other modules with processing capabilities; no specific limitations are made here.

[0163] It should be noted that in the above embodiments, the decision module is deployed in the control device. In some possible implementations, the decision module can also be deployed in the cloud, thereby enabling remote control of the controlled device via the cloud. In the case where the decision module is deployed in the cloud, the control device also includes a communication module for communication between the control device and the decision module (or the cloud). Specifically, after the information acquisition module in the control device obtains the output information of the controlled device, it can send the output information to the decision module in the cloud via the communication module. Upon receiving the output information, the decision module analyzes the output information and generates control commands. Then, it sends the control commands to the USB controller in the control device via the communication module (or alternatively, it can send the control commands to the processing module in the control device via the communication module, and the processing module then sends the control commands to the USB controller). Upon receiving the control commands, the USB controller sends them to the controlled device, thus achieving remote control of the controlled device.

[0164] To facilitate understanding of the control device and hardware control method provided in the embodiments of this application, several more specific implementation methods will be described below. In these implementation methods, the control device is a PC, and the output information of the controlled device is the screen display of the PC.

[0165] Figure 5 This is a schematic diagram of the structure of a control device according to another embodiment of this application. Figure 5 The control device shown includes a USB controller, a DisplayPort Alternate module, a keyboard simulation module, a mouse simulation module, a mass storage simulation module, an RNIDs module, an image encoder, a USB Gadget module, an MCU module, a Dynamic Host Configuration Protocol (DHCP) module, a Flash module, a decision module, and a communication module. The control device is connected to a PC via the USB controller, and the PC includes a USB Type-C interface.

[0166] The specific implementation method for controlling the PC is as follows:

[0167] 1. Connect the control device to the PC's USB Type-C full-function interface (the Thunderbolt interface also has a similar function).

[0168] 2. After the control device is connected to the PC, the USB controller automatically recognizes the peer device (i.e., the PC).

[0169] Optionally, if the control device has a screen, the control terminal can prompt the user whether to enter the hardware control state. After entering the hardware control state, the controlled device can be controlled at the hardware level.

[0170] 3. The USB controller enumerates the DisplayPort function module to capture the screen display of the PC.

[0171] 4. The screen display image is encoded by the encoder of the control device and then transmitted to the MCU. The MCU sends the image to the decision module for analysis and generates control commands.

[0172] 5. The decision module sends control commands to the MCU.

[0173] After receiving the control command, the MCU can determine whether to send hardware enumeration and control commands to the USB Gadget. If so, it can enumerate the corresponding hardware modules of the PC according to the control commands and execute the corresponding control operations by simulating those hardware modules. The simulated hardware modules here can be... Figure 5 The keyboard simulation module, mouse simulation module, and large-capacity storage simulation module are shown.

[0174] 6. The MCU module transmits control commands to the PC via the USB controller to achieve remote control.

[0175] Figure 5 The control device shown acquires and digitizes display signals by simulating the display device, mouse, keyboard, large-capacity storage device or other hardware peripherals of a PC. The screen signals are used to make decisions through a large language model. The decision is then converted into control commands input to the PC. This enables the control device to automatically control the PC through hardware simulation, thereby achieving the function of remotely and automatically maintaining PC equipment.

[0176] Figure 6 This is a schematic diagram of the structure of a decision module deployed on the control device side in one embodiment of this application.

[0177] Figure 6 In this system, the decision-making module is deployed on the control equipment side. Figure 6(Other modules in the control device are not shown). The decision module includes an image recognition module and a large language model. The image recognition module includes an OCR module, an icon recognition module, and an image understanding module. When the decision module analyzes the PC's screen display and generates control commands, the image recognition module can first perform initial recognition of the screen display. For example, the OCR module can recognize the text coordinates in the screen display, the icon recognition module can recognize the icon coordinates in the screen display, and the image understanding module can understand the screen display to obtain a preliminary recognition result. After obtaining the preliminary recognition result, the image recognition module can send it to the large language model. The large language model can perform scene analysis and decision-making based on the preliminary recognition result and generate control commands for controlling the PC.

[0178] It should be noted that, Figure 6 This explanation uses an example where the decision-making module includes an image recognition module and a large language model. In some possible implementations, the image recognition module and the large language model can also be modules independent of the decision-making module. For example... Figure 7 As shown, Figure 7 The decision-making module, image recognition module, and large language model are three independent modules. Figure 7 (Other modules in the control device are not shown). When the decision module receives the screen display, it can send the screen display to the image recognition module for preliminary recognition. After preliminary recognition, the image recognition module sends the preliminary recognition results to the large language model. After analysis and decision-making, the large language model sends the decision results to the decision module. The decision module generates control commands based on the decision results (or the large language model generates control commands based on the decision results and sends the control commands to the decision module).

[0179] Figure 8 This is a schematic diagram of a decision module deployed in the cloud, as shown in one embodiment of this application.

[0180] Figure 8 In the cloud, a decision-making module is deployed, and the control equipment includes a communication module. Figure 8 (Other modules in the control device are not shown). After acquiring the screen display from the PC, the control device can send the screen display to the decision module in the cloud via the communication module. The decision module analyzes the display and generates control commands, which are then sent back to the control device via the communication module. The decision module may include an image recognition module and a large language model. Alternatively, the decision module, image recognition module, and large language model can be separate modules. The decision module generates control commands by interacting with the image recognition module and the large language model. For specific implementation details, please refer to [link to implementation details]. Figure 6 and Figure 7The embodiments shown are not described in detail here.

[0181] Figure 9 This is a schematic diagram illustrating the control of multiple PCs according to an embodiment of this application.

[0182] Figure 9 In this context, the control device may include an MCU and three USB controllers (i.e., Figure 9 The USB controller 1, USB controller 2, and USB controller 3 shown, and the communication module ( Figure 9 (Other modules in the control device are not shown.) The control device can connect to PC1 via USB controller 1 to control PC1, connect to PC2 via USB controller 2 to control PC2, and connect to PC3 via USB controller 3 to control PC3. The decision module is deployed in the cloud. The control device can interact with the decision module via the communication module to obtain control commands generated by the decision module. When the decision module sends control commands to the control device via the communication module, it can also send them to the MCU module, which will then forward the control commands to the corresponding PC.

[0183] Figure 10 This is a schematic diagram of a control device designed inside a PC, according to an embodiment of this application.

[0184] In the foregoing embodiments, the control device and the PC are two independent devices. In some possible implementations, the control device is designed to be integrated within the PC, depending on the application scenario, such as... Figure 10 As shown. Figure 10 In the system, the control equipment deployed inside the PC includes an image acquisition module, a USB controller, an MCU, and a communication module. A decision module is deployed in the cloud. The decision module can interact with the control equipment through the communication module, thereby realizing remote control of the PC.

[0185] Figure 11 This is a schematic diagram of the connection between the control device and the PC in one embodiment of this application.

[0186] When a control device is connected to a PC via a USB controller, it can connect to the PC's USB Type-C port through the USB controller. However, some PCs do not have a USB Type-C port. In this case, to connect the control device and the PC, the USB Type-C cable can be split into a USB cable and a High Definition Multimedia Interface (HDMI) cable (or Video Graphics Array (VGA) cable, DisplayPort (DP) cable). The control device's USB controller can then be connected to the PC via the USB cable and the HDMI cable, such as... Figure 11 As shown.

[0187] Based on the embodiments of this application described above, when the control device controls the controlled device, it can achieve automatic control of the controlled device at the hardware level through hardware control. Compared with the method of controlling the controlled device based on software-based remote control protocols, it is not limited by the software system state or network environment. Furthermore, since the control command for the controlled device is generated by analyzing the output information of the controlled device, the control command is more closely matched to the current state of the controlled device. Controlling the controlled device through this command can better meet the control requirements of the controlled device, thereby achieving accurate control of the controlled device.

[0188] This application also provides a device control method. When controlling the controlled device, the device control method can realize automatic control of the controlled device at the hardware level through hardware control, and is not limited by the software system status or network environment of the controlled device.

[0189] Figure 12 This is a schematic flowchart of a device control method according to an embodiment of this application. Figure 12 The device control method shown can be controlled by Figure 1 or Figure 5 The control device shown performs the operation and is connected to the controlled device via a USB controller. Figure 12 The device control method shown includes the following steps.

[0190] S122: Obtain the output information of the controlled device.

[0191] When the control device is connected to the controlled device via a USB controller, it can obtain the output information of the controlled device and control the controlled device based on the output information.

[0192] In some implementations, the output information of the controlled device may include the screen display of the controlled device. Accordingly, when acquiring the output information of the controlled device, the screen display of the controlled device may be acquired. For example, the control device can acquire the screen display of the controlled device by enumerating the display devices of the controlled device.

[0193] Since the screen display of the controlled device can usually intuitively reflect the current status of the controlled device, accurate control of the controlled device can be achieved by acquiring the screen display and analyzing the screen display to generate control commands.

[0194] Alternatively, in other possible implementations, the output information of the controlled device may also include other information, such as the logs of the controlled device, etc., which are not specifically limited here.

[0195] S124: Analyze the output information and generate control commands.

[0196] The analysis of output information here can be the identification and understanding of the output information, and the generated control commands can be used to control the controlled equipment.

[0197] In some implementations, the analysis of output information and the generation of control commands can be achieved in either of the following two ways:

[0198] The first method involves recognizing the output information, analyzing the recognition results of the output information based on the large language model, and generating control commands based on the analysis results.

[0199] The control device can have a built-in information recognition module and a large language model. When recognizing the output information, it can be recognized by the information recognition module. After obtaining the recognition result, it can be analyzed by the large language model and control commands can be generated based on the analysis result.

[0200] Optionally, when the output information includes the screen display, the information recognition module may include an image recognition module, which may include an OCR module, an icon recognition module, and an image understanding module, thereby enabling better recognition of the output information of the controlled device and improving the accuracy of the recognition results.

[0201] The second method involves sending the output information to the cloud, where it analyzes the information and generates control commands; and then receiving the control commands sent from the cloud.

[0202] The control device can have a built-in communication module. After acquiring the output information from the controlled device, it can send the output information to the cloud via the communication module. The cloud can have a built-in decision module, which includes an information recognition module and a large language model (or, the cloud can have a built-in decision module, information recognition module, and large language model). After receiving the output information, the decision module can analyze the output information and generate control commands through the information recognition module and the large language model. Then, the remote end can send the control commands to the control device via the communication module in the control device. The control device receives the control commands sent from the cloud via the communication module.

[0203] It should be noted that in practical applications, the method described above for analyzing the output information and generating control commands can be determined based on the actual scenario; no specific limitations are made here.

[0204] In some implementations, the controlled device can also be controlled by a user. Specifically, this may include the following steps:

[0205] Receive user commands, which are used to control the controlled equipment;

[0206] This includes analyzing the output information and generating control commands, including:

[0207] Analyze user commands and output information to generate control commands.

[0208] User commands can be voice commands or text commands, without specific limitations. Optionally, when the control device includes a display screen, after the control device is connected to the controlled device via a USB controller, it can display a prompt message on the display screen. This prompt message is used to prompt the user whether to enter assistant mode. In assistant mode, the user can issue user commands to the control device to control the controlled device via voice or text. If the user confirms entering assistant mode, they can send a confirmation message to the control device (for example, a confirmation option can be provided below the prompt message, and the user can send a confirmation message by clicking the confirmation option). After receiving the confirmation message, the control device can enter assistant mode. At this time, the user can issue user commands to the control device to control the controlled device via voice or text. After receiving the user commands, the control device can analyze the user commands and output information and generate control commands.

[0209] Because it can generate control commands for the controlled device by combining user instructions, it can not only achieve accurate control of the controlled device, but also meet the user's control needs for the controlled device. For the user, it can achieve the purpose of controlling the controlled device through voice or text.

[0210] S126: Send control commands to the controlled device. The control commands are used to control the controlled device.

[0211] After generating control commands, the control device can send the control commands to the controlled device. For example, the control commands can be sent to the control device via a USB controller. The control device can then perform corresponding operations based on the control commands, thereby enabling control of the controlled device.

[0212] In this embodiment, the control instructions generated by the control device may include hardware operation instructions, which are operation instructions that need to be executed by the hardware module in the controlled device. For example, the control instruction is a mouse click operation, which needs to be executed by the mouse in the controlled device. In some implementations, when the control instructions include hardware operation instructions, the following steps may also be included:

[0213] The hardware modules corresponding to the controlled device are enumerated according to the control instructions, and the hardware modules are simulated to perform the operations corresponding to the control instructions.

[0214] In other words, when a control command requires the hardware module in the controlled device to perform a corresponding operation, the control device can enumerate and simulate the hardware module in the controlled device to perform the corresponding operation (specifically, the USB controller or processing module in the control device can enumerate and simulate the hardware module in the controlled device to perform the corresponding operation), thereby enabling control of the controlled device at the hardware level.

[0215] Taking the mouse click operation as an example, the control device can simulate the mouse of the controlled device to perform the click operation (without the mouse of the controlled device performing the click operation). At this time, the control device can be equivalent to the mouse of the controlled device. Thus, the control device can be controlled at the hardware level by simulating the hardware module of the controlled device.

[0216] It should be noted that the hardware modules of the controlled devices mentioned above include, but are not limited to, mice, keyboards, and large-capacity storage devices, etc., without specific limitations here.

[0217] In S122 above, the control device can obtain the output information of the controlled device after connecting to the controlled device via a USB controller. In some embodiments, the control device can also obtain the output information of the controlled device in a specific scenario after connecting to the controlled device via a USB controller, thereby enabling automatic control of the controlled device through hardware control in a specific scenario.

[0218] Specifically, the process of a control device acquiring the output information of a controlled device may include the following steps:

[0219] Enumerate Rndis ports to communicate with the controlled device; if communication with the controlled device fails, obtain the output information of the controlled device; or,

[0220] If communication with the controlled device is successful, obtain the status of the controlled device; if the status of the controlled device is abnormal, obtain the output information of the controlled device.

[0221] After the control device connects to the controlled device via the USB controller, it can enumerate the Rnids ports to communicate with the controlled device. This communication can be considered as the control device attempting to establish a network connection with the controlled device. If the control device fails to communicate with the controlled device, it indicates that the control device cannot establish a network connection with the controlled device. Consequently, the control device cannot control the controlled device via software. In this case, the control device can control the controlled device via the hardware control method provided in this application embodiment. This allows it to obtain the output information of the controlled device and then control the controlled device using the methods described in S124 and S126 to meet the control requirements of the controlled device.

[0222] Optionally, if the control device successfully communicates with the controlled device, it indicates that the control device has successfully established a network connection with the controlled device. In this case, the control device can control the controlled device through software control methods in related technologies, or control the controlled device through a combination of hardware control methods provided in the embodiments of this application and software control methods in related technologies.

[0223] When the control device successfully communicates with the controlled device, it obtains the status of the controlled device, such as network status and operating status. If the status of the controlled device is abnormal, preventing the control device from controlling the controlled device through software control, the control device can control the controlled device through the hardware control method provided in this application embodiment. This allows the control device to obtain the output information of the controlled device and then control the controlled device through the methods described in S124 and S126 to meet the control requirements of the controlled device.

[0224] In some embodiments, after sending control commands to the controlled device, the control device may further include:

[0225] Re-enumerate the Rndis ports and re-communicate with the controlled device.

[0226] In other words, after controlling the controlled device via hardware, the control device can retry communicating with the controlled device. If communication is successful, the control device can control the controlled device via software control methods in related technologies. If communication fails, it can continue to control the controlled device via the hardware control methods provided in this application embodiment. Thus, by continuously attempting to communicate with the controlled device and using software control methods in related technologies when communication is successful, flexible switching of control methods can be achieved to meet control requirements.

[0227] To facilitate understanding of the device control method provided in the embodiments of this application, several more specific implementation methods will be described below. Please refer to [link to relevant documentation]. Figures 13 to 15 . Figures 13 to 15 The following explanation uses a PC as the controlled device and the screen display as the output information of the controlled device.

[0228] Figure 13 This is a schematic flowchart of another embodiment of the device control method of this application. Figure 13 The device control method shown includes the following steps.

[0229] Step 1: Insert the control device into the PC.

[0230] The control device can be connected to a PC via a USB controller, and the insertion of the PC can be detected when the connection is established.

[0231] Step 2: Control the device to enumerate Rnids ports and establish an Rnids connection with the PC.

[0232] The control device can automatically enumerate Rnids ports to obtain IP addresses, communicate with the PC, and establish an Rnids connection. If communication is successful, proceed to step 10. If communication fails, proceed to step 3.

[0233] Step 3: Identify PC anomalies.

[0234] Step 4: Start the hardware control scheme.

[0235] Step 5: Obtain the PC's screen display.

[0236] Step 6: The decision module understands the screen display and determines the cause of the PC malfunction.

[0237] The causes of these abnormalities could include system crashes, blue screens, or inability to boot.

[0238] Step 7: The decision module generates control commands.

[0239] Optionally, the decision-making module can also generate control coordinates through the OCR module and the icon recognition module.

[0240] Step 8: Enumerate the operations corresponding to the control instructions executed by the PC's hardware modules.

[0241] After the decision module generates control commands, it can transmit these commands to the MCU in the control device. The MCU then enumerates hardware modules such as keyboards and mice as needed to control the PC. If operations such as installing an operating system are required, it can enumerate large-capacity storage devices or optical discs to install the operating system.

[0242] Step 9: End after completing the control commands.

[0243] Step 10: Establish communication between the client on the control device side and the server on the PC side.

[0244] If communication fails, proceed to step 3; if communication is successful, proceed to step 11.

[0245] Step 11: Obtain PC status.

[0246] Step 12: The PC is running normally. Control the PC via software.

[0247] For example, you can control your PC via the SSH protocol.

[0248] Alternatively, if the PC malfunctions, step 3 can be performed.

[0249] Figure 14 This is a schematic flowchart of another embodiment of the device control method of this application. Figure 14 The illustrated embodiment can automatically detect and resolve PC device status issues during operation. Specifically, during operation, the PC is considered to be in an abnormal state when it cannot communicate with the remote server, when the client on the device side cannot communicate with the server on the PC side, or when the device screen is detected abnormally at regular intervals.

[0250] Figure 14 The device control methods shown include the following scenarios.

[0251] Scenario 1: If the PC cannot communicate with the remote server, the PC can automatically check its network status, such as pinging the server or checking the firewall. If the PC's network status is abnormal, the control device can detect the PC's screen display. If either the PC's network status or screen display is abnormal, the control device can generate control commands based on the PC's screen display to control the PC at the hardware level and resolve the PC's problems, such as... Figure 14 As shown.

[0252] Scenario 2: If the client on the control device side cannot communicate with the server on the PC side, the control device can automatically enumerate the Rndis port and check whether the IP address is assigned correctly in the DHCP service, attempt to check the network connectivity with the PC side, attempt to restart the Rndis port, and restart the server to resolve the PC's problem. Optionally, the control device can first determine whether the PC has crashed based on the PC's screen display. If the PC has crashed, the control device can generate control commands based on the PC's screen display. If the PC has not crashed, the control device can determine whether the IP address is assigned correctly in the DHCP service. If it is correct, the server on the PC side can be restarted; if it is not correct, the Rndis port can be re-enumerated and the DHCP service checked to re-establish communication with the PC. Figure 14 As shown.

[0253] Scenario 3: If an anomaly is detected during the periodic screen status check, the control device can generate control commands based on the PC's screen display to resolve the PC's problem, such as... Figure 14 As shown. Optionally, the control device can first determine the executable state of the PC screen based on the PC's screen display, and control the PC to perform the corresponding operation. Then, it can further determine whether the PC screen is responsive. If there is no response, it is determined that the PC is malfunctioning, and the decision module issues control commands to resolve the problem.

[0254] Figure 15 This is a schematic flowchart of another embodiment of the device control method of this application. Figure 15 The scenario corresponding to the embodiment shown can be a terminal device with a display screen as the control device. In this scenario, the user can issue user commands through the display screen, and the control device can control the PC according to the user commands and the PC's screen display. Figure 15 The illustrated embodiment includes the following steps:

[0255] Step 1: Insert the terminal device into the PC.

[0256] Step 2: The terminal device prompts whether to enter PC Assistant mode.

[0257] In PC-assisted mode, users can issue user commands to control the PC.

[0258] Step 3: After the user confirms entering PC Assistant mode, receive user commands.

[0259] User commands can be voice commands or text commands, etc.

[0260] Step 4: The terminal device controls the PC according to the user's instructions.

[0261] Specifically, the terminal device can identify and understand user commands through the decision module, generate corresponding control commands, and then send the control commands to the PC to control the PC.

[0262] Step 5: The terminal device obtains the screen display of the PC.

[0263] Step 6: The terminal device sends the PC's screen display to the decision module.

[0264] Step 7: The decision module generates control commands based on the screen display on the PC.

[0265] Specifically, the terminal device can first identify the PC's screen display through the image recognition module in the decision module, then analyze the image recognition results through the large language model in the decision module, and finally generate control commands in combination with user instructions.

[0266] Step 8: The terminal device enumerates the PC's hardware modules according to the control instructions and executes the operations corresponding to the control instructions.

[0267] For example, you can enumerate the PC's keyboard, mouse, or large-capacity storage devices.

[0268] Step 9: End after completing the control commands.

[0269] The device control method provided in this application allows the controlling device to connect to the controlled device via a USB controller, acquire the output information of the controlled device, analyze the output information, generate control commands for controlling the controlled device, and send the control commands to the controlled device via the USB controller. This hardware-based control enables automatic control of the controlled device at the hardware level, which, compared to software-based remote control protocols, is not limited by software system status or network environment. Furthermore, since the control commands are generated by analyzing the output information of the controlled device, they are more closely matched to the current state of the controlled device. Controlling the controlled device using these commands better meets the control requirements, thus achieving accurate control.

[0270] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0271] Figure 16 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 16 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0272] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 16 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0273] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0274] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming a device control unit at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0275] Acquire the output information of the controlled device;

[0276] The output information is analyzed and control commands are generated;

[0277] The control command is sent to the controlled device, and the control command is used to control the controlled device.

[0278] The above is as stated in this application. Figure 16The method executed by the device control apparatus disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0279] The electronic device can also perform Figures 12 to 15 The method, and realize the equipment control device in Figures 12 to 15 The functions described in the illustrated embodiments will not be repeated here.

[0280] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0281] In summary, the above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0282] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0283] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0284] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0285] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

Claims

1. A control device comprising a USB controller, an information collection module and a decision module, wherein: the USB controller is configured to connect the control device and a controlled device; the information collection module is configured to obtain output information of the controlled device and send the output information to the decision module; the decision module is configured to analyze the output information and generate a control instruction, and send the control instruction to the USB controller; and the USB controller is configured to send the control instruction to the controlled device, so as to control the controlled device.

2. The control device of claim 1, wherein the decision module comprises an information recognition module and a large language model; the information recognition module is configured to recognize the output information; and the large language model is configured to analyze the recognition result of the information recognition module and generate the control instruction according to the analysis result.

3. The control device of claim 1, further comprising a receiving module; the receiving module is configured to receive a user instruction and send the user instruction to the decision module, so as to control the controlled device; and the decision module is configured to generate the control instruction according to the analysis result of the output information and the user instruction.

4. The control device of claim 1, wherein when the control instruction comprises a hardware operation instruction: the USB controller is further configured to enumerate a hardware module corresponding to the controlled device according to the control instruction, and simulate the hardware module to perform an operation corresponding to the control instruction.

5. The control device of claim 1, further comprising a processing module; the processing module is configured to receive the control instruction from the decision module and forward the control instruction to the USB controller; and when the control instruction comprises a hardware operation instruction, the processing module is further configured to enumerate a hardware module corresponding to the controlled device according to the control instruction, and simulate the hardware module to perform an operation corresponding to the control instruction.

6. The control device of claim 1, wherein the USB controller is further configured to enumerate a remote network driver interface specification (Rndis) port to communicate with the controlled device.

7. The control device of claim 6, wherein the USB controller is further configured to re-enumerate the Rndis port and re-communicate with the controlled device after sending the control instruction to the controlled device.

8. The control device of any one of claims 1 to 7, wherein the decision module is deployed in the control device or the decision module is deployed in the cloud; when the decision module is deployed in the cloud, the control device further comprises a communication module configured to communicate between the control device and the decision module.

9. A device control method applied to a control device, wherein the control device is connected to a controlled device through a USB controller, and the control method comprises: obtaining output information of the controlled device; analyzing the output information and generating a control instruction; and sending the control instruction to the controlled device. ​ ​ ​ ​ ​ ​ wherein ​ ​ ​ ​ ​ ​ ​ ​ Send the control instruction to the controlled device, wherein the control instruction is used to control the controlled device.

10. The method of claim 9, wherein the analyzing the output information and generating the control instruction comprises: identifying the output information; analyzing the identified output information according to a large language model, and generating the control instruction according to the analysis result; or sending the output information to a cloud, wherein the cloud is configured to analyze the output information and generate the control instruction; and receiving the control instruction sent by the cloud. Receiving a user instruction, wherein the user instruction is used to control the controlled device.

11. The method of claim 9, further comprising: The analyzing the output information and generating the control instruction comprises: analyzing the user instruction and the output information and generating the control instruction.

12. The method of claim 9, wherein when the control instruction comprises a hardware operation instruction, the method further comprises: enumerating a hardware module corresponding to the controlled device according to the control instruction, and simulating the hardware module to perform an operation corresponding to the control instruction.

13. The method of claim 9, wherein the obtaining the output information of the controlled device comprises: enumerating a Rndis port to communicate with the controlled device; when the communication with the controlled device fails, obtaining the output information of the controlled device; or when the communication with the controlled device succeeds, obtaining a state of the controlled device; when the state of the controlled device is abnormal, obtaining the output information of the controlled device.

14. The method of claim 13, wherein after the control instruction is sent to the controlled device, the method further comprises: re-enumerating the Rndis port and re-communicating with the controlled device.

15. An electronic device comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method of any one of claims 9 to 14. ​ ​

Images