Method for controlling a computing device, computing device and computer storage medium

Abstract

The present disclosure relates to a method for controlling a computing device, a computing device and a computer storage medium. The method includes: creating a virtual device in a driver layer of the computing device; searching for a driver port corresponding to an input device of the computing device so as to obtain a function function address corresponding to the driver port; recording the transmission information of a message about an input device operation action through the obtained function function address; and in response to determining that a control presented at the computing device is detected, based on the recorded transmission information, transmitting a message about an input operation identifier generated for simulating an input operation from the driver layer to a system process for controlling a graphics-related subsystem of the computing device so that the computing device automatically executes input about the control. The present disclosure can realize operation automation even for some special controls.

Description

Technical Field

[0001] The present disclosure relates generally to information processing and, in particular, to methods for controlling computing devices, computing devices, and computer storage media. Background Art

[0002] With the development of technologies related to Robotic Process Automation (RPA) systems, RPA systems automate the manual operation processes of users by imitating the manual operation processes performed by users on the Graphical User Interface (GUI) of computing devices, so as to automatically repeat the manual operation processes of users on the GUI of computing devices, thereby significantly improving work efficiency.

[0003] In traditional solutions for controlling computing devices, such as but not limited to RPA-related technologies, when controls are presented in the graphical user interface of the computing device, the automation of the user's manual operation process is hindered. In some solutions for controlling computing devices, the application layer is used to simulate mouse and keyboard events through Win32 functions such as sendevent and sendmessage. However, for some special cases, such as controls with higher security, it is impossible to automate the operation process for the above controls.

[0004] In summary, the disadvantage of the conventional solution for controlling computing devices is that it is impossible to realize automated operation for some special controls, such as controls requiring higher security. Summary of the invention

[0005] The present disclosure provides a method for controlling a computing device, a computing device, and a computer storage medium, which can realize operation automation even for some special controls.

[0006] According to a first aspect of the present disclosure, a method for controlling a computing device is provided. The method includes: creating a virtual device at a driver layer of the computing device; searching for a driver port corresponding to an input device of the computing device to obtain a function function address corresponding to the driver port; recording, via the obtained function function address, transmission information of a message about an input device operation action; and in response to determining that a control presented at the computing device is detected, transmitting, from the driver layer, a message about an input operation identifier generated for simulating an input operation to a system process of a graphics-related subsystem for controlling the computing device based on the recorded transmission information, so that the computing device automatically performs input about the control.

[0007] According to the second aspect of the present invention, a computing device is also provided, which includes: at least one processing unit; at least one memory, which is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, and when the instructions are executed by the at least one processing unit, the computing device executes the method of the first aspect of the present disclosure.

[0008] According to a third aspect of the present disclosure, a computer-readable storage medium is further provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a machine, the method of the first aspect of the present disclosure is executed.

[0009] In some embodiments, creating a virtual device at a driver layer of a computing device includes: mounting a virtual device driver under the created virtual device so that an operating system of the computing device recognizes the virtual device as a physical device.

[0010] In some embodiments, obtaining the function address corresponding to the driver port includes: traversing the executable file corresponding to at least one of the keyboard and the mouse of the computing device via the virtual device driver mounted under the virtual device; and parsing the executable file to obtain the function address corresponding to the driver port of at least one of the keyboard and the mouse.

[0011] In some embodiments, obtaining the function address corresponding to the driver port of at least one of the keyboard and the mouse includes: determining the target driver object based on the driver name of the keyboard and the mouse of the computing device, so as to obtain the currently used device object associated with the target driver object; obtaining the starting address of the effective address space of the target driver object and the size of the effective address space; based on the obtained currently used device object, obtaining the mounted device object mounted under the device object, so as to obtain the device extension address of the deepest mounted device object; and traversing each pointer address backward with the device extension address of the deepest mounted device object as the starting pointer address, so as to determine the first effective address within the effective address space as the function function address corresponding to the driver port.

[0012] In some embodiments, determining the first valid address within the effective address space as the function address corresponding to the driver port includes: confirming whether the current pointer address backward from the starting pointer address meets the address validity condition; in response to confirming that the current pointer address backward from the starting pointer address meets the address validity condition, determining whether the current pointer address is greater than the starting address of the effective address space and less than the sum of the starting address of the effective address space and the size of the effective address space; and in response to determining that the current pointer address is greater than the starting address of the effective address space and less than the sum of the starting address of the effective address space and the size of the effective address space, determining the current pointer address as the function address corresponding to the driver port.

[0013] In some embodiments, based on the recorded transfer information, a message about an input operation identifier generated for simulating an input operation is transferred from a driver layer to a system process for controlling a graphics-related subsystem of a computing device, including: in response to determining that a control presented at the computing device is detected, a message about the input operation identifier is generated, the input operation identifier indicating simulated key information associated with the control; based on the transfer path indicated by the recorded transfer information, the message about the input operation identifier is transferred from the keyboard driver to the keyboard class driver; the operating system enables the system process to poll and read the message about the input operation identifier transferred from the keyboard class driver; and the system process distributes the message about the input operation identifier to corresponding processes so that the input window associated with the control is activated.

[0014] In some embodiments, the method for controlling a computing device further includes: determining a function function address corresponding to a driver port from among a plurality of acquired function function addresses based on a predetermined priority; and registering a device dispatch processing function based on a MajorFunction function pointer array of a device object.

[0015] In some embodiments, obtaining the function address corresponding to the driver port includes: obtaining the function address of the PS / 2 port keyboard driver; obtaining the function address of the USB port keyboard driver; and obtaining the function address of the keyboard class driver.

[0016] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a system for implementing a method for controlling a computing device according to an embodiment of the present disclosure is shown.

[0018] Figure 2 A flowchart of a method for controlling a computing device according to an embodiment of the present disclosure is shown.

[0019] Figure 3 A flow chart of a method for obtaining a function address corresponding to a driver port according to an embodiment of the present disclosure is shown.

[0020] Figure 4 A flowchart of a method for delivering a message regarding an input operation identification to a system process of a computing device according to an embodiment of the present disclosure is shown.

[0021] Figure 5A block diagram of an electronic device suitable for implementing the embodiments of the present disclosure is schematically shown.

[0022] In the various drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0023] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0024] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects.

[0025] As described above, the traditional scheme for controlling computing devices, such as the method of using the application layer to simulate mouse and keyboard events through Win32 functions such as sendevent and sendmessage, cannot implement operations on related interface controls when processing controls with higher security. It has been found that for controls with higher security (such as but not limited to security controls), the computing device will detect the transmission chain of key messages to confirm whether the transmission chain is a link that meets the requirements. The traditional key messages generated by using the application layer to simulate mouse and keyboard events through Win32 functions such as sendevent and sendmessage do not come from a link that meets the requirements, so the key messages will be discarded, which will cause the control to fail to be automatically executed.

[0026] In order to at least partially solve one or more of the above problems and other potential problems, an example embodiment of the present disclosure proposes a scheme for scheduling tasks. In the scheme of the present disclosure, by creating a virtual device in the driver layer of the computing device and finding the function function address of the driver port corresponding to the input device of the computing device, the present disclosure can traverse the function function address corresponding to the input device actually configured by the computing device through the virtual device driver. In addition, through the acquired function function address, the transmission information of the message about the input device operation action is recorded; and when it is determined that the control presented at the computing device is detected, based on the recorded transmission information, the message about the input operation identifier is transmitted from the driver layer to the system process of the computing device. The present disclosure can simulate the way that the operating system transmits the input operation hardware interrupt from the driver layer upward, and throw the message about the input operation identifier of the simulated control input operation to the system process. Since the message transmission chain simulates the transmission method for hardware interrupts, its transmission chain will be confirmed as a link that meets the requirements, so that the system process can successfully read and process the message, thereby enabling the computing device to automatically execute the input about the control. Therefore, the present disclosure can realize operation automation even for some special controls.

[0027] Figure 1 FIG. 1 is a schematic diagram of a system 100 for implementing a method for controlling a computing device according to an embodiment of the present disclosure. Figure 1 As shown, the system 100 includes: a computing device 110, a network 140, and a terminal device 130. The computing device 110 includes, for example, a processing unit 122, a plurality of input devices 120 (such as a keyboard, a mouse, a touch screen, etc.), and a display device. The processing unit includes, for example, a driver layer, a core layer, an event processing layer, and a user space. The computing device 110 and the terminal device 130 perform data exchange, for example, through the network 140.

[0028] Regarding the computing device 110, it is, for example, but not limited to, a server, a personal computer, and a mobile device. The computing device 110 can create at least one virtual device at the driver layer; and find the function function address of the driver port corresponding to the input device of the computing device. The computing device 110 can record the transmission information of the message about the input device operation action; and based on the recorded transmission information, the driver layer will transmit the message about the input operation identifier generated for simulating the input operation to the system process for controlling the graphics-related subsystem of the computing device, so that the computing device automatically executes the input about the control. In some embodiments, the computing device 110 can have one or more processing units, including dedicated processing units such as GPU, FPGA and ASIC, and general processing units such as CPU. In addition, one or more virtual machines can also be running on each computing device. The computing device 110 includes, for example: a virtual device creation unit 112, a function function address acquisition unit 114, a transmission information recording unit 116, and a driver layer transmission input operation identifier message unit 118.

[0029] The virtual device creation unit 112 is used to create a virtual device at the driver layer of the computing device.

[0030] As for the function address acquisition unit 114, it is used to find the driver port corresponding to the input device of the computing device, so as to acquire the function address corresponding to the driver port.

[0031] The transfer information recording unit 116 is used to record the transfer information of the message about the input device operation action via the acquired function address.

[0032] Regarding the driver layer transmission input operation identification message unit 118, it is used to determine whether a control presented at the computing device is detected; and if it is determined that the control presented at the computing device is detected, based on the recorded transmission information, a message about the input operation identification generated for simulating the input operation is transmitted from the driver layer to the system process for controlling the graphics-related subsystem of the computing device, so that the computing device automatically executes the input about the control.

[0033] The terminal device 130 is, for example, another remote computing device (e.g., a server, a personal computer, or a mobile device). In some embodiments, the computing device 110 may traverse the drivers of the input devices actually configured by the terminal device 130; and automatically repeat the user manual operation process on the GUI of the driving terminal device 130, thereby automating the user manual operation process. The terminal device 130 may also, if it is determined that a control presented at the terminal device 130 is detected, pass a message about the input operation identifier generated for simulating the input operation from the driver layer to the system process for controlling the graphics-related subsystem of the terminal device 130 based on the recorded transmission information, so that the terminal device 130 automatically performs the input about the control.

[0034] The following will be combined Figure 2 A method 200 for controlling a computing device according to an embodiment of the present disclosure is described. Figure 2 FIG. 2 is a flowchart of a method 200 for controlling a computing device according to an embodiment of the present disclosure. It should be understood that the method 200 may be implemented in Figure 5 The electronic device 500 described above is executed. Figure 1 The method 200 is executed at the described computing device 110. It should be understood that the method 200 may also include additional actions not shown and / or may omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0035] At step 202, a virtual device is created in the driver layer of the computing device 110. The virtual device is a virtual hardware device.

[0036] For example, the computing device 110 creates a virtual device in the driver layer of the computing device; and mounts the virtual device driver under the virtual device so that the operating system of the computing device recognizes the virtual device as a physical device (or hardware device). Since the operating system of the computing device needs to coordinate between different hardware devices mounted by the operating system, the operating system of the computing device allows the driver ports of other input devices (such as but not limited to a mouse, keyboard, and touch screen) of the computing device to be traversed via the virtual device driver. By adopting the above means, the present disclosure can traverse the drivers of the input devices actually configured by the computing device.

[0037] The driver ports of input devices include, for example but not limited to, the port i8042prt of the PS / 2 keyboard and the port driver Kbdhid of the USB keyboard.

[0038] At step 204, the computing device 110 searches for the driver port corresponding to the input device of the computing device to obtain the function address corresponding to the driver port. For example, the computing device 110 obtains the PS / 2 port keyboard driver function address, the USB port keyboard driver function address, and the keyboard class driver function address.

[0039] The method for obtaining the function address corresponding to the driver port includes, for example: traversing the executable file (e.g., PE file) corresponding to at least one of the keyboard and the mouse of the computing device through the virtual device driver mounted under the virtual device; and parsing the executable file to obtain the function address corresponding to the driver port of at least one of the keyboard and the mouse. For example, by parsing the structure of the PE file corresponding to the keyboard and the mouse of the computing device, the port where the corresponding function is mounted is found to record the corresponding function address.

[0040] For example, the computing device 110 determines the target driver object (DRIVER_OBJECT) based on the driver names of the keyboard and mouse of the computing device, so as to obtain the currently used device object (DRIVER_OBJECT) associated with the target driver object; obtains the starting address of the effective address space of the target driver object and the size of the effective address space; obtains the mounted device object mounted under the device object based on the currently used device object obtained, so as to obtain the device extension address (DeviceExtension) of the deepest mounted device object; and traverses each pointer address backwards with the device extension address of the deepest mounted device object as the starting pointer address, so as to determine the first effective address within the effective address space as the function address corresponding to the driver port. The following will be combined with Figure 3 The method 300 for obtaining the function address corresponding to the driver port is specifically described and will not be repeated here.

[0041] At step 206, if the computing device 110 records the delivery information of the message about the input device operation action via the acquired functional address.

[0042] For example, the computing device 110 records the transfer information of the hardware interrupt triggered by the input operation through the corresponding function function address corresponding to the driver port of the keyboard and mouse obtained. The transfer information includes, for example but not limited to: the transfer path of the hardware interrupt and the input operation identifier transferred. It should be understood that the way how the operating system transfers the hardware interrupt triggered by the input operation upward can be used to simulate the upward transfer behavior of the operating system according to the recorded transfer information to transfer the input interrupt associated with the simulated key operation of the control.

[0043] Regarding the transmission information for transmitting the hardware interrupt triggered by the input operation, the transmission process of the PS / 2 keyboard input information in the Windows operating system is used as an example to illustrate. When the user taps the key K on the keyboard configured by the computing device, the keyboard sends the electrical signal corresponding to the key K to the keyboard controller (i8042), so that the keyboard scan code corresponding to the key K is written to the keyboard I / O interface. The operating system of the computing device determines the corresponding interrupt information based on the acquired keyboard scan code and IOAPIC relocation, so as to determine the interrupt processing function for responding to the interrupt information. The operating system of the computing device calls the interrupt handler, and the interrupt service routine (ISR) encapsulates the input and output request packet (IRP) containing the key information based on the read keyboard scan code. The input and output request packet (IRP) is sent to the driver keyboard port (e.g., i8042prt.sys) so that the key information is sent to the keyboard class driver (e.g., Kdbclass.sys) via the keyboard port driver; the keyboard class driver encapsulates the key information into a system message to send to the system process (e.g., csrss.exe).

[0044] For example, the computing device 110 of the present disclosure can record, via the keyboard function address corresponding to the driver port of the PS / 2 keyboard (for example, i8042prt.sys), the transmission information of the driver port of the PS / 2 keyboard sending the message of key information to the keyboard class driver (keyboard class driver, used to control keyboard layout, keyboard voice, etc.), and the transmission information of the keyboard class driver encapsulating the key information into the system message and sending the key information message to the system process (for example, csrss.exe).

[0045] In some embodiments, method 200 further includes: determining a function function address corresponding to the driver port from among the acquired multiple function functions based on a predetermined priority; and registering a dispatch processing function based on the MajorFunction function pointer array of the device object.

[0046] The following code schematically illustrates how to register a dispatch processing function.

[0047] At step 208, the computing device 110 determines whether a control presented at the computing device is detected. The control is, for example but not limited to, an input box for security control. In some embodiments, the computing device 110 may also determine whether a control presented at the terminal device is detected.

[0048] At step 210, if the computing device 110 determines that a control presented at the computing device is detected, based on the recorded transfer information, a message regarding an input operation identifier generated for simulating an input operation is transferred from the driver layer to a system process for controlling a graphics-related subsystem of the computing device so that the computing device automatically executes input regarding the control.

[0049] For example, the computing device 110 simulates the message transmission process of the operating system from the driver layer regarding the input operation identifier generated for simulating the input operation based on the recorded transmission information, so that the system process for controlling the graphics-related subsystem of the computing device reads and processes the message regarding the input operation identifier.

[0050] Regarding the system process for controlling the graphics-related subsystem of the computing device, for example, the client / server runtime subsystem process (csrss.exe process) of the Windows system. The csrss.exe process (Client / Server RuntimeServer Subsystem process) is one of the core processes of the Windows system, which is used to control the graphics-related subsystem of the Windows system and manage threads.

[0051] Regarding the method of transmitting a message about an input operation identifier from a driver layer to a system process of a computing device, it includes, for example: if the computing device 110 determines that a control presented at the computing device is detected, a message about the input operation identifier is generated, and the input operation identifier indicates simulated key information associated with the control; based on the transmission path indicated by the recorded transmission information, the message about the input operation identifier is transmitted from the keyboard driver to the keyboard class driver (keyboard class driver); the operating system enables the system process to poll and read the message about the input operation identifier transmitted by the keyboard class driver; and the system process distributes the message about the input operation identifier to the corresponding process, so that the input window associated with the control is activated. The following will be combined with Figure 4 The method 400 for transmitting a message regarding an input operation identifier to a system process of a computing device is specifically described and will not be described in detail herein.

[0052] In the above scheme, by creating a virtual device in the driver layer of the computing device and finding the function function address of the driver port corresponding to the input device of the computing device, the present disclosure can traverse the function function addresses corresponding to the input devices actually configured by the computing device through the virtual device driver. In addition, through the acquired function function address, the transmission information of the message about the input device operation action is recorded; and when it is determined that the control presented at the computing device is detected, based on the recorded transmission information, the message about the input operation identifier is transmitted from the driver layer to the system process of the computing device. The present disclosure can simulate the way in which the operating system transmits the input operation hardware interrupt from the driver layer upward, and throw the message about the input operation identifier of the simulated control input operation to the system process. Since the message transmission chain simulates the transmission method for hardware interrupts, its transmission chain will be confirmed as a link that meets the requirements, so that the system process can successfully read and process the message, thereby enabling the computing device to automatically execute the input about the control. Therefore, the present disclosure can realize operation automation even for some special controls.

[0053] The following will be combined Figure 3 A method 300 for obtaining a function address corresponding to a driver port according to an embodiment of the present disclosure is described. Figure 3 FIG. 3 is a flowchart of a method 300 for obtaining a function address corresponding to a driver port according to an embodiment of the present disclosure. It should be understood that the method 300 may be implemented in Figure 5 The electronic device 500 described above is executed. Figure 1 The method 300 is executed at the described computing device 110. It should be understood that the method 300 may also include additional actions not shown and / or may omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0054] At step 302, the computing device 110 determines the target driver object based on the driver names of the keyboard and mouse of the computing device, so as to obtain the currently used device object associated with the target driver object. For example, the driver objects of the keyboard and mouse can be determined based on the driver object names of the keyboard and mouse through the ObReferenceObjectByName method, so as to obtain the information of the currently used device object.

[0055] Regarding driver objects and device objects, it should be understood that driver objects will create multiple device objects and stack these device objects into a vertical structure. This vertical structure is called a device stack. The input and output request packet (IRP) is first sent to the top-level device on the device stack. If this device does not process it, it can be sent down to the lower-level device objects until a certain device ends the input and output request packet (IRP).

[0056] Each driver has a unique PDRIVER_OBJECT class structure, which includes an array of MajorFunction function pointers.

[0057] The following code schematically illustrates the PDRIVER_OBJECT class structure.

[0058] The MajorFunction function pointer array in the PDRIVER_OBJECT class structure is a set of function pointer arrays used to record the main types of IRPs, which record the callback function that processes the current request. The MajorFunction array can be used to associate the input and output request package (I / O Request Package, IRP) with the dispatch function that processes it. When the application layer has an I / O request for a device object, it will find the corresponding MajorFunction array structure according to the driver where the device object is located, and then find its corresponding processing function according to the request type.

[0059] At step 304 , the computing device 110 obtains the start address of the effective address space of the target drive object and the size of the effective address space.

[0060] At step 306 , the computing device 110 obtains the mounted device object mounted under the device object based on the obtained currently used device object, so as to obtain the device extension address of the deepest mounted device object.

[0061] At step 306, the computing device 110 traverses each pointer address backwards with the device extension address of the deepest mounted device object as the starting pointer address, so as to determine the first effective address in the effective address space as the function address corresponding to the driver port.

[0062] Regarding determining the first valid address within the valid address space as the functional address of the function corresponding to the driver port, it includes, for example: the computing device 110 confirms whether the current pointer address backward from the starting pointer address meets the address validity condition; if the computing device 110 confirms that the current pointer address backward from the starting pointer address meets the address validity condition (MmIsAddressValid), determine whether the current pointer address is greater than the starting address of the valid address space (DriverStart) and less than the sum of the starting address of the valid address space and the size of the valid address space (DriverStart+DriverSize); if it is determined whether the current pointer address is greater than the starting address of the valid address space and less than the sum of the starting address of the valid address space and the size of the valid address space, determine the current pointer address as the functional address corresponding to the driver port.

[0063] For example, if the computing device 110 confirms that the current pointer meets MmIsAddressValid and is greater than DriverStart and less than DriverStart+DriverSize, the current pointer address is determined as the function address corresponding to the driver port.

[0064] By adopting the above means, the present disclosure can accurately obtain the functional address of the driver port corresponding to the input device of the computing device.

[0065] The following will be combined Figure 4 A method 400 for delivering a message regarding an input operation identification to a system process of a computing device according to an embodiment of the present disclosure is described. Figure 4 FIG. 4 is a flowchart of a method 400 for delivering a message about an input operation identifier to a system process of a computing device according to an embodiment of the present disclosure. It should be understood that the method 400 may be implemented, for example, in Figure 4 The electronic device 400 described above is executed. Figure 1 The method 400 is executed at the described computing device 110. It should be understood that the method 400 may also include additional actions not shown and / or may omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0066] At step 402 , if computing device 110 determines that a control presented at the computing device is detected, a message regarding an input operation identifier is generated, the input operation identifier indicating simulated key information associated with the control.

[0067] The message regarding the input operation identifier is, for example, an input / output request packet (IRP) encapsulating simulated key information.

[0068] At step 404 , the computing device 110 transmits a message regarding the input operation identifier from the keyboard driver to the keyboard class driver based on the transmission path indicated by the recorded transmission information.

[0069] For example, the computing device 110 passes information about simulated key presses from the keyboard driver to the keyboard class driver; the keyboard class driver encapsulates the simulated key press information into a system message, and passes the system message encapsulated with the simulated key press information to the kernel boundary (for example, the kernel boundary of the Windows system, zwCreate / zwReadFile); and then passes the system message encapsulated with the simulated key press information to the system call until the kernel boundary is penetrated (for example, zwCreate / zwReadFile).

[0070] At step 406 , the computing device 110 operating system enables the system process to poll and read the message about the input operation identifier transmitted by the keyboard class driver.

[0071] For example, a system process (eg, csrss.exe process) for controlling a graphics-related subsystem of a computing device reads and processes a system message encapsulating simulated key information and transmitted by a keyboard class driver.

[0072] At step 408 , the computing device 110 system process distributes a message regarding the input operation identifier to a corresponding process so that an input window associated with the control is activated.

[0073] For example, the system process distributes simulated key information to the thread message queues of each application; the application associated with the input window of the control reads the simulated key information from the thread message queue to call the user interface related application interface to update the user interface of the computing device.

[0074] By adopting the above-mentioned means, even for controls with higher security requirements, the computing device can detect that the source of the simulated operation information about the control is used to control the graphics-related subsystem of the computing device, and therefore considers the transmission link of the simulated operation information to be legal, thereby allowing the automatic execution of the control input.

[0075] Figure 5 The block diagram of an electronic device (or computing device) 500 suitable for implementing the embodiments of the present disclosure is schematically shown. The device 500 may be a device for implementing the execution Figures 2 to 4 The apparatus of methods 200, 300 and 400 shown. Figure 5 As shown, the device 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 502 or computer program instructions loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM, various programs and data required for the operation of the device 500 can also be stored. The CPU, ROM, and RAM are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0076] The multiple components in the device 500 are connected to the input / output (I / O) 505, including: an input unit 506, an output unit 507, a storage unit 508, and a central processing unit 501 to perform the various methods and processes described above, such as performing methods 200 to 500. For example, in some embodiments, methods 200, 300 and 400 can be implemented as a computer software program, which is stored in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 500 via the ROM and / or the communication unit 509. When the computer program is loaded into the RAM and executed by the CPU, one or more operations of the methods 200, 300 and 400 described above can be performed. Alternatively, in other embodiments, the CPU can be configured to perform one or more actions of the methods 200, 300 and 400 by any other suitable means (for example, by means of firmware).

[0077] It should be further noted that the present disclosure may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0078] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0079] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0080] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as C language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of a computer-readable program instruction, and the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0081] Various aspects of the present disclosure are described herein with reference to the flowchart and / or step diagram of the method, device (system), and computer program product according to the embodiments of the present disclosure. It should be understood that each of the steps in the flowchart and / or step diagram and the combination of the steps in the flowchart and / or step diagram can be implemented by computer-readable program instructions.

[0082] These computer-readable program instructions can be provided to a processor in a voice interaction device, a general-purpose computer, a special-purpose computer, or a processing unit of other programmable data processing devices, thereby producing a machine, so that when these instructions are executed by a processing unit of a computer or other programmable data processing device, a device that implements the functions / actions specified in one or more steps in a flowchart and / or a step diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions enable a computer, a programmable data processing device, and / or other devices to work in a specific manner, so that a computer-readable medium storing instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more steps in a flowchart and / or a step diagram.

[0083] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more steps in the flowchart and / or step diagram.

[0084] The flowcharts and step diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each step in the flowchart or step diagram may represent a module, a program segment or a portion of an instruction, which contains one or more executable instructions for implementing a specified logical function.

[0085] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0086] The above are only optional embodiments of the present disclosure and are not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for controlling a computing device, comprising: Create a virtual device at the driver layer of the computing device; Finding the driver port corresponding to the input device of the computing device to obtain the function address corresponding to the driver port, including: traversing the function addresses corresponding to the input devices actually configured by the computing device through the virtual device driver; Recording, via the acquired function function address, the transfer information of the hardware interrupt triggered by the input operation transmitted through the corresponding function function; the transfer information at least includes: the transfer path of the hardware interrupt and the transmitted input operation identifier; and In response to determining that a control presented at the computing device is detected, based on the recorded delivery information, a message generated for simulating an input operation and regarding an input operation identifier is delivered from the driver layer to a system process for controlling a graphics-related subsystem of the computing device so that the computing device automatically executes input regarding the control.

2. The method according to claim 1, wherein creating a virtual device at a driver layer of a computing device comprises: A virtual device driver is mounted under the created virtual device so that the operating system of the computing device recognizes the virtual device as a physical device.

3. The method according to claim 1, wherein obtaining the function address corresponding to the driver port comprises: Traversing the executable file corresponding to at least one of a keyboard and a mouse of the computing device via the virtual device driver mounted under the virtual device; as well as The executable file is parsed to obtain a function address corresponding to a driver port of at least one of the keyboard and the mouse.

4. The method according to claim 3, wherein obtaining the function address corresponding to the driver port of at least one of the keyboard and the mouse comprises: Determine a target driver object based on driver names of a keyboard and a mouse of the computing device so as to obtain a currently used device object associated with the target driver object; Get the start address and size of the effective address space of the target driver object; Based on the currently used device object obtained, the mounted device object mounted under the device object is obtained, so as to obtain the device extension address of the deepest mounted device object; as well as The device extension address of the deepest mounted device object is used as the starting pointer address to traverse each pointer address backwards, so as to determine the first valid address in the valid address space as the function address corresponding to the driver port.

5. The method according to claim 4, wherein determining the first effective address in the effective address space as the function address corresponding to the driver port comprises: Confirm whether the current pointer address backward from the starting pointer address meets the address validity condition; In response to confirming that the current pointer address backward of the start pointer address meets the address validity condition, determining whether the current pointer address is greater than the start address of the effective address space and less than the sum of the start address of the effective address space and the size of the effective address space; as well as In response to determining that the current pointer address is greater than the start address of the effective address space and less than the sum of the start address of the effective address space and the size of the effective address space, the current pointer address is determined as the function address corresponding to the driver port.

6. The method according to claim 1, wherein based on the recorded transmission information, transmitting a message about an input operation identifier generated for simulating an input operation from a driver layer to a system process for controlling a graphics-related subsystem of a computing device comprises: In response to determining that a control presented at the computing device is detected, generating a message regarding an input operation identifier indicating simulated key information associated with the control; Based on the transmission path indicated by the recorded transmission information, the keyboard driver transmits the message about the input operation identifier to the keyboard class driver; The operating system enables the system process to poll and read the message about the input operation identifier transmitted by the keyboard class driver; as well as The system process distributes a message about the input operation identifier to a corresponding process so that an input window associated with the control is activated.

7. The method according to claim 4, further comprising: Based on a predetermined priority, determining a function function address corresponding to the driving port from among the acquired multiple function function addresses; as well as Register the device dispatch processing function based on the MajorFunction function pointer array of the device object.

8. The method according to claim 1, wherein obtaining the function address corresponding to the driver port comprises: Get the PS / 2 port keyboard driver function address; Get the USB port keyboard driver function address; as well as Get the address of the keyboard driver function.

9. A computing device comprising: at least one processing unit; At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the computing device to perform the method of any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 8 when executed by a machine.

Images