Human-computer interaction device interaction system and method based on microcontroller chip

Abstract

The application discloses a microcontroller chip-based human-computer interaction device interaction system and method, which comprises the following steps: after a USB controller of a host device and a USB controller of a microcontroller chip establish a communication channel, the host device identifies the microcontroller chip as an authenticated human-computer interaction device; the microcontroller chip starts a soft access point and opens a web server in the microcontroller chip for hosting a virtual human-computer interaction device control webpage while being authenticated as the human-computer interaction device; a mobile device connects with the microcontroller chip through the soft access point and accesses a network address of the web server, opens the virtual human-computer interaction device control webpage, transmits a human-computer interaction operation instruction of a user on the mobile device to the web server through the control webpage, and converts the human-computer interaction operation instruction of the user on the mobile device into a human-computer interaction control instruction of the host device; and the microcontroller chip controls human-computer interaction control of the host device through the human-computer interaction control instruction.

Description

Technical Field

[0001] This invention relates to the field of computer peripheral devices, specifically to a human-computer interaction system and method based on a microcontroller chip. Background Technology

[0002] With the increasing prevalence of smart homes, multimedia presentations, and remote control scenarios, users often need to input data on computers (such as HTPCs connected to TVs, media centers, and presentation computers) from a certain distance. Traditional wireless keyboards and mice are limited by the receiver distance and require carrying the physical device, making them inconvenient.

[0003] Several solutions exist in the existing technology: Mobile App Solution: This involves installing specific client software on a mobile phone to simulate a keyboard and mouse via WiFi or Bluetooth. The disadvantages are that users need to pre-install the app, compatibility across different platforms is poor, and development and maintenance costs are high.

[0004] Dedicated hardware solutions: such as air mice, air mice, etc. These are still dedicated hardware devices that need to be carried and charged.

[0005] Software sharing solutions: Software such as Synergy allows a single keyboard and mouse to be shared across multiple computers over a network, but it cannot be used on devices without the client installed.

[0006] Therefore, there is an urgent need in this field for a remote keyboard and mouse input solution that requires no dedicated hardware, no specific software installation, is ready to use, and is cross-platform. Summary of the Invention

[0007] The purpose of this invention is to provide a human-computer interaction system and method based on a microcontroller chip. This system and method can realize the input function of mobile devices such as mobile phones as virtual keyboards and mice through hotspot servers and HID device enumeration technology.

[0008] To achieve this objective, the present invention provides a human-computer interaction system based on a microcontroller chip, comprising: Host devices, microcontroller chips, and mobile devices; The host communication controller in the host device and the chip communication controller in the microcontroller chip establish a communication channel through a communication protocol. The host device uses the communication channel to identify the microcontroller chip as an authenticated human-computer interaction device. The microcontroller chip is used to start the soft access point while authenticating the human-computer interaction device, and at the same time start the web server in the microcontroller chip used to host the control webpage of the virtual human-computer interaction device; The mobile device is used to connect to the microcontroller chip via a soft access point, access the network address of the web server, and open the virtual human-computer interaction device control webpage. The virtual human-computer interaction device control webpage transmits the user's human-computer interaction operation instructions for the mobile device to the web server. The web server converts the user's human-computer interaction operation instructions for the mobile device into human-computer interaction control instructions for the host device. Microcontroller chips are used for human-machine interaction control of host devices via human-machine interaction control commands.

[0009] Furthermore, the USB controller in the host device and the USB controller in the microcontroller chip establish a communication channel via the USB protocol. The host device uses the communication channel to identify the microcontroller chip as an authenticated human-computer interaction device through an enumeration process.

[0010] Furthermore, the USB controller in the host device and the USB controller in the microcontroller chip establish a communication connection via the USB protocol. The method by which the host device identifies the microcontroller chip as an authenticated human-machine interface device through an enumeration process includes: after the microcontroller chip is inserted into the host device via the USB interface, the host device provides USB bus power to the microcontroller chip; the USB controller in the host device sends an inquiry request to the USB controller in the microcontroller chip to query the device description; the USB controller in the microcontroller chip sends the device category, device manufacturer, product ID number, and configuration quantity of the microcontroller chip to the USB controller in the host device; and the USB controller in the host device, based on the microcontroller chip... The system determines whether the current microcontroller chip is a matching human-machine interface device based on the device category, manufacturer, product ID, and configuration quantity. It then assigns a corresponding device address to the identified matching human-machine interface device. The USB controller in the host device sends a request to the corresponding device address to obtain the configuration descriptor, interface descriptor, endpoint descriptor, and HID report descriptor of the USB controller in the microcontroller chip. Based on these parameters, the USB controller in the host device loads the corresponding driver, thus identifying the microcontroller chip as a USB-compliant human-machine interface device, i.e., a certified human-machine interface device.

[0011] Furthermore, the method for the microcontroller chip to start a soft access point while authenticating as a human-computer interaction device includes: while authenticating as a human-computer interaction device, the microcontroller chip runs a firmware program stored in the microcontroller chip, the firmware program contains a pre-imported Wi-Fi function library, and by calling the API function in the Wi-Fi function library for starting a soft access point, an independent Wi-Fi network with a preset password and name is formed.

[0012] Furthermore, the device locates the WiFi network name of the microcontroller chip on the mobile device and enters a preset password to connect to the WiFi network, enabling the mobile device to connect to the microcontroller chip via a soft access point.

[0013] Furthermore, after the mobile device connects to the microcontroller chip via a WiFi network, it can enter the web server's network address in a browser to open a virtual human-computer interaction device control webpage on the mobile device.

[0014] Furthermore, the virtual human-computer interaction device control webpage is used to receive human-computer interaction operation instructions from the user on the mobile device, and transmit the human-computer interaction operation instructions from the user on the mobile device to the web server of the microcontroller chip via the HTTP communication protocol. The web server of the microcontroller chip converts the received human-computer interaction operation instructions from the user on the mobile device into standard HID data packets of human-computer interaction control instructions for the host device.

[0015] Furthermore, the method for transmitting user human-computer interaction operation commands for mobile devices to the web server of the microcontroller chip via the HTTP communication protocol specifically includes: when the mobile device performs touch operations on the virtual human-computer interaction device control webpage, the JavaScript program in the virtual human-computer interaction device control webpage captures the touch or click events of the mobile device, converts the touch or click events into specific mouse displacement amounts or key values, packages them into JSON command data packets, and transmits them to the web server of the microcontroller chip via the WiFi network through the HTTP communication protocol.

[0016] Furthermore, the method by which the web server of the microcontroller chip converts the received human-computer interaction operation instructions of the user to the mobile device into standard HID data packets of human-computer interaction control instructions of the host device includes: the web server of the microcontroller chip queries a preset key value mapping table according to the key value in the JSON instruction data packet, maps the key value to the key code defined in the HID keyboard usage page, or directly obtains the mouse displacement amount; the microcontroller chip calls a preset human-computer interaction device API function according to the mouse displacement amount or key code, generates a standard HID data packet conforming to the USB standard, and simulates real human-computer interaction device input operation.

[0017] Furthermore, a human-computer interaction method based on a microcontroller chip according to the system includes: The host communication controller in the host device and the chip communication controller in the microcontroller chip establish a communication channel through a communication protocol. The host device uses the communication channel to identify the microcontroller chip as an authenticated human-computer interaction device. The microcontroller chip activates the soft access point while authenticating the human-computer interaction device, and simultaneously starts the web server in the microcontroller chip used to host the control webpage of the virtual human-computer interaction device. After the mobile device connects to the microcontroller chip via a soft access point, it accesses the network address of the web server and opens the virtual human-computer interaction device control webpage. The virtual human-computer interaction device control webpage transmits the user's human-computer interaction operation instructions for the mobile device to the web server. The web server converts the user's human-computer interaction operation instructions for the mobile device into human-computer interaction control instructions for the host device. The microcontroller chip controls the host device's human-machine interaction control through human-machine interaction control commands.

[0018] The beneficial effects of this invention are as follows: Existing wireless input solutions suffer from limitations such as limited reception distance and the need to carry physical devices, while mobile app solutions suffer from pre-installed software, poor cross-platform compatibility, and high maintenance costs. Software sharing solutions rely on installing client programs on the host machine. The technical solution proposed in this invention effectively overcomes these shortcomings through its dual-mode collaborative design: When the microcontroller chip is connected to the host via USB, it can simultaneously enumerate as a standard HID keyboard and mouse device and activate an independent Wi-Fi hotspot and an embedded web server. Users can then connect to this hotspot using any browser-enabled mobile device and access the control webpage for operation within the graphical interface. The webpage sends touch commands to the controller in real time, which is then converted into standard HID data packets by the controller's firmware and sent to the host via the USB interface, thus simulating physical keyboard and mouse input. This technical solution achieves cross-platform control based entirely on standard web technology, requiring no dedicated software or drivers to be installed on the host or mobile device. It simultaneously completes device enumeration, network creation, and command forwarding through a single chip, providing a truly plug-and-play experience. Ultimately, the solution achieved remarkable technical results: it completely eliminated the dependence on dedicated physical peripherals or specific clients, providing users with a highly portable, readily available, and operating system and browser platform-independent remote input method, greatly improving the convenience and flexibility of remote operation in scenarios such as smart homes, business presentations, and industrial control, while also possessing good economic efficiency and prospects for promotion due to its simple hardware structure. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the control webpage for the virtual human-computer interaction device of the present invention; Figure 2 This is a schematic diagram of the digital keypad control webpage of the present invention; Figure 3 This is a schematic diagram of the structure of the present invention; Figure 4This is a schematic diagram of the overall structure of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0021] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: Example 1 like Figure 3 As shown, a human-computer interaction system based on a microcontroller chip includes: Host devices, microcontroller chips, and mobile devices; The host communication controller in the host device and the chip communication controller in the microcontroller chip establish a communication channel through a communication protocol. The host device uses the communication channel to identify the microcontroller chip as an authenticated human-computer interaction device. The microcontroller chip is used to start the soft access point while authenticating the human-computer interaction device, and at the same time start the web server in the microcontroller chip used to host the control webpage of the virtual human-computer interaction device; The mobile device is used to connect to the microcontroller chip via a soft access point, access the network address of the web server, and open the virtual human-computer interaction device control webpage. The virtual human-computer interaction device control webpage transmits the user's human-computer interaction operation instructions for the mobile device to the web server. The web server converts the user's human-computer interaction operation instructions for the mobile device into human-computer interaction control instructions for the host device. Microcontroller chips are used for human-machine interaction control of host devices via human-machine interaction control commands.

[0022] In some embodiments, such as Figure 4As shown, the overall structure of this invention consists of three main components: a mobile device, an ESP32-S2 module, and a host computer. The mobile device and the ESP32-S2 module establish a connection via Wi-Fi wireless technology, while the ESP32-S2 module and the host computer are physically connected via a USB wired interface. Users access a control webpage provided by the ESP32-S2 module's built-in web server through a browser on the mobile device and generate commands through the webpage interface. These commands are transmitted to the ESP32-S2 module via the Wi-Fi wireless network. The ESP32-S2 module, as the core processing unit, is responsible for receiving and parsing the commands from the mobile device, ultimately converting them into standard USB HID (Human Interface Device) data packets, and sending them to the host computer via the USB interface, thereby achieving remote control of the host computer.

[0023] In some technical solutions, the USB controller in the host device and the USB controller in the microcontroller chip establish a communication channel via the USB protocol. The host device uses the communication channel to identify the microcontroller chip as an authenticated human-computer interaction device through an enumeration process.

[0024] A plug-and-play, highly compatible underlying physical connection is established between the host device and the microcontroller chip via the USB protocol. This ensures that the host device (such as a PC or smart TV) can instantly and without drivers recognize the microcontroller chip (including but not limited to the ESP32-S2 microcontroller chip) as a standard keyboard and mouse. This provides a stable, reliable data transmission channel natively supported by the operating system for all subsequent wireless remote control functions. Secondly, the authentication and recognition achieved through this standardized enumeration process ensures broad compatibility of human-computer interaction devices (supporting all host platforms with USB interfaces and standard HID drivers), and enables the microcontroller chip to operate as an independent network and control unit while serving as a trusted input device.

[0025] In some technical solutions, the USB controller in the host device and the USB controller in the microcontroller chip establish a communication connection via the USB protocol. The method by which the host device identifies the microcontroller chip as an authenticated human-machine interface device through an enumeration process includes: after the microcontroller chip is inserted into the host device via the USB interface, the host device provides USB bus power to the microcontroller chip. The USB controller in the host device sends an inquiry request to the USB controller in the microcontroller chip for a device description. The USB controller in the microcontroller chip sends the device category, device manufacturer, product ID number, and number of configurations (i.e., the total number of functional configurations supported by the USB device) of the microcontroller chip to the USB controller in the host device. The B controller determines whether the current microcontroller chip is a matching human-machine interface device based on the device category, device manufacturer, product ID number, and configuration quantity of the microcontroller chip. It then assigns a corresponding device address to the identified matching human-machine interface device. The USB controller in the host device obtains the configuration descriptor, interface descriptor, endpoint descriptor, and HID report descriptor of the USB controller in the microcontroller chip by sending a request to the corresponding device address. The USB controller in the host device loads the corresponding driver based on the configuration descriptor, interface descriptor, endpoint descriptor, and HID report descriptor of the USB controller in the microcontroller chip, thus identifying the microcontroller chip as a human-machine interface device that conforms to the USB standard, i.e., a certified human-machine interface device.

[0026] Through a standardized USB enumeration process, plug-and-play and seamless compatibility between the host device and the microcontroller chip are achieved. While providing bus power, the host device sequentially obtains the device descriptor, assigns a unique address, and acquires detailed configuration and HID report descriptors, and loads the corresponding driver accordingly. This confirms the microcontroller chip as a certified human-machine interface device compliant with the USB protocol specification. This process not only ensures that the device can be natively recognized by the operating system as a keyboard and mouse input device, eliminating the tedious process of manual driver installation for users, but also establishes a stable, reliable, and trustworthy communication link through a strict descriptor verification and address allocation mechanism. This lays a hardware-level compliance and interoperability foundation for the subsequent realization of low-latency, high-precision virtual keyboard and mouse remote control.

[0027] In some technical solutions, the method for a microcontroller chip to activate a soft access point while authenticating as a human-computer interaction device includes: the microcontroller chip, while authenticating as a human-computer interaction device, runs a firmware program stored in the microcontroller chip. The firmware program contains a pre-imported Wi-Fi function library. By calling the API functions in the Wi-Fi function library used to activate the soft access point, an independent Wi-Fi network with a preset password and name is formed. This Wi-Fi network is the soft access point.

[0028] The soft access point setup achieves a high degree of device integration and extreme simplification of user experience: by simultaneously executing USB HID device enumeration and Wi-Fi soft access point startup on a single microcontroller chip, this solution enables the device to be recognized as a standard keyboard and mouse upon plug-and-play upon insertion into the host, and immediately creates an independent wireless control network out of the box. This eliminates the need for any additional network configuration, driver installation, or software downloads by the user, allowing them to directly access full remote keyboard and mouse control capabilities via a universal browser using their mobile device. This not only completely solves the problems of inconvenience in carrying traditional physical keyboards and mice and the strong dependence on dedicated wireless transceivers, but also ensures that the low latency reliability of wired transmission and the convenience of wireless control coexist on the same device through parallel processing of hardware functions. This significantly improves device portability, deployment flexibility, and cross-platform compatibility in scenarios such as smart homes, business presentations, and industrial control.

[0029] In some technical solutions, the mobile device finds the WiFi network name of the microcontroller chip and enters a preset password to connect to the WiFi network, thereby enabling the mobile device to connect to the microcontroller chip through a soft access point.

[0030] By establishing a direct, private, and device-specific wireless communication channel through a soft access point, a point-to-point instant connection between the mobile device and the microcontroller chip is achieved. Users can complete network configuration without relying on any external routers or internet infrastructure, greatly simplifying the usage process and improving the ease of device deployment. At the same time, through preset SSID and password (or open connection), the connection process is guaranteed to be highly user-friendly and provide an "instant search and connect" experience, while also ensuring the independence and controllability of the communication link at the physical layer. This effectively avoids interference and competition in the public network environment and provides a stable and reliable dedicated data transmission foundation for subsequent low-latency, high-response virtual keyboard and mouse remote control. This is the prerequisite for realizing the core advantages of the entire solution: wireless, portable, and ready to use.

[0031] In some technical solutions, after a mobile device connects to a microcontroller chip via a WiFi network, the user can enter the web server's network address in a browser to open a virtual human-computer interaction device control webpage on the mobile device.

[0032] After a mobile device connects to the microcontroller chip via a WiFi network, entering the web server's network address in a browser opens a virtual human-computer interaction device control webpage on the mobile device. This achieves cross-platform compatibility and zero-installation deployment, allowing any mobile device supporting a modern browser (regardless of iOS, Android, or other systems) to immediately obtain complete virtual keyboard and mouse control capabilities through a standard Wi-Fi connection and web access process without pre-installing any dedicated applications. This significantly reduces the user's learning curve and device adaptation costs. Furthermore, it constructs a centralized and unified control entry point and interactive interface. Through a graphical webpage hosted in a browser, it can flexibly present customized virtual keyboards, touchpads, and macro command panels, and utilizes web technologies (such as JavaScript) to achieve low-latency real-time command transmission. This ensures intuitive operation and functional scalability while encapsulating complex device simulation logic in the backend, providing users with an immersive remote control experience indistinguishable from a physical keyboard and mouse, and significantly enhancing the maintainability and iterability of the solution.

[0033] In some embodiments, the virtual human-computer interaction device control webpage includes a graphical virtual keyboard interface and / or a virtual touchpad interface. One embodiment of the virtual human-computer interaction device control webpage is, for example... Figure 1 As shown, the connected status and IP address 192.168.4.1 at the top of the webpage indicate that the ESP32-S2 (microcontroller chip) has successfully started a soft access point and hosted a lightweight web server (i.e., a web server). Mobile devices can establish a network connection with the chip by connecting to this hotspot and accessing this fixed IP. The text input boxes and various keys in the keyboard control area (including letters, function keys, and combination keys such as Ctrl+C) fully demonstrate the function of the webpage as a virtual human-computer interaction interface. Any clicks or input operations by the user here will be captured by the JavaScript program in the page and converted into structured instruction data (such as key values ​​or coordinates). This data will be transmitted to the ESP32-S2 chip in real time via the HTTP communication protocol. After receiving the instructions, the ESP32-S2 firmware will convert them into standard HID key codes or mouse displacements according to the preset key value mapping table, and call the corresponding HID API to generate keyboard or mouse report data packets that conform to the USB protocol. Finally, it will be sent to the host via the USB interface, thereby simulating real keyboard and mouse input operations. The entire process realizes seamless, low-latency remote control from webpage touch to host response.

[0034] A second implementation of a virtual human-computer interaction device controlling a webpage, for example Figure 2As shown in the figure, a wireless HID (Human Machine Interface) keyboard simulator based on the ESP32-S3 microcontroller is displayed in practice. This interface is a control webpage hosted by the ESP32-S3's built-in web server. Users can access this page by connecting to a Wi-Fi hotspot created by the ESP32-S3 using mobile devices such as smartphones. Its core function is to transform the mobile device's touchscreen into a virtual numeric keypad with a complete key layout. The figure clearly shows that the device has been successfully connected ("Connected: ESP32-Keypad") and is in "Numerical Mode," with the Num Lock indicator light on ("ON"). This confirms that the ESP32-S3 is operating stably in dual working modes: on one hand, it acts as a USB HID device connected to a host (such as a computer) and recognized as a standard keyboard; on the other hand, it acts as a soft access point and web server, responsible for receiving user touch commands from this webpage interface. When the user clicks the virtual keys on the webpage (such as "7", "Enter", "Num Lock"), the keyboard is activated. When "Lock" is selected, JavaScript captures the operation and converts it into a specific key value instruction, which is then sent to the ESP32-S3 via the HTTP communication protocol. The chip firmware then converts the instruction into the corresponding standard HID keyboard scan code according to the preset key value mapping table, and finally sends it to the host in the form of an HID report through the USB interface, thereby realizing remote keyboard input and control of the host. The whole process realizes the technical goal of transforming general mobile devices into plug-and-play, cross-platform dedicated wireless input peripherals.

[0035] In some embodiments, a portion of the implementation is extracted as follows: Figure 2 The example program for defining the keys on the numeric keypad is as follows: # HID Keyboard Settings `from adafruit_hid.keyboard import Keyboard`: Imports the Keyboard class from the HID library to create a software object that simulates a keyboard.

[0036] `from adafruit_hid.keycode import Keycode`: Imports the keycode constant library. `Keycode` contains the HID keycode values ​​for all standard keyboard keys such as A, B, Enter, and Space, used to map virtual keys to standard scan codes. # Initialize the keyboard `keyboard = Keyboard(usb_hid.devices)`: Initializes the keyboard device, creating a `Keyboard` instance and binding it to the USB HID device stack. After execution, the firmware can send key press reports to the host through methods of the `keyboard` object, such as `press()` and `release()`.

[0037] # WiFi Hotspot Configuration SSID = "ESP32-Keypad": Defines the WiFi name. PASSWORD = "12345678": Define the WiFi password. print("Starting up the ESP32-S3 numeric keypad..."): Prints a startup status message to the serial console, used for debugging and to inform the user that device initialization has begun.

[0038] # Start Hotspot try: This begins an exception handling block to catch and handle runtime errors that may occur. `wifi.radio.start_ap(SSID, PASSWORD)`: This is the core API call to start the soft access point. It calls the `start_ap` method of the radio control object in the Wi-Fi module, passing in the preset SSID (network name) and PASSWORD (password) as parameters. After executing this method, the ESP32-S3's Wi-Fi hardware will begin broadcasting a wireless signal with the specified name, forming an independent Wi-Fi network.

[0039] print(f"Hotspot started: {SSID}"): Uses f-string formatted output to print the message "Hotspot started successfully" to the serial console and displays the hotspot's SSID. This is feedback to developers or users to confirm that the network has been created.

[0040] `print(f"IP address: {wifi.radio.ipv4_address_ap}")`: Prints the IP address of the ESP32-S3 when it is used as an access point (AP). The `wifi.radio.ipv4_address_ap` property returns the IPv4 address in AP mode (usually 192.168.4.1 or other default values). This address is the entry point for the mobile device access control webpage. except Exception as e: This catches any exceptions (errors) that may be thrown in the try block and assigns the exception object to the variable e for later processing.

[0041] print(f"Hotspot startup failed: {e}"): If starting the hotspot fails (e.g., hardware failure, invalid parameters), an error message will be printed on the serial port, including specific exception details (e), to help debug the problem.

[0042] `microcontroller.reset()`: Calls the microcontroller hardware reset function after a hotspot startup failure. This restarts the entire ESP32-S3 chip, a simple means of recovering from a serious error by attempting to bring the device back to a working state through reinitialization.

[0043] # Create a socket pool and server `pool = socketpool.SocketPool(wifi.radio)`: Creates a network socket pool `pool`, where `wifi.radio` is the underlying radio frequency interface object of the previously imported Wi-Fi module. `SocketPool` manages the underlying socket resources required for all network connections based on this interface. It is the infrastructure that enables subsequent web servers to handle multiple network requests simultaneously, providing the necessary network I / O capabilities for HTTP communication.

[0044] `server = Server(pool, " / static", debug=True)`: This function uses the socket pool `pool` created in the previous step to construct an HTTP server instance `server`. The parameter `" / static"` specifies the root directory path where the server will store static resources (such as images and CSS files). The parameter `debug=True` enables debug mode, in which the server will output detailed runtime logs (such as received requests and error messages) to the console. This `server` object will be used to define routes and handle requests.

[0045] # Num Lock status `num_lock = True`: Defines a Boolean global state variable `num_lock` and initializes it to `True`. It simulates and tracks the state of the Num Lock indicator on the keyboard. This variable is crucial for subsequent logic: when the user clicks the virtual Num Lock key on the webpage, the program toggles the value of this variable (switching between `True` and `False`) and, based on its current state, determines whether to map the numeric keypad keys to numeric input (when `num_lock` is `True`) or the arrow / control keys (when `num_lock` is `False`).

[0046] # Key mapping - Num Lock enabled (number key mode) key_mapping_num = {: This creates a dictionary object to store key-value pair mappings. The keys of the dictionary are key identifier strings sent by the webpage (e.g., 7), and their corresponding values ​​are predefined standard HID keycode constants (e.g., Keycode.KEYPAD_SEVEN).

[0047] "numlock": Keycode.KEYPAD_NUMLOCK indicates that when the webpage sends the string "numlock", the firmware should simulate pressing the Num Lock key on the keyboard.

[0048] "divide": Keycode.KEYPAD_FORWARD_SLASH, indicates the division sign key ( / ) on the numeric keypad corresponding to the "divide" identifier on the webpage. "multiply": Keycode.KEYPAD_ASTERISK, "subtract": Keycode.KEYPAD_MINUS, "7": Keycode.KEYPAD_SEVEN, "8": Keycode.KEYPAD_EIGHT, "9": Keycode.KEYPAD_NINE, "add": Keycode.KEYPAD_PLUS, "4": Keycode.KEYPAD_FOUR, "5": Keycode.KEYPAD_FIVE, "6": Keycode.KEYPAD_SIX, "1": Keycode.KEYPAD_ONE, "2": Keycode.KEYPAD_TWO, "3": Keycode.KEYPAD_THREE, "enter": Keycode.KEYPAD_ENTER, "0": Keycode.KEYPAD_ZERO, "dot": Keycode.KEYPAD_PERIOD } : Similarly, in the key mapping code above, "7" to "9", "4" to "6", "1" to "3" and "0" are mapped to the numeric keys 7-9, 4-6, 1-3 and 0 on the numeric keypad, respectively. "add", "subtract", and "multiply" correspond to the addition (+), subtraction (-), and multiplication (*) keys on the numeric keypad, respectively. "enter" corresponds to the Enter key on the numeric keypad, and "dot" corresponds to the dot (.) key on the numeric keypad.

[0049] # Key mapping - Num Lock off (function key mode) key_mapping_func = {: Creates a new dictionary object to store key mappings in function key mode.

[0050] "numlock": Keycode.KEYPAD_NUMLOCK, the webpage identifies numlock as still mapping to the Num Lock key itself, used for switching modes.

[0051] "divide": Keycode.KEYPAD_FORWARD_SLASH, divide corresponds to the / key on the numeric keypad.

[0052] "multiply": Keycode.KEYPAD_ASTERISK, multiply corresponds to the asterisk (*) on the numeric keypad. "subtract": Keycode.KEYPAD_MINUS, subtract corresponds to the minus sign on the numeric keypad. "7": Keycode.HOME, # 7 -> Home: The number 7 is mapped to the Home key (used to move the cursor to the beginning of a line or the beginning of a document). "8": Keycode.UP_ARROW, # 8 -> Up arrow, the number 8 is mapped to the up arrow key.

[0053] "9": Keycode.PAGE_UP, # 9 -> Page Up, the number 9 is mapped to the Page Up key (to scroll up the page).

[0054] "add": Keycode.KEYPAD_PLUS, add corresponds to the plus sign on the numeric keypad.

[0055] "4": Keycode.LEFT_ARROW, # 4 -> Left arrow, the number 4 is mapped to the left arrow key.

[0056] "5": Keycode.KEYPAD_FIVE, # 5 remains as is (some keyboards are designed this way), the number 5 remains as is (mapped to the number 5 on the numeric keypad).

[0057] "6": Keycode.RIGHT_ARROW, # 6 -> Right Arrow, the number 6 is mapped to the right arrow key.

[0058] "1": Keycode.END, # 1 -> End, the number 1 is mapped to the End key (moves the cursor to the end of the line or the end of the document).

[0059] "2": Keycode.DOWN_ARROW, # 2 -> Down arrow, the number 2 is mapped to the down arrow key.

[0060] "3": Keycode.PAGE_DOWN, # 3 -> Page Down, the number 3 is mapped to the Page Down key (to scroll down the page).

[0061] "enter": Keycode.KEYPAD_ENTER, the Enter key function remains unchanged, still corresponding to the Enter key on the numeric keypad.

[0062] "0": Keycode.INSERT, # 0 -> Insert, the number 0 is mapped to the Insert key (to switch between insert / overwrite input modes).

[0063] "dot": Keycode.DELETE # . -> Delete, the dot (.) key is mapped to the Delete key.

[0064] } # Get current key mapping def get_key_mapping(): Defines a function named get_key_mapping.

[0065] The function returns the result of a ternary conditional expression: if the global variable `num_lock` is True (i.e., Num Lock is enabled), it returns the previously defined `key_mapping_num` dictionary (for numeric key mappings); otherwise, it returns the `key_mapping_func` dictionary (for function key mappings). This ensures that the firmware can select the correct key mapping table based on the current Num Lock state.

[0066] # Get button display name The function `get_key_display_name(key, isnumlockon)` is defined. It takes two arguments: `key` (a string representing the key identifier) ​​and `isnumlockon` (a boolean value indicating whether Num Lock is enabled). It is used to determine the text label that should be displayed for each key on the webpage.

[0067] If key in ["divide", "multiply", "subtract", "add", "enter", "dot", "numlock"]: Check if the passed key belongs to the special function key list (including division, multiplication, subtraction, addition, enter, decimal point, and Num Lock key). If the key is specified, then the key itself is returned as the display name. if isnumlockon: If this condition is true (i.e., numeric mode is enabled). return key # If displaying numbers in numeric mode, return key as the display name.

[0068] else: # Function mode displays the function name `func_names = {` creates a Python dictionary named `func_names` to store the mapping from specific key identifiers in function mode to user-friendly display names.

[0069] "7": "Home", "8": "↑", "9": "PgUp", This maps key identifier 7 to the display name Home. In function mode, the number 7 key corresponds to the Home function. Key identifier 8 is mapped to the display name ↑ (up arrow symbol). Key identifier 9 is mapped to the display name PgUp (an abbreviation of Page Up). "4": "←", "5": "5", "6": "→", Mapping key identifier 4 to display name "←" (left arrow symbol); Mapping key identifier 5 to display name 5, in function mode, the number 5 key usually retains its original number function; Mapping key identifier 6 to display name "→" (right arrow symbol). "1": "End", "2": "↓", "3": "PgDn", Mapping key identifier 1 to the display name End; mapping key identifier 2 to the display name "↓" (down arrow symbol); mapping key identifier 3 to the display name "PgDn" (short for Page Down). "0": "Ins", "dot": "Del", maps the key identifier 0 to the display name Ins (short for Insert); maps the key identifier dot (decimal point key) to the display name Del (short for Delete).

[0070] } The function `func_names.get(key, key)` performs a dictionary lookup operation, retrieving the display name corresponding to the given key from the `func_names` dictionary using the `get()` method. If the key is not found in the dictionary, the function returns the key itself as the default value.

[0071] # API Routing: Handling Key Requests `@server.route(" / api / keypress", POST)` is a decorator that registers the following function as a route on the web server. It specifies that the `keypress_handler` function will be invoked when a client sends a POST request to the path ` / api / keypress`.

[0072] The function `keypress_handler(request: Request)` is defined. It takes a `Request` object as a parameter, which contains all the information about the HTTP request sent by the mobile client.

[0073] The global num_lock declares a global variable num_lock that will be used and may be modified inside a function. This allows the function to change the state of Num Lock.

[0074] `try:` begins an exception handling block. If an exception occurs in subsequent code, the `except` clause can be used to catch and handle it, preventing program crashes. `data = request.json()` parses the body of the HTTP request and converts it from JSON format into a Python data structure, which is a key step in processing JSON data from web pages.

[0075] `key = data.get("key", "")` securely retrieves the value named `key` from the parsed data. If it does not exist, it returns an empty string `""`. This `key` is a key identifier that controls the webpage to send key presses.

[0076] `current_mapping = get_key_mapping()` calls the previously defined `get_key_mapping()` function to obtain the corresponding key mapping dictionary (numeric mode or function mode) based on the current `num_lock` state.

[0077] if key in current_mapping: Check if the received key identifier exists in the currently valid key mapping dictionary.

[0078] # Handling Num Lock toggling, explaining the special logic for handling the Num Lock key.

[0079] if key == "numlock": Determine if the received key press is the Num Lock key.

[0080] `num_lock = not num_lock` toggles the Num Lock state. If it's currently True (enabled), it changes to False (disabled), and vice versa. This implements the toggle function of Num Lock.

[0081] `print(f"NumLock:{'ON'ifnum_lockelse'OFF'}")` prints the corresponding status information to the console based on the value of `num_lock`, which is useful for debugging.

[0082] In some technical solutions, the virtual human-computer interaction device control webpage is used to receive human-computer interaction operation instructions from users on mobile devices, and transmits these instructions to the web server of the microcontroller chip via the HTTP communication protocol. The web server of the microcontroller chip then converts the received human-computer interaction operation instructions into standard HID data packets for human-computer interaction control instructions of the host device.

[0083] By capturing user touch input commands from mobile devices via a control webpage and forwarding them to a web server on the microcontroller chip via the HTTP communication protocol, and then having the server-side firmware convert them into standard HID data packets, a universal remote human-machine interface based entirely on web technology and independent of operating systems and hardware platforms is created. This allows any mobile device with a modern browser to be instantly transformed into a virtual keyboard and mouse without installing any dedicated applications or drivers, greatly reducing the user's learning curve and device adaptation costs. Furthermore, it fully utilizes existing mature web communication protocols (HTTP General Request) and the USB HID standard, ensuring real-time and reliable command transmission while achieving compatibility with all USB standard-supporting devices. The plug-and-play native compatibility of the HID protocol with host devices (such as Windows, macOS, Linux computers, smart TVs, embedded hosts, etc.) allows the host system to fully recognize it as a physical keyboard and mouse, thus seamlessly integrating it into the existing ecosystem. In short, this architecture completely encapsulates the complex instruction mapping, protocol conversion, and data encapsulation processes in the microcontroller firmware and web logic at the software level, providing users with a direct operating experience almost like a physical keyboard and mouse. It also provides a highly flexible, rapidly deployable, and cost-effective wireless, cross-platform human-computer interaction solution for scenarios such as smart offices, industrial control, and home entertainment.

[0084] In some technical solutions, the method of transmitting user human-computer interaction operation commands for mobile devices to the web server of the microcontroller chip via the HTTP communication protocol specifically includes: when the mobile device performs touch operation on the virtual human-computer interaction device control webpage, the JavaScript program in the virtual human-computer interaction device control webpage captures the touch or click events of the mobile device, converts the touch or click events into specific mouse displacement amounts or key values, packages them into JSON command data packets, and transmits them to the web server of the microcontroller chip via the WiFi network through the HTTP communication protocol.

[0085] The human-computer interaction operation command transmission process achieves highly flexible, on-demand remote human-computer interaction control: by converting the touch operation of the mobile device into standardized JSON command data packets in real time, and relying on the wide compatibility of the HTTP communication protocol for transmission, users can perform precise keyboard and mouse control of the host through any mobile terminal that supports a modern browser without carrying or connecting any physical input device. This not only solves the carrying constraints and space limitations of traditional hardware keyboards and mice, but also achieves cross-platform, zero-installation universal compatibility with the Web (i.e., web server) technology stack, while ensuring the real-time performance and reliability of command transmission. It provides a lightweight, highly scalable, and cost-effective wireless input solution for scenarios such as smart office, industrial remote operation, and multimedia interaction.

[0086] In some technical solutions, the method by which the web server of the microcontroller chip converts the received human-computer interaction operation instructions of the user to the mobile device into standard HID data packets of human-computer interaction control instructions of the host device includes: the web server of the microcontroller chip queries a preset key value mapping table according to the key value in the JSON instruction data packet, maps the key value to the key code defined in the HID keyboard usage page (in the USB HID specification, the unique identifier value assigned to each physical key in the "keyboard / numeric keypad page" is called the Usage ID), or directly obtains the mouse displacement amount. The microcontroller chip calls a preset human-computer interaction device API function according to the mouse displacement amount or key code to generate a standard HID data packet that conforms to the USB standard, simulating real human-computer interaction device input operation.

[0087] By mapping platform-independent JSON application-layer commands from web pages to standardized underlying hardware signals (HID keycodes and reports) in real time and unambiguously, this ensures that user operations on any mobile device browser can be accurately and reliably translated into input commands that the host operating system can natively recognize and execute. This perfectly bridges the gap between web interaction and hardware input at the software level. Secondly, by calling preset, highly abstract HID device API functions, this process not only significantly simplifies the complexity of firmware development and improves code maintainability and portability, but also ensures that the generated HID data packets fully comply with the official USB device class specification. This allows the microcontroller chip to be seamlessly recognized by the host as a real physical keyboard and mouse, achieving a plug-and-play experience indistinguishable from native devices, extremely low input latency, and broad compatibility across multiple operating systems such as Windows, macOS, and Linux. Finally, this technical solution achieves the core goal of simulating hardware devices in a purely software-based manner while ensuring functional reliability and cross-platform compatibility, providing users with a flexible and convenient remote control solution that is ready to use without the need for dedicated physical peripherals.

[0088] In some embodiments, the implementation process of this technical solution includes: Hardware implementation: The core hardware is the ESP32-S2 module (such as the LOLIN S2 Mini development board or a custom PCB), which needs to be connected to a USB Type-C or Micro-USB interface for connecting to the host and powering the device.

[0089] Software implementation (firmware portion, based on the CircuitPython example): USB HID device initialization: Configure the USB peripherals of the ESP32-S2 so that they can be correctly enumerated as keyboard and mouse.

[0090] Wi-Fi Soft AP Startup: Call the relevant API, set the SSID (which can be fixed as Virtual-KeyMouse-XXXX, where XXXX is the device ID suffix) and password (can be empty or a simple password), and start the access point. Web Server Setup: Configure an HTTP server to serve HTML pages for the root path / .

[0091] Send keyboard and mouse reports using the corresponding HID API.

[0092] Software implementation (web page portion): HTML structure: Constructing a touchpad area and virtual buttons <button>The interface.

[0093] CSS styles: Enhance the appearance to adapt to different mobile screen sizes.

[0094] JavaScript logic: Use TouchEvent and MouseEvent to listen for user actions on the touchpad and buttons.

[0095] The operations are converted into data such as displacement and key values.

[0096] Establish communication with the ESP32-S2 and send serialized instruction data in real time.

[0097] After the hardware and software are powered on and booted up, the user plugs this device into the USB port of the living room computer. The computer recognizes the new keyboard and mouse. The user takes out their mobile phone, finds the network named Virtual-KeyMouse-A1B2 in the Wi-Fi list, and connects. Opening a browser and entering 192.168.4.1, a virtual touchpad and keyboard appear on the screen. The user can then operate the living room computer from their mobile phone, controlling functions such as playing videos and adjusting volume.

[0098] Example 2 A human-computer interaction method based on a microcontroller chip according to the system includes: The host communication controller in the host device and the chip communication controller in the microcontroller chip establish a communication channel through a communication protocol. The host device uses the communication channel to identify the microcontroller chip as an authenticated human-computer interaction device. The microcontroller chip activates the soft access point while authenticating the human-computer interaction device, and simultaneously starts the web server in the microcontroller chip used to host the control webpage of the virtual human-computer interaction device. After the mobile device connects to the microcontroller chip via a soft access point, it accesses the network address of the web server and opens the virtual human-computer interaction device control webpage. The virtual human-computer interaction device control webpage transmits the user's human-computer interaction operation instructions for the mobile device to the web server. The web server converts the user's human-computer interaction operation instructions for the mobile device into human-computer interaction control instructions for the host device. The microcontroller chip controls the host device's human-machine interaction control through human-machine interaction control commands.

[0099] Example 3 The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the method described in Embodiment 2.

[0100] This invention can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented in whole or in part as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0101] Those skilled in the art will readily understand that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, combinations, substitutions, improvements, etc., made under the spirit and principles of the present invention are included within the protection scope of the present invention.

[0102] The contents not described in detail in this specification are existing technologies known to those skilled in the art.< / button>

Claims

1. A human-computer interaction device interaction system based on a microcontroller chip, characterized in that, It includes: Host devices, microcontroller chips, and mobile devices; The host communication controller in the host device and the chip communication controller in the microcontroller chip establish a communication channel through a communication protocol. The host device uses the communication channel to identify the microcontroller chip as an authenticated human-computer interaction device. The microcontroller chip is used to start the soft access point while authenticating the human-computer interaction device, and at the same time start the web server in the microcontroller chip used to host the control webpage of the virtual human-computer interaction device; The mobile device is used to connect to the microcontroller chip via a soft access point, access the network address of the web server, and open the virtual human-computer interaction device control webpage. The virtual human-computer interaction device control webpage transmits the user's human-computer interaction operation instructions for the mobile device to the web server. The web server converts the user's human-computer interaction operation instructions for the mobile device into human-computer interaction control instructions for the host device. Microcontroller chips are used for human-machine interaction control of host devices via human-machine interaction control commands.

2. The human-computer interaction system based on a microcontroller chip according to claim 1, characterized in that: The USB controller in the host device and the USB controller in the microcontroller chip establish a communication channel via the USB protocol. The host device uses the communication channel to identify the microcontroller chip as an authenticated human-computer interaction device through an enumeration process.

3. The human-computer interaction system based on a microcontroller chip according to claim 2, characterized in that: The USB controller in the host device and the USB controller in the microcontroller chip establish a communication connection via the USB protocol. The method by which the host device identifies the microcontroller chip as an authenticated human-machine interface device through an enumeration process includes: after the microcontroller chip is inserted into the host device via a USB interface, the host device provides USB bus power to the microcontroller chip; the USB controller in the host device sends an inquiry request to the USB controller in the microcontroller chip for a device description; the USB controller in the microcontroller chip sends the device category, device manufacturer, product ID number, and configuration quantity of the microcontroller chip to the USB controller in the host device; and the USB controller in the host device then determines the device description based on the microcontroller chip's specifications. The system determines whether the current microcontroller chip is a matching human-machine interface device by identifying the device category, manufacturer, product ID, and configuration quantity. It then assigns a corresponding device address to the identified matching device. The USB controller in the host device sends a request to the corresponding device address to obtain the configuration descriptor, interface descriptor, endpoint descriptor, and HID report descriptor of the USB controller in the microcontroller chip. Based on these parameters, the USB controller in the host device loads the corresponding driver, thus identifying the microcontroller chip as a USB-compliant human-machine interface device, i.e., a certified human-machine interface device.

4. The human-computer interaction system based on a microcontroller chip according to claim 1, characterized in that: The method for a microcontroller chip to start a soft access point while authenticating as a human-computer interaction device includes: while authenticating as a human-computer interaction device, the microcontroller chip runs a firmware program stored in the microcontroller chip. The firmware program contains a pre-imported Wi-Fi function library. By calling the API function in the Wi-Fi function library used to start the soft access point, an independent Wi-Fi network with a preset password and name is formed.

5. The human-computer interaction system based on a microcontroller chip according to claim 4, characterized in that: Locate the WiFi network name of the microcontroller chip on the mobile device, enter the preset password to connect to the WiFi network, and enable the mobile device to connect to the microcontroller chip through a soft access point.

6. The human-computer interaction system based on a microcontroller chip according to claim 5, characterized in that: After a mobile device connects to a microcontroller chip via a WiFi network, it can open a virtual human-computer interaction device control webpage on the mobile device by entering the web server's network address in a browser.

7. The human-computer interaction system based on a microcontroller chip according to claim 6, characterized in that: The virtual human-computer interaction device control webpage is used to receive human-computer interaction operation instructions from users on mobile devices, and transmit these instructions to the web server of the microcontroller chip via the HTTP communication protocol. The web server of the microcontroller chip converts the received human-computer interaction operation instructions into standard HID data packets of human-computer interaction control instructions for the host device.

8. The human-computer interaction system based on a microcontroller chip according to claim 7, characterized in that: The method for transmitting user human-computer interaction commands for mobile devices to a web server of a microcontroller chip via the HTTP communication protocol specifically includes: when a mobile device performs touch operations on a virtual human-computer interaction device control webpage, a JavaScript program in the virtual human-computer interaction device control webpage captures touch or click events of the mobile device, converts the touch or click events into specific mouse displacement amounts or key values, packages them into JSON command data packets, and transmits them to the web server of the microcontroller chip via the WiFi network through the HTTP communication protocol.

9. The human-computer interaction system based on a microcontroller chip according to claim 8, characterized in that: The method by which the web server of the microcontroller chip converts the received human-computer interaction operation instructions of the user to the mobile device into standard HID data packets of human-computer interaction control instructions of the host device includes: the web server of the microcontroller chip queries a preset key value mapping table according to the key value in the JSON instruction data packet, maps the key value to the key code defined in the HID keyboard user page, or directly obtains the mouse displacement amount, and the microcontroller chip calls a preset human-computer interaction device API function according to the mouse displacement amount or key code to generate a standard HID data packet conforming to the USB standard.

10. A human-computer interaction device interaction method based on a microcontroller chip according to the system of claim 1, characterized in that, include: The host communication controller in the host device and the chip communication controller in the microcontroller chip establish a communication channel through a communication protocol. The host device uses the communication channel to identify the microcontroller chip as an authenticated human-computer interaction device. The microcontroller chip activates the soft access point while authenticating the human-computer interaction device, and simultaneously starts the web server in the microcontroller chip used to host the control webpage of the virtual human-computer interaction device. After the mobile device connects to the microcontroller chip via a soft access point, it accesses the network address of the web server and opens the virtual human-computer interaction device control webpage. The virtual human-computer interaction device control webpage transmits the user's human-computer interaction operation instructions for the mobile device to the web server. The web server converts the user's human-computer interaction operation instructions for the mobile device into human-computer interaction control instructions for the host device. The microcontroller chip controls the host device's human-machine interaction control through human-machine interaction control commands.

Images