A multi-modal input method and device based on a pressure-sensitive pen and a physical keyboard

Abstract

This application provides a multimodal input method and device based on the collaboration of a pressure-sensitive pen and a physical keyboard. The method includes: detecting the sensing signal generated by the electromagnetic pen above the screen of a tablet device and determining the hovering distance value of the electromagnetic pen; when the physical keyboard is connected and the hovering distance value is less than a first distance threshold, determining the initial input mode of the current foreground application; acquiring the pressure value received by the tip of the electromagnetic pen, performing analog-to-digital conversion on the pressure value to generate a pressure-sensitive value, and scanning the level changes of each key on the physical keyboard to generate a key scan code; switching the initial input mode according to the pressure-sensitive value and the key scan code, determining the pressure sensitivity level identifier according to the value range of the pressure-sensitive value, combining the modifier key value in the key scan code with the pressure sensitivity level identifier, and searching for the corresponding compound instruction from a preset shortcut key mapping table for input control. This application can maintain operational continuity and significantly improve input efficiency.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to a multimodal input method and device based on the collaboration of a pressure-sensitive pen and a physical keyboard. Background Technology

[0002] With the development of mobile computing devices, tablets have evolved from simple audio-visual entertainment tools into productivity devices that combine office and creative functions. To meet users' input needs in different scenarios, more and more tablet devices are beginning to support both physical keyboard connections and electromagnetic pressure-sensitive styluses as input peripherals. Physical keyboards have an efficiency advantage in text input scenarios, while electromagnetic pressure-sensitive pens can provide a natural writing experience close to pen and paper in scenarios such as drawing and handwriting. However, in actual use, users often need to frequently switch between keyboard input and handwriting input. For example, a user may suddenly need to sketch while editing text, or need to input text annotations while using drawing software. In existing technologies, users must manually click virtual buttons on the screen, plug or unplug the keyboard, or switch input modes through the system settings menu. This process not only interrupts the user's thought process but also reduces operational efficiency. Summary of the Invention

[0003] This application provides a multimodal input method and device based on the collaboration of a pressure-sensitive pen and a physical keyboard, which solves the problem of cumbersome input mode switching methods and reduced operation efficiency in related technologies.

[0004] The first aspect of this application provides a multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard, the multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard includes: The electromagnetic pen generates an induction signal above the screen of the tablet device by detecting the induction signal through an electromagnetic induction circuit, and the hovering distance of the electromagnetic pen is determined based on the intensity change of the induction signal. When the physical keyboard is connected and the hovering distance is less than the first distance threshold, the initial input mode of the current foreground application is determined according to the application type label of the current foreground application. The pressure value received by the tip of the electromagnetic pen is collected, the pressure value is converted from analog to digital to generate a pressure sensitivity value, and the level change of each key on the physical keyboard is scanned to generate a key scan code. The initial input mode is switched according to the pressure sensitivity value and the key scan code, and the pressure sensitivity level identifier is determined according to the value range of the pressure sensitivity value. After combining the modifier key value in the key scan code with the pressure sensitivity level identifier, the corresponding compound instruction is searched from the preset shortcut key mapping table for input control.

[0005] Optionally, in a first implementation of the first aspect of this application, the step of detecting the sensing signal generated above the screen of the tablet device by the electromagnetic induction circuit and determining the hovering distance value of the electromagnetic pen based on the intensity change of the sensing signal includes: The electromagnetic pen emits an alternating excitation signal through the drive coil in the electromagnetic induction circuit, receives the induced signals fed back to each induction coil by the resonant circuit of the electromagnetic pen, and performs analog-to-digital conversion on the induced signals to generate an induced electromotive force distribution matrix. By comparing each amplitude in the induced electromotive force distribution matrix, the maximum amplitude and the corresponding coordinate position of the target induction coil are determined, and the amplitude attenuation ratio is generated by performing a ratio calculation based on the maximum amplitude and a preset reference amplitude table. The offset vector between the coordinate position of the target induction coil and the preset center coordinate of the screen is determined, and the hovering distance value of the electromagnetic pen is generated by logarithmic inversion calculation based on the amplitude attenuation ratio and the exponential parameter in the preset electromagnetic field attenuation model.

[0006] Optionally, in the second implementation of the first aspect of this application, the step of switching the initial input mode based on the pressure sensitivity value and the key scan code includes: When the pressure sensitivity value is greater than or equal to the first pressure sensitivity threshold, the button scan code is masked to generate a masked empty scan sequence. Based on the application type tag, the corresponding shortcut key mapping table is extracted from the preset shortcut key mapping library. By matching the key scan code with the key value in the shortcut key mapping table, a shortcut command is generated, and the initial input mode is switched to the handwriting enhancement mode.

[0007] Optionally, in the third implementation of the first aspect of this application, the step of switching the initial input mode based on the pressure sensitivity value and the key scan code further includes: When the pressure sensitivity value is less than the second pressure sensitivity threshold and the key scan code is a non-empty sequence, a threshold comparison result is generated by comparing the pressure sensitivity value with the second pressure sensitivity threshold. Based on the threshold comparison result, the key scan code is transmitted transparently to generate a character input command, and the initial input mode is switched to keyboard priority mode.

[0008] Optionally, in the fourth implementation of the first aspect of this application, the step of determining the pressure sensitivity level identifier based on the numerical range of the pressure sensitivity value, combining the modifier key value in the key scan code with the pressure sensitivity level identifier, and then searching for the corresponding compound instruction in a preset shortcut key mapping table for input control includes: The pressure sensitivity value is compared with N preset interval thresholds to determine the target interval into which the pressure sensitivity value falls, and a pressure sensitivity level identifier is generated based on the interval number of the target interval. By extracting the bit segments from the key scan code, the bit segments corresponding to the modifier keys in the key scan code are obtained, and the modifier key value is determined according to the binary value of the corresponding bit segment. The modifier key value is then concatenated with the pressure sensitivity level identifier to generate a combined index code. Using the combined index code as the lookup key, a hash lookup is performed on the preset shortcut key mapping table to obtain the composite instruction corresponding to the combined index code, so as to realize multimodal input control between the pressure-sensitive pen and the physical keyboard.

[0009] Optionally, in the fifth implementation of the first aspect of this application, after the steps of determining the pressure sensitivity level identifier based on the numerical range of the pressure sensitivity value, combining the modifier key value in the key scan code with the pressure sensitivity level identifier, and searching for the corresponding compound instruction in the preset shortcut key mapping table for input control, the method further includes: The instruction type field of the composite instruction is parsed to generate an instruction type identifier; When the instruction type is identified as text input, the cursor position information in the display buffer of the tablet device is read to obtain the current cursor coordinates, and the focus window of the input method manager is redirected according to the current cursor coordinates to generate a handwriting input area handle; Collect a continuous trajectory point sequence of the electromagnetic pen on the screen area corresponding to the handle of the handwriting input area, and perform curve fitting on the continuous trajectory point sequence to generate handwriting contour data; By performing time-normalized distance calculations on the handwriting outline data and a preset character feature library, the text character corresponding to the handwriting outline data is determined, a recognition character code is generated, and the recognition character code is inserted at the current cursor coordinates.

[0010] A second aspect of this application provides a multimodal input device based on the collaboration of a pressure-sensitive pen and a physical keyboard. This multimodal input device is used to implement a multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard. The multimodal input device includes: The detection module is used to detect the sensing signal generated above the screen of the tablet device by the electromagnetic pen through the electromagnetic induction circuit, and to determine the hovering distance value of the electromagnetic pen based on the intensity change of the sensing signal. The determination module is used to determine the initial input mode of the current foreground application based on the application type label of the current foreground application when the physical keyboard is connected and the hover distance value is less than a first distance threshold. The generation module is used to collect the pressure value received by the tip of the electromagnetic pen, perform analog-to-digital conversion on the pressure value to generate pressure sensitivity value, and scan the level changes of each key on the physical keyboard to generate key scan code. The control module is used to switch the initial input mode according to the pressure sensitivity value and the key scan code, determine the pressure sensitivity level identifier according to the value range of the pressure sensitivity value, combine the modifier key value in the key scan code with the pressure sensitivity level identifier, and search for the corresponding compound instruction from the preset shortcut key mapping table for input control.

[0011] Optionally, the multimodal input device based on the collaboration of a pressure-sensitive pen and a physical keyboard further includes: The control module is also used to parse the instruction type field of the composite instruction and generate an instruction type identifier; when the instruction type identifier is a text input type, it reads the cursor position information in the display buffer of the tablet device to obtain the current cursor coordinates, and redirects the focus window of the input method manager according to the current cursor coordinates to generate a handwriting input area handle; it collects the continuous trajectory point sequence of the electromagnetic pen on the screen area corresponding to the handwriting input area handle, and performs curve fitting on the continuous trajectory point sequence to generate handwriting contour data; it calculates the time-normalized distance between the handwriting contour data and the preset character feature library to determine the text character corresponding to the handwriting contour data, generates a recognition character code, and inserts the recognition character code at the current cursor coordinates.

[0012] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute a computer program stored in the memory, and when the processor executes the computer program, it implements the steps of the multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard provided in the first aspect of this application.

[0013] The fourth aspect of this application provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard provided in the first aspect of this application.

[0014] In summary, the multimodal input method and device based on the collaboration of a pressure-sensitive pen and a physical keyboard provided in this application involves detecting the sensing signal generated by the electromagnetic pen above the tablet screen through an electromagnetic induction circuit, and determining the hovering distance value of the electromagnetic pen based on the intensity change of the sensing signal; when the physical keyboard is connected and the hovering distance value is less than a first distance threshold, determining the initial input mode of the current foreground application based on the application type tag of the current foreground application; collecting the pressure value of the electromagnetic pen tip, performing analog-to-digital conversion on the pressure value to generate a pressure-sensitive value, and scanning the level changes of each key on the physical keyboard to generate a key scan code; switching the initial input mode based on the pressure-sensitive value and the key scan code, and determining the pressure sensitivity level identifier based on the value range of the pressure-sensitive value; combining the modifier key value in the key scan code with the pressure sensitivity level identifier, and searching for the corresponding compound instruction in a preset shortcut key mapping table for input control. This application achieves seamless switching between keyboard text input and pressure-sensitive pen handwriting input by automatically sensing the status of the pressure-sensitive pen, the keyboard input status, and the application scenario. This eliminates the need for users to manually operate the switching button or repeatedly operate between different input modes, thereby maintaining operational continuity and fluency of thought, and significantly improving input efficiency. Attached Figure Description

[0015] Figure 1 A flowchart illustrating the multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard provided in this application embodiment; Figure 2 A schematic diagram of the program module of a multimodal input device based on the collaboration of a pressure-sensitive pen and a physical keyboard, provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0016] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] To address the cumbersome input mode switching methods and reduced operational efficiency of tablet devices in related technologies, this application provides a multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard, such as... Figure 1 This is a flowchart illustrating the multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard provided in this embodiment. The multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard includes the following steps: Step 110: Detect the induction signal generated by the electromagnetic pen above the screen of the tablet device through the electromagnetic induction circuit, and determine the hovering distance value of the electromagnetic pen based on the intensity change of the induction signal.

[0018] Specifically, the electromagnetic induction circuit detects the induction signal generated by the electromagnetic pen above the tablet screen, obtains information on the spatial position change of the electromagnetic pen, and continuously samples and converts the amplitude of the induction signal into an analog-to-digital matrix to form a numerical matrix reflecting the hovering state of the electromagnetic pen. At the same time, the relationship between the maximum amplitude and the coordinates of each induction coil is calculated to generate a hovering distance value. This distance value describes the height of the electromagnetic pen relative to the screen surface and provides spatial information for subsequent input mode determination. In addition, by comparing it with a preset reference amplitude table, the amplitude attenuation ratio is obtained, providing a physical reference for the calculation of the hovering distance. By recording the offset vector between the target induction coil coordinates and the screen center coordinates, basic data is established for locating the relative position of the pen tip.

[0019] Step 120: When the physical keyboard is connected and the hovering distance is less than the first distance threshold, determine the initial input mode of the current foreground application based on the application type label of the current foreground application.

[0020] Specifically, when the physical keyboard is detected as connected, the hovering distance value is compared with a first distance threshold to determine whether the electromagnetic pen is close to the screen surface. The corresponding application category information is read according to the type label of the foreground application. By analyzing the category information, an initial input mode indication is generated. This input mode indication identifies the type of input environment that the system should respond to. At the same time, by combining the physical keyboard connection status and hovering distance judgment results, the system provides a basis for selecting keyboard priority, handwriting enhancement, or mixed input modes, and provides a pre-judgment condition for subsequent pressure sensitivity value processing and key scanning, ensuring that the input status can maintain a correspondence with the foreground application type.

[0021] Step 130: Collect the pressure value of the electromagnetic pen tip, perform analog-to-digital conversion on the pressure value to generate pressure sensitivity value, and scan the level changes of each key on the physical keyboard to generate key scan code.

[0022] Specifically, the pressure applied by the tip of the electromagnetic pen is collected and converted into digital pressure-sensitive values ​​using an analog-to-digital converter. Simultaneously, the level changes of each key on the physical keyboard are scanned to generate key scan codes. These scan codes reflect the pressed state of each key and modifier key combination information. By storing and time-stamping the continuously sampled data, pressure-sensitive value sequences and keyboard event sequences are formed. Pressure sensitivity levels are divided according to the value range, providing a quantitative basis for input mode switching and shortcut operation mapping. At the same time, a keyboard input state database is established through the collection of key scan codes, providing multi-source signal input for determining user intent.

[0023] Step 140: Switch the initial input mode according to the pressure sensitivity value and the key scan code, determine the pressure sensitivity level identifier according to the value range of the pressure sensitivity value, combine the modifier key value in the key scan code with the pressure sensitivity level identifier, and find the corresponding compound instruction from the preset shortcut key mapping table for input control.

[0024] Specifically, when switching the initial input mode based on the pressure sensitivity value and the key scan code, the pressure sensitivity value is compared with a preset threshold to determine the priority of handwriting or keyboard input. A compound instruction is generated by comparing the key scan code with the key value in the shortcut key mapping table. At the same time, the pressure sensitivity level identifier is combined with the modifier key value to generate a combination index code. The corresponding operation instruction is obtained from the preset shortcut key mapping table through a hash lookup. Based on the input mode determination result, the keyboard input signal and the pressure pen input signal are selectively responded to and controlled. During the processing, the key scan code is shielded or transparently transmitted. At the same time, the cursor position and the trajectory data of the handwriting area are recorded and analyzed to ensure that the multimodal input signal can be transmitted and responded according to the determination logic, thereby completing the input process of collaborative control between the pressure pen and the physical keyboard.

[0025] In one optional implementation, the step of detecting the induced signal generated above the screen of the tablet device by the electromagnetic induction circuit and determining the hovering distance value of the electromagnetic pen based on the intensity change of the induced signal includes: transmitting an alternating excitation signal through the drive coil in the electromagnetic induction circuit, receiving the induced signals fed back to each induction coil by the resonant circuit of the electromagnetic pen, and performing analog-to-digital conversion on the induced signals to generate an induced electromotive force distribution matrix; determining the maximum amplitude and the corresponding coordinate position of the target induction coil by comparing each amplitude in the induced electromotive force distribution matrix, and generating an amplitude attenuation ratio by performing a ratio calculation based on the maximum amplitude and a preset reference amplitude table; determining the offset vector between the coordinate position of the target induction coil and the preset screen center coordinates, and performing logarithmic inversion calculation based on the amplitude attenuation ratio and the exponential parameter in the preset electromagnetic field attenuation model to generate the hovering distance value of the electromagnetic pen.

[0026] In this embodiment, an alternating excitation signal is emitted by the drive coil in the electromagnetic induction circuit, forming an alternating electromagnetic field on the surface of the tablet screen. This alternating electromagnetic field induces a current in the resonant circuit within the electromagnetic pen, causing the pen's own resonant circuit to generate a corresponding feedback electromagnetic field. This feedback electromagnetic field is transmitted to the induction coils on the tablet screen via spatial coupling. Each induction coil receives a voltage signal with a different amplitude. These voltage signals reflect the position and distance changes of the electromagnetic pen on the screen plane. The amplitude of the induced signal decays exponentially with the distance from the electromagnetic pen to the induction coil. By performing analog-to-digital conversion on the voltage signals received by each induction coil, the analog voltage signals can be converted into digital signals, forming a multi-channel digitized induced electromotive force distribution matrix. Each element in this matrix corresponds to the electromotive force amplitude and spatial position measured by an induction coil at the current moment, thus providing continuous spatial distribution information of the electromagnetic pen's position above the screen surface. For example, when drawing a simple handwritten note, when the stylus hovers over the upper left corner of the screen, the amplitude of the induction coil in the upper left corner will be significantly higher than that of the coil farther away from the stylus, while the amplitude of the coil in the lower right corner of the screen will be lower. This difference in amplitude in the matrix data can be used for subsequent position determination. After forming the induced electromotive force distribution matrix, by comparing the amplitudes in the matrix, the maximum amplitude and its corresponding target induction coil coordinates can be determined. The position of the induction coil corresponding to the maximum amplitude is the projection position of the stylus on the screen surface. By recording the target induction coil coordinates and combining them with the physical center coordinates of the screen, the offset vector of the stylus relative to the center of the screen can be calculated. This offset vector is a two-dimensional vector, including relative displacement values ​​in the X and Y directions, which can clearly describe the specific position of the stylus on the screen plane. For example, if the screen resolution is 2560×1600 pixels, the electromagnetic pen is in the upper left quadrant, the coil coordinates corresponding to the maximum amplitude are (400, 300), and the screen center coordinates are (1280, 800), then the offset vector is (-880, -500). This vector provides accurate spatial positioning information of the electromagnetic pen relative to the screen center. The amplitude attenuation ratio is generated by comparing the maximum amplitude with a preset reference amplitude table. The preset reference amplitude table records the electromotive force amplitude received by each induction coil at a known reference distance. By comparing the values, the attenuation ratio of the current amplitude relative to the reference amplitude can be obtained, thus reflecting the change in distance between the electromagnetic pen and the screen. To convert the amplitude attenuation ratio into the actual hovering distance, a logarithmic inversion calculation is required using the electromagnetic field attenuation characteristics. In this process, an exponential parameter is introduced, which describes the exponential relationship of electromagnetic signal attenuation with distance, i.e., the amplitude decreases exponentially with increasing distance. The inversion calculation can be performed by taking the natural logarithm of the amplitude attenuation ratio and dividing by the exponential parameter to obtain the vertical distance value of the electromagnetic pen hovering.For example, when the reference amplitude is 1 volt, the actual measured maximum amplitude is 0.5 volts, and the exponent parameter is 2, logarithmic inversion can calculate that the distance between the electromagnetic pen and the screen is approximately 0.35 units. This distance corresponds to the actual hovering height of the electromagnetic pen. Through the above continuous processing, the three-dimensional spatial information of the electromagnetic pen above the screen can be obtained. The X and Y directions are provided by the coordinates of the target induction coil and the offset vector, while the Z direction is obtained by inverting the amplitude attenuation ratio and the exponent parameter to obtain the hovering distance, thus achieving a precise digital representation of the electromagnetic pen's hovering position. In specific applications, such as when a user hovers the electromagnetic pen in drawing software to doodle, the system can determine when to activate the handwriting input detection module based on the electromagnetic pen's hovering distance. Simultaneously, it combines the X and Y directional positions to determine the screen coordinates corresponding to the electromagnetic pen, thereby displaying a real-time preview of the handwriting outline at the cursor position, ensuring the synchronization and accuracy of input and display. The entire process forms a closed loop through continuous sampling and numerical processing of the digital matrix, enabling dynamic tracking of the electromagnetic pen's position. This provides a reliable data foundation for subsequent pressure-sensitive acquisition and multimodal input switching, while ensuring accurate input judgment of the tablet device in handwriting, drawing, and keyboard input scenarios.

[0027] In one optional implementation, the step of switching the initial input mode based on the pressure sensitivity value and the key scan code includes: when the pressure sensitivity value is greater than or equal to a first pressure sensitivity threshold, the key scan code is masked to generate a masked empty scan sequence; the corresponding shortcut key mapping table is extracted from a preset shortcut key mapping library according to the application type label; a shortcut command is generated by matching the key scan code with the key value in the shortcut key mapping table; and the initial input mode is switched to the handwriting enhancement mode.

[0028] In this embodiment, when the pressure sensor of the pressure-sensitive pen measures a value greater than or equal to a first pressure threshold, the system determines that the current user's intention is biased towards handwriting operation. Therefore, it needs to mask the input signal from the physical keyboard to prevent keyboard key input from interfering with the stylus input. The masking process involves re-encoding the original binary encoding sequence of the key scan code, setting all key press signals to logical null values, and generating a masked empty scan sequence. This sequence, when passed to the input management module, will not trigger character input or shortcut functions, thus ensuring that the pressure-sensitive pen exclusively occupies the input channel during handwriting operations. In the specific implementation, the state of each keyboard key is represented by a scan code. Each bit of the scan code represents the level state of a key or modifier key. By clearing the entire scan code's bit segment, all key signals are temporarily ignored, while key timing information is retained for subsequent restoration of the original state. After generating the empty scan sequence, a preset shortcut key mapping library is accessed based on the foreground application's type tag. This library contains keyboard function mapping tables for different application types. Application type tags are identifiers of currently running programs provided by the operating system. These tags allow the extraction of corresponding shortcut key mapping tables from a mapping library. These tables map keyboard keys or key combinations to specific operation commands; for example, in drawing software, one key might be mapped to undo, and another to toggle pen thickness. The system generates shortcut commands by matching empty scan sequences with the key values ​​in the mapping table. Even if the original keyboard signals are blocked, predefined operations or functions can still be triggered. This matching is achieved by comparing the bit segment identifiers of the scan code with the key value encoding in the mapping table. Simultaneously, the corresponding mapping table is selected based on the current application tag, ensuring consistency between the shortcut commands and the application environment. In handwriting enhancement scenarios, these generated shortcut commands are used in conjunction with pressure-sensitive pen input. The pressure of the pen not only determines whether keyboard input is blocked but also affects the execution parameters of the commands. For example, the pressure value can control the width or transparency of the drawing brush. Combined with operation commands obtained from the shortcut key mapping table, multi-dimensional input control is achieved. By combining empty scan sequences with pressure-sensitive data, the system can send the pen trajectory input to the application's handwriting input buffer while ensuring that keyboard keys do not interfere with characters or commands. For example, when a user performs high-pressure writing in drawing software, if the pressure sensitivity exceeds a threshold, the key scan code is masked as an empty sequence. The application type label indicates that the current foreground application is drawing. The system retrieves a dedicated shortcut key table for drawing from the mapping library, matches the empty scan sequence to undo and switch pen operations, and maps the pressure-sensitive pen trajectory input to the canvas, achieving pen-priority interaction. After completing the masking and shortcut command generation, the initial input mode status indicator is switched to handwriting enhancement mode. This mode is used to identify that the current input is dominated by the pressure-sensitive pen and, combined with the previously obtained hover distance, pressure sensitivity value, and coordinate position information, provides the input manager with complete handwriting control conditions.In handwriting enhancement mode, the system keeps keyboard keys disabled, continuously monitoring the pen tip pressure and position to ensure accurate display of the user's handwriting input at the cursor. Simultaneously, shortcuts can be dynamically invoked through a mapping table, enabling multimodal input collaboration between the pen and shortcut keys within the same application environment. For example, when a user is doodling in drawing software, if the pen tip pressure exceeds a threshold, keyboard input is disabled. Shortcuts in the mapping table control pen switching, allowing the user to seamlessly switch colors or brush types without accidentally inputting characters via the keyboard, ensuring input consistency and operational continuity.

[0029] In one optional implementation, the step of switching the initial input mode based on the pressure sensitivity value and the key scan code further includes: when the pressure sensitivity value is less than a second pressure sensitivity threshold and the key scan code is a non-empty sequence, generating a threshold comparison result by comparing the pressure sensitivity value with the second pressure sensitivity threshold; transmitting the key scan code according to the threshold comparison result to generate a character input command, and switching the initial input mode to keyboard priority mode.

[0030] In this embodiment, when the pressure value of the pressure-sensitive pen is lower than the second pressure threshold and the physical keyboard key scan code is not empty, the system determines the user's input intention by comparing the current pressure value with the preset second pressure threshold. This comparison is based on a sequence of digital pressure values, each corresponding to the instantaneous pressure of the pen tip on the screen. The second pressure threshold is a reference value set according to the range of the hardware pressure sensor and the touch response characteristics, used to distinguish between slight contact and effective writing pressure. When the pressure value is lower than the threshold, it indicates that the user has not applied significant pressure, and the pen input should not be considered as handwriting or drawing operations; therefore, keyboard input should be prioritized. The threshold comparison result generated by the comparison calculation is a logical judgment signal used to indicate whether the pressure-sensitive pen is in an effective writing state under the current input environment. This signal is used as a conditional control in subsequent processing to determine whether keyboard scan codes are allowed to be passed through. The pass-through processing directly transmits the original binary information of the key scan code to the input management module, enabling key events to trigger character input or shortcut operations in the application. Key scan codes are generated by the keyboard circuitry when a key is pressed or released. Each bit or segment represents the electrical level of a specific key. The scan code accurately describes the key press time, duration, and modifier key combination information. During pass-through, the system does not mask or re-encode the scan code; instead, it maintains the original encoding to generate a character input command. This command contains the key code value and its timing information, which can be directly interpreted as character input or command operation by the operating system or foreground application. For example, when a user gently hovers the stylus above the screen in word processing software without applying writing pressure, and simultaneously presses the letter keys "A" and "B," the pressure sensitivity is below a second threshold. The threshold comparison indicates that the pen tip has not activated handwriting input, so the key scan code is passed through. The resulting character input command will display the letters "A" and "B" sequentially in the application's text input box without triggering the handwriting recognition module. After completing the pass-through generation of the character input command, the initial input mode is switched to keyboard priority mode. This mode indicates that the system currently prioritizes physical keyboard input and logically ignores the interference of stylus input on character input. In keyboard-priority mode, the input management module continuously monitors the status of key scan codes to ensure that user keyboard operations are fully recorded and responded to in real time. It also retains the collection of pressure-sensitive pen data but does not trigger handwriting input or interfere with character generation. Combined with the previously obtained hovering distance information, the system can ensure that keyboard input is not interfered with when the pen tip lightly touches the screen, guaranteeing the priority of keyboard operations in text editing environments.In specific application scenarios, such as when a user is preparing to compose an email in an email client, they can hover the stylus over the screen to browse the email content while simultaneously typing on the keyboard. If the pressure sensitivity value is below the second pressure sensitivity threshold, the threshold comparison result indicates that handwriting input is not enabled. The character input command generated by the key scan code will immediately appear in the email body, and the position of the stylus and slight touches will not interfere with text input, allowing the user to maintain a continuous keyboard input experience. Through a combination of conditional judgment of pressure sensitivity value and scan code, pass-through control, and input mode switching, the system can accurately distinguish between light stylus pressure and effective writing pressure in mixed input environments, prioritizing keyboard input while retaining the pressure sensitivity data foundation for subsequent handwriting enhancement operations.

[0031] In one optional implementation, the steps of determining a pressure sensitivity level identifier based on the numerical range of the pressure sensitivity value, combining the modifier key value in the key scan code with the pressure sensitivity level identifier, and then searching for the corresponding compound instruction in a preset shortcut key mapping table for input control include: comparing the pressure sensitivity value with N preset interval thresholds to determine the target interval in which the pressure sensitivity value falls, and generating a pressure sensitivity level identifier based on the interval number of the target interval; extracting bit segments from the key scan code to obtain the bit segments corresponding to the modifier keys in the key scan code, determining the modifier key value based on the binary value of the corresponding bit segments, concatenating the modifier key value with the pressure sensitivity level identifier to generate a combination index code; and using the combination index code as a lookup key to perform a hash lookup in the preset shortcut key mapping table to obtain the compound instruction corresponding to the combination index code, thereby realizing multimodal input control between the pressure-sensitive pen and the physical keyboard.

[0032] In this embodiment, the pressure sensor of the pressure-sensitive pen outputs a continuous voltage signal when the pen tip touches the screen or when pressure is applied. This voltage signal is converted into a digital pressure-sensitive value by an analog-to-digital converter, which accurately represents the instantaneous pressure applied by the pen tip. To distinguish between different levels of pressure in multimodal input, the pressure-sensitive value is compared with N preset interval thresholds. These N interval thresholds are either evenly divided into the pressure sensor's range or are empirically set gradations. For example, the pressure-sensitive value from 0 to 1023 is divided into three intervals: low pressure, medium pressure, and high pressure, each interval corresponding to a different level of operation sensitivity. By continuously comparing the current pressure-sensitive value with each interval threshold, the target interval into which the value falls can be determined, thus clarifying the pressure level applied by the pen tip. A pressure-sensitive level identifier is generated based on the target interval number. This identifier is a number or coded symbol used in the input management module to represent the current pressure level and serves as one of the conditions for generating subsequent input commands. For example, in drawing software, when a user lightly presses the electromagnetic pen, the pressure sensitivity falls into the low-pressure range, generating a pressure sensitivity level label of "01". When applying greater pressure to draw lines, the pressure sensitivity enters the high-pressure range, generating a pressure sensitivity level label of "03". This label distinguishes different levels of pen pressure. After generating the pressure sensitivity level label, the key scan codes collected from the physical keyboard are processed to extract bit segments. Key scan codes are digitized key status information; each key or modifier key corresponds to a specific binary bit segment, where a value of 1 indicates a key is pressed and 0 indicates a key is not pressed. By extracting the bit segment corresponding to the modifier key in the scan code, the currently pressed modifier key combination information can be obtained, such as Ctrl, Shift, Alt, etc. By reading the binary value, the modifier key value can be obtained, which is used to identify the combination key function or shortcut operation. During the extraction process, each segment needs to be located and read according to the definition table of the scan code. The obtained modifier key value is then concatenated with the previously generated pressure sensitivity level identifier to form a combined index code. This combined index code contains the pressure sensitivity level and keyboard modifier key information, achieving unified encoding for pressure input and keyboard operation. For example, in drawing software, when the Ctrl key is pressed and medium pressure is applied for drawing, the pressure sensitivity level identifier is "02", the Ctrl key segment value is "1", and the combined index code "02_1" is generated. This index code uniquely identifies the input state. The combined index code is used as the lookup key to perform a hash lookup on the preset shortcut key mapping table. The hash function maps the combined index code to the corresponding storage address in the table, thereby quickly obtaining the compound instruction that matches the combined index code. The compound instruction contains operation type, operation parameters, and execution object information. The instruction can represent functions such as color sampling, clearing layers, and adjusting brush size in drawing software. Hash lookup can efficiently match the combined index code with the corresponding instruction, ensuring instant response of pressure pen pressure and keyboard operation in multimodal input.For example, when a user presses the Ctrl key and draws with high pressure, the combined index code "03_1" might correspond to the compound instruction "clear layer". Upon receiving this input, the system sends the instruction to the drawing application via the input management module, deleting the current layer and simultaneously recording the current stroke trajectory for drawing other content. This process enables multimodal input control between the pressure-sensitive pen and the physical keyboard. By fusing pressure levels and keyboard modifier key information into a single code, and utilizing combined index codes for rapid lookup and instruction generation, a single operation can trigger a wealth of compound functions. Simultaneously, it ensures that inputs from different pressure ranges and keyboard states can be accurately distinguished and responded to. This allows for collaborative operation between the pressure-sensitive pen and keyboard in drawing, text editing, or handwritten note-taking scenarios, enabling users to perform multiple operations based on pressure changes and key combinations without switching input devices or manually activating functions.

[0033] In one optional implementation, after determining the pressure sensitivity level identifier based on the numerical range of the pressure sensitivity value, combining the modifier key value in the key scan code with the pressure sensitivity level identifier, and searching for the corresponding compound instruction in the preset shortcut key mapping table for input control, the method further includes: parsing the instruction type field of the compound instruction to generate an instruction type identifier; when the instruction type identifier is text input, obtaining the current cursor coordinates by reading the cursor position information in the display buffer of the tablet device, and redirecting the focus window of the input method manager according to the current cursor coordinates to generate a handwriting input area handle; collecting the continuous trajectory point sequence of the electromagnetic pen on the screen area corresponding to the handwriting input area handle, and performing curve fitting on the continuous trajectory point sequence to generate handwriting contour data; determining the text character corresponding to the handwriting contour data by performing time-normalized distance calculation on the handwriting contour data and the preset character feature library, generating a recognition character code, and inserting the recognition character code at the current cursor coordinates.

[0034] In this embodiment, the compound instruction contains multiple fields to describe the operation type, target object, and execution parameters. The instruction type field identifies the operation category, such as text input, drawing, or system control. By parsing the instruction type field, an instruction type identifier can be generated. This identifier is an internal code used to distinguish the processing logic of different operations. When the instruction type identifier indicates text input, the system needs to map the stylus input to the text input buffer. This requires obtaining the cursor position information in the tablet device's display buffer. The cursor position refers to the two-dimensional coordinates of the cursor in the display area on the screen, including X and Y coordinates, representing the precise position of the character insertion point. After reading the cursor coordinates, the input method manager redirects the focus window to the cursor position based on these coordinates, generating a handwriting input area handle. This handle is a unique identifier used by the operating system or application to identify the input area, essentially establishing a target drawing area for subsequent stylus input. Through the handwriting input area handle, the system can correctly map the electromagnetic pen trajectory to the application's text input area, ensuring the accuracy of handwriting input regardless of the text box's position or scrolling state within the window. For example, in an email editor, when a user places the cursor in the middle of the email body, the handle points to that cursor area, and the handwritten input data is imported into that area. After establishing the handwriting input area handle, the system collects a continuous sequence of trajectory points on the screen corresponding to that area. These trajectory points represent the X and Y positions of the pen tip in the screen coordinate system and their corresponding timestamps, forming a time-continuous handwriting sequence used to describe the trajectory shape of the user's writing or drawing. To generate data usable for text recognition, curve fitting is required on the continuous trajectory point sequence. Curve fitting transforms the discrete point sequence into a smooth curve using interpolation or least squares methods, obtaining handwriting contour data. This data not only contains spatial information about the trajectory but also allows calculation of stroke direction, length, and pressure variation characteristics, providing complete geometric information for subsequent character recognition. For example, when a user writes the letter "E" in the handwriting input box, the handwriting point sequence, after fitting, forms a continuous curve composed of three horizontal lines and one vertical line. The contour data describes the start and end points of the stroke and the stroke order. After generating handwriting outline data, the system compares it with a preset character feature library and determines the corresponding text character through time-warped distance calculation. The character feature library stores standard handwriting outline templates for commonly used text characters, including stroke order, stroke direction, and relative proportions. Time-warped distance calculation involves interpolating and aligning the user's handwriting time series with the feature library template series, comparing the points of the two series one by one, calculating the offset and difference values, and obtaining the best-matching character code, thereby generating a recognition character code. This recognition character code is a standard text encoding that can be recognized by the operating system, such as ASCII or Unicode, and is used to insert it into the text buffer.By inserting the recognized character codes at the current cursor coordinates, the system can display a digital text representation of the user's handwriting at the cursor position, while keeping the cursor position updated to receive subsequent input. For example, when a user writes the word "Hello" in an email body, the trajectory point sequence of the electromagnetic pen is curve-fitted to generate handwriting contour data. This data is then matched with a character feature library through time warping calculations, ultimately generating the recognized character codes "H", "e", "l", "l", and "o", which are then inserted sequentially at the cursor position, allowing the handwritten content to be converted into editable text in real time. The entire process achieves closed-loop control from compound instruction parsing to handwriting trajectory acquisition, curve fitting, character recognition, and text insertion. This ensures that electromagnetic pen input can correctly and continuously generate text in keyboard input or other multimodal input environments, while preserving the spatial and temporal characteristics of the handwriting. This provides accurate data support for subsequent handwriting recognition and multimodal input management, enabling users to seamlessly switch between handwriting and keyboard input on tablet devices.

[0035] In one optional implementation, an application context state vector is generated by collecting the window layout information of the foreground application, the screen area where the cursor is located, and the historical operation sequence. Based on the application context state vector, the current pressure sensitivity value, and the key scan code, the initial input mode is dynamically adjusted to automatically match the optimal input mode in different screen areas or editing modes. Based on the application context state vector, the corresponding operation set is selected from the preset shortcut key mapping table to generate a scene-adaptive composite instruction, and the composite instruction is applied to the current input control.

[0036] In this embodiment, in the multimodal input system, the system first collects the layout information of the foreground application window, obtaining the position and size of the editing area, toolbar, menu bar, and other interactive components contained in each window. The window layout information is represented by two-dimensional coordinates, including the coordinates of the upper left corner, width, and height of each component, as well as the overall screen position and hierarchical order of the window. The screen area where the cursor is located further refines the target point of the user's operation. The cursor position is represented by X and Y coordinates, indicating the screen coordinate range of the character insertion point or drawing start point. The historical operation sequence includes information such as recent user key inputs, pressure-sensitive pen stroke trajectory, pressure sensitivity level, and time interval between key presses and pen strokes. This data is sorted by timestamps to form a continuous operation record, which can reflect the user's operating habits and intentions. The window layout, cursor position, and historical operation sequence are integrated to generate an application context state vector. This vector is a structured data entity containing the spatial information of the current operation object, the editing mode type, the user's operation history, and the input device status, used to describe the overall characteristics of the current input environment. For example, in an email editor, the window layout includes the email body editing area, toolbar, and attachment area. The cursor is located in the middle of the body. The historical operation sequence records the user's most recent Ctrl+C copy and stylus light pressure operations. This information is encoded into a context state vector, providing basic data for subsequent input decisions. Based on the application context state vector, the system combines the current pressure sensitivity value with the keyboard scan code to dynamically adjust the initial input mode. The pressure sensitivity value represents the pressure applied to the screen by the stylus, and the scan code records the keyboard key states and modifier key combinations. By jointly analyzing the pressure sensitivity value, scan code, and context state vector, the system can determine the user's intent and select the most appropriate input mode. For example, in the email editor's body area, when the user lightly presses the pen tip and the keyboard keys are not activated, the system switches the input mode to handwriting enhancement mode, allowing handwritten text to be recognized and inserted in real time. If a shortcut key combination is pressed in the toolbar area while lightly pressing the pen tip, the system can determine to prioritize the shortcut key function, avoiding interference from handwriting input with tool operation. In different areas of the screen or different editing modes, the dynamic adjustment mechanism matches the input mode to the current operation scenario, eliminating the need for manual switching by the user and ensuring input continuity and accuracy. The application context state vector is also used to filter the set of operations in a preset shortcut key mapping table. The shortcut key mapping table contains function operation instructions corresponding to keys or key-pressor combinations in different application scenarios. For example, in the text editing area, low pressure + Ctrl corresponds to "copy text," while in the drawing area, high pressure + Ctrl corresponds to "clear layer." The system filters the set of operations relevant to the current scenario based on the context state vector, generating a scenario-adaptive composite instruction. This composite instruction not only specifies the operation type but also includes the operation target, operation parameters, and trigger conditions, thereby ensuring a high degree of matching between the instruction and the current input environment.For example, when a user presses the Ctrl key and applies medium pressure in the email body area, the system recognizes this area as a text input area through contextual state vectors, mapping the operation to "select text" rather than a graphical operation, ensuring that the input and output meet expectations. The generated scenario-adaptive composite instructions are applied to the current input control, translating the instructions into actual input behaviors, such as text insertion, tool switching, or drawing operations, through the operating system or application programming interface. During control, the system can queue, prioritize, and manage conflicts for instructions to ensure accurate execution of user intent under complex multimodal operations. For instance, in an email editor, when a user continuously handwrites and simultaneously types on the keyboard, the system uses contextual state vectors to determine the cursor position and operation type, prioritizing handwritten text insertion while delaying shortcut key operations until the cursor insertion is complete, ensuring continuous, error-free input that meets user expectations. Through these mechanisms, context awareness enables adaptive matching between input modes and operation scenarios, making the multimodal collaboration between the pressure-sensitive pen and keyboard more efficient, accurate, and intelligent.

[0037] In one optional implementation, when the pressure-sensitive pen tip touches the screen while a physical keyboard key is pressed, a conflict judgment signal is generated based on the current application type, historical input records, and the time interval between the pressure sensitivity value and the key scan code. Based on the conflict judgment signal, the initial input mode is prioritized, and the corresponding compound instruction set is selected for execution to ensure that the system outputs the operation that best matches the user's intent in the event of a multimodal input conflict. When performing conflict handling, the unselected key scan code or pressure-sensitive pen handwriting data is cached or delayed to support the continued processing of subsequent inputs.

[0038] In this embodiment, in a multimodal input environment, when the pressure-sensitive pen tip touches the screen and a physical keyboard key is simultaneously pressed, the system generates a conflict judgment signal by comprehensively analyzing the current application type, historical input records, and the time interval between the pressure sensitivity value and the key scan code. The application type is identified by the software or editing environment running in the foreground, such as a text editor, drawing software, or email client. Each application type has different input priority preferences in multimodal operation. Historical input records include the user's recent key presses, the pressure-sensitive pen's trajectory, the pressure sensitivity level, and the time sequence of key presses. This information reflects user habits, such as a preference for keyboard input in certain areas and a preference for high-pressure pen strokes in drawing areas. The pressure sensitivity value reflects the force applied by the pen tip to the screen, and the key scan code indicates the currently pressed key and the state of the modifier key. By calculating the time interval between the pressure sensitivity change and the key press event, the system can determine whether two inputs overlap or occur consecutively, thereby generating a conflict judgment signal. A conflict detection signal is a logical signal that identifies input conflicts. Its value or state can indicate the type, intensity, or priority of the conflict. For example, in a text editor, a light pen press and a rapid keyboard keystroke simultaneously may indicate a stronger keyboard input intention, while in drawing software, a high-pressure pen press combined with a keyboard shortcut may indicate that the drawing command takes precedence. Based on the conflict detection signal, the system prioritizes the initial input mode and selects the corresponding compound instruction set for execution. The compound instruction set is a set of operations generated based on the generated combination index code and shortcut key mapping table, including the type of executable operation, the target object, and the execution parameters. Through priority adjustment, the system can select the operation that best matches the user's expectation when there is a conflict between the pressure-sensitive pen and keyboard input intentions. For example, in drawing software, if a user presses the Ctrl key while applying a high-pressure pen press, the conflict detection signal indicates that the pen operation takes precedence, so the system executes the "clear layer" operation and delays the execution of the Ctrl shortcut key function. In a text editor, if a light pen press is accompanied by a rapid keystroke event, the conflict detection signal may determine that the keyboard input takes precedence, and the system will postpone handwriting input and execute character insertion or shortcut key operations first. Priority adjustment not only dynamically selects based on the current input device status but also considers application type and historical input habits to ensure that multimodal input outputs the operation most closely matching the user's intent during high-frequency interactions, thereby reducing misoperations and operation interruptions. During conflict resolution, the system caches or delays the application of unselected key scan codes or pressure-sensitive pen handwriting data to support the continuity and integrity of subsequent input. The caching mechanism provides a temporary storage area for key scan codes, recording information about keys not selected during conflict resolution, including key values, key event timestamps, and related modifier key states.The handwriting data cache stores the sequence of points along the pen tip's movement on the screen, along with the corresponding pressure sensitivity values. This allows processing to continue after a conflict is resolved. For example, in a text editor, if a light pressure stroke isn't immediately applied, handwritten text can be recognized and inserted at the cursor position after keyboard input, ensuring continuity. In drawing software, when a high-pressure stroke is delayed, the handwriting trajectory cache allows the system to continue drawing the complete graphic or performing the corresponding shortcut operation after the conflict is resolved. Through caching and delayed application mechanisms, the system maintains a complete record of user actions in multimodal input conflict scenarios, ensuring the continuity and accuracy of input data. This also improves the naturalness of input and the smoothness of interaction, enabling the pressure-sensitive pen and physical keyboard to work collaboratively without interfering with each other in complex operating environments.

[0039] According to the multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard provided in this application, the electromagnetic induction circuit detects the induction signal generated by the electromagnetic pen above the screen of the tablet device, and determines the hovering distance value of the electromagnetic pen based on the intensity change of the induction signal. When the physical keyboard is connected and the hovering distance value is less than a first distance threshold, the initial input mode of the current foreground application is determined according to the application type tag of the current foreground application. The pressure value of the electromagnetic pen tip is collected, the pressure value is converted from analog to digital to generate a pressure-sensitive value, and the level change of each key on the physical keyboard is scanned to generate a key scan code. The initial input mode is switched according to the pressure-sensitive value and the key scan code, and the pressure sensitivity level identifier is determined according to the value range of the pressure-sensitive value. The modifier key value in the key scan code is combined with the pressure sensitivity level identifier, and the corresponding compound instruction is searched from the preset shortcut key mapping table for input control. This application achieves seamless switching between keyboard text input and pressure-sensitive pen hand-drawn input by automatically sensing the pressure-sensitive pen status, keyboard input status, and application scenario. This eliminates the need for users to manually operate the switching button or repeatedly operate between different input modes, thereby maintaining operational continuity and smooth thinking, and significantly improving input efficiency.

[0040] Figure 2 This application provides a multimodal input device based on the collaboration of a pressure-sensitive pen and a physical keyboard. This multimodal input device can be used to implement the multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard in the aforementioned embodiments. Figure 2 As shown, this multimodal input device based on the collaboration of a pressure-sensitive pen and a physical keyboard mainly includes: The detection module 10 is used to detect the sensing signal generated by the electromagnetic pen above the screen of the tablet device through the electromagnetic induction circuit, and to determine the hovering distance value of the electromagnetic pen based on the intensity change of the sensing signal. The determination module 20 is used to determine the initial input mode of the current foreground application based on the application type label of the current foreground application when the physical keyboard is connected and the hovering distance value is less than the first distance threshold. The generation module 30 is used to collect the pressure value of the tip of the electromagnetic pen, perform analog-to-digital conversion on the pressure value, generate pressure sensitivity value, and scan the level change of each key on the physical keyboard to generate key scan code. The control module 40 is used to switch the initial input mode according to the pressure sensitivity value and the key scan code, determine the pressure sensitivity level identifier according to the value range of the pressure sensitivity value, combine the modifier key value in the key scan code with the pressure sensitivity level identifier, and search for the corresponding compound instruction from the preset shortcut key mapping table for input control.

[0041] In one optional implementation of this embodiment, the detection module is specifically used to: transmit an alternating excitation signal through the drive coil in the electromagnetic induction circuit, receive the induced signals fed back to each induction coil by the resonant circuit of the electromagnetic pen, and perform analog-to-digital conversion on the induced signals to generate an induced electromotive force distribution matrix; determine the maximum amplitude and the corresponding coordinate position of the target induction coil by comparing each amplitude in the induced electromotive force distribution matrix, and perform a ratio calculation based on the maximum amplitude and a preset reference amplitude table to generate an amplitude attenuation ratio; determine the offset vector between the coordinate position of the target induction coil and the preset screen center coordinates, and perform logarithmic inversion calculation based on the amplitude attenuation ratio and the exponential parameter in the preset electromagnetic field attenuation model to generate the hovering distance value of the electromagnetic pen.

[0042] In an optional implementation of this embodiment, the control module is further configured to: when the pressure sensitivity value is greater than or equal to the first pressure sensitivity threshold, generate a masked empty scan sequence by masking the key scan code; extract the corresponding shortcut key mapping table from the preset shortcut key mapping library according to the application type label; generate a shortcut command by matching the key scan code with the key value in the shortcut key mapping table; and switch the initial input mode to the handwriting enhancement mode.

[0043] In an optional implementation of this embodiment, the control module is further configured to: when the pressure sensitivity value is less than the second pressure sensitivity threshold and the key scan code is a non-empty sequence, generate a threshold comparison result by comparing the pressure sensitivity value with the second pressure sensitivity threshold; transmit the key scan code according to the threshold comparison result to generate a character input command, and switch the initial input mode to keyboard priority mode.

[0044] In an optional implementation of this embodiment, the control module is further configured to: compare the pressure-sensitive value with N preset interval thresholds to determine the target interval into which the pressure-sensitive value falls, and generate a pressure-sensitive level identifier based on the interval number of the target interval; extract the bit segments from the key scan code to obtain the bit segments corresponding to the modifier keys in the key scan code, determine the modifier key value based on the binary value of the corresponding bit segments, concatenate the modifier key value with the pressure-sensitive level identifier to generate a combined index code; use the combined index code as a lookup key to perform a hash lookup on a preset shortcut key mapping table to obtain the composite instruction corresponding to the combined index code, so as to realize multimodal input control between the pressure-sensitive pen and the physical keyboard.

[0045] In one optional implementation of this embodiment, the control module is further configured to parse the instruction type field of the composite instruction and generate an instruction type identifier; when the instruction type identifier is a text input type, the current cursor coordinates are obtained by reading the cursor position information in the display buffer of the tablet device, and the focus window of the input method manager is redirected according to the current cursor coordinates to generate a handwriting input area handle; the continuous trajectory point sequence of the electromagnetic pen on the screen area corresponding to the handwriting input area handle is collected, and the continuous trajectory point sequence is curve fitted to generate handwriting contour data; the text character corresponding to the handwriting contour data is determined by performing time-normalized distance calculation between the handwriting contour data and the preset character feature library, a recognition character code is generated, and the recognition character code is inserted at the current cursor coordinates.

[0046] According to the multimodal input device based on the collaboration of a pressure-sensitive pen and a physical keyboard provided in this application, the electromagnetic induction circuit detects the induction signal generated by the electromagnetic pen above the screen of the tablet device, and determines the hovering distance value of the electromagnetic pen based on the intensity change of the induction signal. When the physical keyboard is connected and the hovering distance value is less than a first distance threshold, the initial input mode of the current foreground application is determined according to the application type label of the current foreground application. The pressure value of the electromagnetic pen tip is collected, the pressure value is converted from analog to digital to generate a pressure-sensitive value, and the level change of each key on the physical keyboard is scanned to generate a key scan code. The initial input mode is switched according to the pressure-sensitive value and the key scan code, and the pressure sensitivity level is determined according to the value range of the pressure-sensitive value. The modifier key value in the key scan code is combined with the pressure sensitivity level indicator, and the corresponding compound instruction is searched from the preset shortcut key mapping table for input control. This application achieves seamless switching between keyboard text input and pressure-sensitive pen hand-drawn input by automatically sensing the pressure-sensitive pen status, keyboard input status, and application scenario. This eliminates the need for users to manually operate the switching button or repeatedly operate between different input modes, thereby maintaining operational continuity and smooth thinking, and significantly improving input efficiency.

[0047] According to the scheme provided in this application Figure 3An electronic device is provided as an embodiment of this application. This electronic device can be used to implement the multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard as described in the foregoing embodiments, and mainly includes: The system includes a memory 301, a processor 302, and a computer program 303 stored on the memory 301 and executable on the processor 302. The memory 301 and the processor 302 are connected via communication. When the processor 302 executes the computer program 303, it implements the multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard as described in the preceding embodiments. The number of processors can be one or more.

[0048] The memory 301 can be a high-speed random access memory (RAM) or a non-volatile memory, such as a disk storage device. The memory 301 is used to store executable program code, and the processor 302 is coupled to the memory 301.

[0049] Furthermore, embodiments of this application also provide a computer-readable storage medium, which may be disposed in the electronic device described in the above embodiments, and the computer-readable storage medium may be as described above. Figure 3 The memory in the illustrated embodiment.

[0050] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard as described in the preceding embodiments. Furthermore, the computer-readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, external hard drive, read-only memory (ROM), RAM, magnetic disk, or optical disk.

[0051] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0052] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0053] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard, characterized in that, include: The electromagnetic pen generates a sensor signal above the screen of a tablet device by detecting the electromagnetic induction signal through an electromagnetic induction circuit, and the hovering distance of the electromagnetic pen is determined based on the intensity change of the sensor signal. When the physical keyboard is connected and the hovering distance is less than the first distance threshold, the initial input mode of the current foreground application is determined according to the application type label of the current foreground application. The pressure value received by the tip of the electromagnetic pen is collected, the pressure value is converted from analog to digital to generate a pressure sensitivity value, and the level change of each key on the physical keyboard is scanned to generate a key scan code. The initial input mode is switched according to the pressure sensitivity value and the key scan code, and the pressure sensitivity level identifier is determined according to the value range of the pressure sensitivity value. After combining the modifier key value in the key scan code with the pressure sensitivity level identifier, the corresponding compound instruction is searched from the preset shortcut key mapping table for input control.

2. The multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard according to claim 1, characterized in that, The step of detecting the sensing signal generated above the tablet screen by the electromagnetic pen through an electromagnetic induction circuit, and determining the hovering distance value of the electromagnetic pen based on the intensity change of the sensing signal, includes: The electromagnetic pen emits an alternating excitation signal through the drive coil in the electromagnetic induction circuit, receives the induced signals fed back to each induction coil by the resonant circuit of the electromagnetic pen, and performs analog-to-digital conversion on the induced signals to generate an induced electromotive force distribution matrix. By comparing each amplitude in the induced electromotive force distribution matrix, the maximum amplitude and the corresponding coordinate position of the target induction coil are determined, and the amplitude attenuation ratio is generated by performing a ratio calculation based on the maximum amplitude and a preset reference amplitude table. The offset vector between the coordinate position of the target induction coil and the preset center coordinate of the screen is determined, and the hovering distance value of the electromagnetic pen is generated by logarithmic inversion calculation based on the amplitude attenuation ratio and the exponential parameter in the preset electromagnetic field attenuation model.

3. The multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard according to claim 1, characterized in that, The step of switching the initial input mode based on the pressure sensitivity value and the key scan code includes: When the pressure sensitivity value is greater than or equal to the first pressure sensitivity threshold, the button scan code is masked to generate a masked empty scan sequence. Based on the application type tag, the corresponding shortcut key mapping table is extracted from the preset shortcut key mapping library. By matching the key scan code with the key value in the shortcut key mapping table, a shortcut command is generated, and the initial input mode is switched to the handwriting enhancement mode.

4. The multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard according to claim 3, characterized in that, The step of switching the initial input mode based on the pressure sensitivity value and the key scan code further includes: When the pressure sensitivity value is less than the second pressure sensitivity threshold and the key scan code is a non-empty sequence, a threshold comparison result is generated by comparing the pressure sensitivity value with the second pressure sensitivity threshold. Based on the threshold comparison result, the key scan code is transmitted transparently to generate a character input command, and the initial input mode is switched to keyboard priority mode.

5. The multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard according to claim 1, characterized in that, The step of determining the pressure sensitivity level identifier based on the numerical range of the pressure sensitivity value, combining the modifier key value in the key scan code with the pressure sensitivity level identifier, and then searching for the corresponding compound instruction in a preset shortcut key mapping table for input control includes: The pressure sensitivity value is compared with N preset interval thresholds to determine the target interval into which the pressure sensitivity value falls, and a pressure sensitivity level identifier is generated based on the interval number of the target interval. By extracting the bit segments from the key scan code, the bit segments corresponding to the modifier keys in the key scan code are obtained, and the modifier key value is determined according to the binary value of the corresponding bit segment. The modifier key value is then concatenated with the pressure sensitivity level identifier to generate a combined index code. Using the combined index code as the lookup key, a hash lookup is performed on the preset shortcut key mapping table to obtain the composite instruction corresponding to the combined index code, so as to realize multimodal input control between the pressure-sensitive pen and the physical keyboard.

6. The multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard according to claim 5, characterized in that, After the steps of determining the pressure sensitivity level identifier based on the numerical range of the pressure sensitivity value, combining the modifier key value in the key scan code with the pressure sensitivity level identifier, and searching for the corresponding compound instruction in the preset shortcut key mapping table for input control, the method further includes: The instruction type field of the composite instruction is parsed to generate an instruction type identifier; When the instruction type is identified as text input, the cursor position information in the display buffer of the tablet device is read to obtain the current cursor coordinates, and the focus window of the input method manager is redirected according to the current cursor coordinates to generate a handwriting input area handle; Collect a continuous trajectory point sequence of the electromagnetic pen on the screen area corresponding to the handle of the handwriting input area, and perform curve fitting on the continuous trajectory point sequence to generate handwriting contour data; By performing time-normalized distance calculations on the handwriting outline data and a preset character feature library, the text character corresponding to the handwriting outline data is determined, a recognition character code is generated, and the recognition character code is inserted at the current cursor coordinates.

7. A multimodal input device based on the collaboration of a pressure-sensitive pen and a physical keyboard, characterized in that, The multimodal input device based on the collaboration of a pressure-sensitive pen and a physical keyboard is used to implement the multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard as described in claim 1. The multimodal input device based on the collaboration of a pressure-sensitive pen and a physical keyboard includes: The detection module is used to detect the sensing signal generated above the screen of the tablet device by the electromagnetic pen through the electromagnetic induction circuit, and to determine the hovering distance value of the electromagnetic pen based on the intensity change of the sensing signal. The determination module is used to determine the initial input mode of the current foreground application based on the application type label of the current foreground application when the physical keyboard is connected and the hover distance value is less than a first distance threshold. The generation module is used to collect the pressure value received by the tip of the electromagnetic pen, perform analog-to-digital conversion on the pressure value to generate pressure sensitivity value, and scan the level changes of each key on the physical keyboard to generate key scan code. The control module is also used to switch the initial input mode according to the pressure sensitivity value and the key scan code, determine the pressure sensitivity level identifier according to the value range of the pressure sensitivity value, combine the modifier key value in the key scan code with the pressure sensitivity level identifier, and search for the corresponding compound instruction from the preset shortcut key mapping table for input control.

8. The multimodal input device based on the collaboration of a pressure-sensitive pen and a physical keyboard according to claim 7, characterized in that, The multimodal input device based on the collaboration of a pressure-sensitive pen and a physical keyboard also includes: The control module is also used to parse the instruction type field of the composite instruction and generate an instruction type identifier; when the instruction type identifier is a text input type, it reads the cursor position information in the display buffer of the tablet device to obtain the current cursor coordinates, and redirects the focus window of the input method manager according to the current cursor coordinates to generate a handwriting input area handle; it collects the continuous trajectory point sequence of the electromagnetic pen on the screen area corresponding to the handwriting input area handle, and performs curve fitting on the continuous trajectory point sequence to generate handwriting contour data; it calculates the time-normalized distance between the handwriting contour data and the preset character feature library to determine the text character corresponding to the handwriting contour data, generates a recognition character code, and inserts the recognition character code at the current cursor coordinates.

9. An electronic device, characterized in that, Includes memory and processor, of which: The processor is used to execute computer programs stored in the memory; When the processor executes the computer program, it implements the steps in the multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps in the multimodal input method based on the collaboration of a pressure-sensitive pen and a physical keyboard as described in any one of claims 1 to 7.

Images