A touchscreen interaction method, a touchscreen, a steering wheel with an integrated touchscreen, and a racing simulator.
By integrating a touchscreen onto the steering wheel of a racing simulator and dynamically determining gesture priorities using telemetry data, the limited interactive functionality of existing racing simulator instrument panels has been resolved. This has enabled high-precision touch gesture recognition and host interaction, thus improving the user experience.
Patent Information
- Application Number
- CN202611272134.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-20
- Publication Date
- 2026-11-10
AI Technical Summary
The existing touch functionality of racing simulator dashboards is limited to switching display interfaces and cannot handle more interactive tasks. Furthermore, the touch gesture recognition is not deeply coupled with the racing game telemetry data, resulting in inaccurate recognition. Users need to use a mouse and keyboard for interaction, which affects operational safety and immersion.
A touchscreen is integrated into the steering wheel of the racing simulator. Gesture recognition is performed by acquiring the user's touch coordinate set, and gesture priority is dynamically determined by combining the racing game telemetry data. This achieves deep coupling between touch gestures and telemetry data, supports mouse and keyboard simulation in host interaction mode, and has autonomous operation in instrument interaction mode.
It improves the accuracy of touch gesture recognition, enables the instrument panel to function as both a mouse and keyboard, reduces the need for additional equipment, enhances ease of operation and immersion, and reduces complexity and cost.
Smart Images

Figure CN123067780A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of racing simulation technology, and in particular to a touch screen interaction method, a touch screen, a steering wheel with an integrated touch screen, and a racing simulator. Background Technology
[0002] In recent years, racing simulator instrument panels have evolved from simple displays to interactive displays. Current interactive displays add basic touch functionality compared to traditional displays, but this is limited to switching the display interface and cannot handle more complex interactive tasks. To interact with the racing simulator's main unit, users still need a mouse and keyboard, which is extremely inconvenient in a driving environment.
[0003] Some studies have added touch gesture recognition to interactive instruments to replace mouse and keyboard interaction with the host, but the touch gesture recognition process is not deeply coupled with the telemetry data of racing games, resulting in inaccurate touch gesture recognition. Summary of the Invention
[0004] The purpose of this application is to provide an interaction method for a touch screen, a touch screen, a steering wheel with an integrated touch screen, and a racing simulator, which can deeply couple the touch gesture recognition process with the telemetry data of the racing game to improve the accuracy of touch gesture recognition.
[0005] To achieve the above objectives, this application provides the following solution.
[0006] Firstly, this application provides an interaction method for a touchscreen, the interaction method comprising: Obtain the set of touch point coordinates generated by the user touching the touch screen, perform gesture recognition based on the set of touch point coordinates, and obtain the initial touch gesture; The gesture priority of each initial touch gesture is dynamically determined based on the telemetry data of the racing game, and the final target touch gesture is determined from all initial touch gestures based on the gesture priority. Determine the operating mode of the touchscreen based on the display page; If the working mode is host interaction mode, the interaction information corresponding to the target touch gesture is determined according to the preset correspondence between touch gestures and interaction information in host interaction mode, and the interaction information is sent to the host so that the host generates the corresponding system input event; wherein, host interaction mode includes mouse simulation mode and keyboard simulation mode. Mouse simulation mode is used to simulate operations performed on the host through the mouse, and keyboard simulation mode is used to simulate operations performed on the host through the keyboard.
[0007] Optionally, the telemetry data includes game status, current vehicle speed, and current vehicle steering angular velocity; The gesture priority of each initial touch gesture is dynamically determined based on telemetry data from the racing game, specifically including: If the game state indicates that the vehicle is currently in a race, the current vehicle speed is greater than the preset vehicle speed, and the current vehicle steering angular velocity is greater than the preset angular velocity, then the category priority of the first type of touch gesture is determined to be higher than the category priority of other types of touch gestures, so as to obtain the category priority of each initial touch gesture; the first type of touch gesture includes touch gestures related to vehicle control; If the game state indicates that the game is currently in the process of entering the parking area, then the category priority of the second type of touch gesture is determined to be higher than the category priority of other types of touch gestures, so as to obtain the category priority of each initial touch gesture; the second type of touch gesture includes touch gestures related to the strategy configuration; For each initial touch gesture, a comprehensive priority is calculated based on its category priority and its own priority to dynamically determine its gesture priority.
[0008] Optionally, the initial touch gesture has different priorities in different working modes.
[0009] Optionally, the final target touch gesture is determined from all initial touch gestures based on this gesture priority, specifically including: The initial touch gesture with the highest overall priority is selected as the final target touch gesture.
[0010] Optionally, the interaction method of the touch screen further includes: if the working mode is the instrument interaction mode, determining and executing the instrument operation corresponding to the target touch gesture according to the preset correspondence between touch gestures and instrument operations in the instrument interaction mode.
[0011] Optionally, the methods for entering the touchscreen's operating mode include: The system obtains the first click command generated by the user clicking on a control on the touchscreen, and opens the display page according to the first click command to enter the working mode; different display pages correspond to different working modes. Alternatively, obtain the set of touch point coordinates generated by the user touching the touch screen, perform gesture recognition based on the set of touch point coordinates, and if a quick access gesture is recognized, open the display page according to the recognized quick access gesture to enter the working mode; different display pages correspond to different working modes.
[0012] Optionally, the interaction method of the touch screen further includes: Obtain the second click instruction generated by the user clicking the parameter menu item on the touch screen, and adjust the parameters of the touch screen or the host according to the second click instruction.
[0013] Optionally, the interaction method of the touch screen further includes: The system obtains a third click command generated by the user clicking a configuration menu item on the touchscreen, and switches or edits at least one of the instrument configuration file, lighting configuration file, and touch configuration file according to the third click command. The instrument configuration file is used to determine the instrument displayed on the touchscreen, the lighting configuration file is used to determine the lighting displayed on the touchscreen, and the touch configuration file is used to determine the preset correspondence between touch gestures and interactive information in the host interaction mode and the preset correspondence between touch gestures and instrument operations in the instrument interaction mode.
[0014] Optionally, the contact coordinate set includes the sampling time, coordinates, pressure, and contact area of each contact. Gesture recognition is performed based on the set of touch point coordinates to obtain the initial touch gesture, which specifically includes: Based on the set of contact point coordinates, calculate the distance between the starting and ending contact points, and calculate the touch movement speed; the distance between two contact points is calculated based on the coordinates of the two contact points. If the distance is less than the preset distance, the touch movement speed is less than the preset speed, the pressure of each touch point is less than the preset pressure, or the contact area of each touch point is greater than the preset area, then delete the touch point coordinate set. Otherwise, gesture recognition is performed based on the set of touch point coordinates to obtain the initial touch gesture.
[0015] Optionally, when performing gesture recognition based on the set of touch point coordinates to obtain the initial touch gesture, the touch recognition threshold is adjusted based on the telemetry data of the racing game and the touch gesture characteristics.
[0016] Optionally, after determining the working mode of the touch screen, the interaction method of the touch screen further includes: determining whether mouse events and keyboard events are allowed based on the telemetry data of the racing game; if not, then if the working mode is the host interaction mode, then the next set of touch point coordinates is collected.
[0017] Secondly, this application provides a touch screen, which includes a control component that operates using the aforementioned touch screen interaction method.
[0018] Thirdly, this application provides a steering wheel with an integrated touch screen, wherein the steering wheel with the integrated touch screen includes the aforementioned touch screen.
[0019] Fourthly, this application provides a racing simulator, which includes a host and a steering wheel with an integrated touch screen, wherein the host and the control components of the touch screen of the steering wheel with the integrated touch screen are communicatively connected. The control unit is used to encapsulate interactive information, obtain action messages, and transmit the action messages to the host. The host is used to parse the action message to reconstruct the interaction information, and generate the corresponding system input event based on the interaction information to control the input device of the racing simulator; the input device includes at least one of the following: steering wheel, touch screen, seat and pedals.
[0020] Optionally, the interactive information includes: simulated input operation commands; In terms of parsing action messages to reconstruct interaction information, the host is specifically used to: generate candidate simulated input operation instructions based on the action messages, and determine the confidence level of the candidate simulated input operation instructions. If the candidate simulated input operation instructions are in the operation allowed list and their confidence level is greater than the preset confidence level, then the candidate simulated input operation instructions are determined to be the reconstructed interaction information.
[0021] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a touchscreen interaction method, a touchscreen, a steering wheel integrating a touchscreen, and a racing simulator. The instrument panel uses a touchscreen. The method acquires a set of touch point coordinates generated by the user touching the touchscreen, performs gesture recognition based on the touch point coordinate set to obtain initial touch gestures, dynamically determines the gesture priority of each initial touch gesture based on telemetry data from the racing game, and determines the final target touch gesture from all initial touch gestures based on the dynamically determined gesture priority. This deeply couples the touch gesture recognition process with the telemetry data of the racing game, improving the accuracy of touch gesture recognition. Subsequently, the working mode of the touchscreen is determined based on the display page of the touchscreen. In host interaction mode, the system determines the interaction information corresponding to the target touch gesture based on the preset correspondence between touch gestures and interaction information in host interaction mode, and sends the interaction information to the host so that the host generates the corresponding system input event. Host interaction mode includes mouse simulation mode and keyboard simulation mode. Mouse simulation mode is used to simulate operations performed on the host through a mouse, and keyboard simulation mode is used to simulate operations performed on the host through a keyboard. In this way, the system can simulate mouse and keyboard operations on the host through the high-precision target touch gestures it recognizes, so that the instrument panel can better perform the functions of both mouse and keyboard, making it easier to interact with the host and improving convenience. Attached Figure Description
[0022] Figure 1 A flowchart illustrating a touchscreen interaction method provided in an embodiment of this application; Figure 2 This is a detailed flowchart illustrating a touchscreen interaction method provided in one embodiment of this application. Detailed Implementation
[0023] The present application will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0024] A racing simulator is a hardware and software combination that highly replicates the driving experience of a real race car. It is used to simulate driving a race car indoors, allowing you to practice, have fun, and train professionally without having to drive a real car on the track. The peripherals of a racing simulator include a cockpit and a host (which can be a computer). The cockpit is equipped with a display screen (for displaying the game screen), a steering wheel (for controlling the steering of the car in the game screen), pedals (for controlling the speed of the car in the game screen), a seat (for supporting the user), and an instrument panel (for displaying game-related data).
[0025] The aforementioned host computer can be equipped with racing game software and host computer software. The racing game software is used to run the racing game, and the host computer software is used to control input devices such as the display screen, steering wheel, pedals, seat, and instrument panel.
[0026] Currently, to interact with the host computer, such as starting or stopping game software or adjusting parameters in the host computer software, an external mouse and keyboard are required. Furthermore, the inventors have discovered the following drawbacks: First, during the driving process, the user is in the cockpit of the pre-built racing simulator. The cockpit space is small, and the extra mouse and keyboard need to take up some space. They are usually placed on the user's lap, which is not convenient to use. Secondly, during driving, if the user needs to interact with the host to switch software interfaces or adjust software parameters, they must release the steering wheel and use a mouse or keyboard, which greatly affects operational safety. Furthermore, the mouse and keyboard are not part of the operating ecosystem of real racing cars, and their use makes the user feel out of place. At the same time, releasing the steering wheel will interrupt the driving process, reducing the user experience and immersion.
[0027] The inventor attempted to equip the device with a touchpad to replace the mouse and keyboard, and to interact with the host computer using the touchpad. However, since the touchpad was an additional device, it still had the aforementioned drawbacks.
[0028] Furthermore, the inventors discovered that some current instrument clusters on steering wheels are purely display-based, such as third-party displays based on SimHub, which can only load preset instrument UIs (User Interfaces). These instrument clusters are unidirectional output devices. Currently, some instrument clusters are limited-interaction instruments. For example, some manufacturers have added simple screen-switching functions, using touch to switch pages. However, the touch function is limited to switching the instrument cluster's display interface and cannot handle more interactive tasks. Although this partially solves the instrument cluster's expansion needs, it still lacks direct interaction with the host computer and still requires an external mouse and keyboard. Therefore, there is an urgent need for an instrument cluster that not only displays information but also handles interactive tasks such as configuration, track or vehicle status selection, and parameter adjustment.
[0029] The inventors considered that if the touch function could be integrated into the instrument panel on the steering wheel, there would be no need to equip an additional touchpad. Therefore, they designed the instrument panel to be a touch screen and added a touch gesture recognition function to replace the mouse and keyboard for interaction with the host. However, the inventors found that the touch gesture recognition process was not deeply coupled with the telemetry data of the racing game, resulting in inaccurate touch gesture recognition.
[0030] In view of this, this application provides a touch screen interaction method, a touch screen, a steering wheel with an integrated touch screen, and a racing simulator to solve the above problems.
[0031] The touchscreen integrated into the steering wheel in this application has the functions of the aforementioned instrument panel and can also be used to interact with the host computer.
[0032] The following describes the interaction methods for the aforementioned touchscreen. For example... Figure 1 As shown, this embodiment provides an interaction method for a touch screen, which includes the following steps S1 to S5.
[0033] Step S1: Obtain the set of touch point coordinates generated by the user touching the touch screen, and perform gesture recognition based on the set of touch point coordinates to obtain the initial touch gesture.
[0034] Specifically, when a user touches the touchscreen, within a complete sampling period, the touchscreen collects raw data such as the coordinates and press timing of all touch points in real time. All touch points share the same sampling time within the same sampling period, and these data are compiled into a single touch event frame. The coordinate data corresponding to a single touch event frame is single-frame coordinate data.
[0035] Touch events are the data source for gesture recognition. A complete touch process forms a touch event, which includes several frames of touch events. Generally, the specific initial touch gesture is determined by comprehensively judging multiple consecutive frames of touch events. Therefore, the above set of touch point coordinates contains multiple consecutive sets (frames) of single-frame coordinate data.
[0036] Step S2: Dynamically determine the gesture priority of each initial touch gesture based on the telemetry data of the racing game, and determine the final target touch gesture to be used from all initial touch gestures based on the gesture priority.
[0037] Step S3: Determine the working mode of the touch screen based on the display page of the touch screen.
[0038] For example, the operating modes include host interaction mode, instrument interaction mode, and other modes.
[0039] The aforementioned host interaction modes may further include: a mouse emulation mode and a keyboard emulation mode. The mouse emulation mode is used to simulate operations performed on the host via a mouse, and the keyboard emulation mode is used to simulate operations performed on the host via a keyboard.
[0040] The aforementioned instrument interaction mode is used to operate the touchscreen itself, such as switching the instrument display and light display on the touchscreen.
[0041] Both the host interaction mode and the instrument interaction mode mentioned above require touch gestures to complete the interaction process. Other modes complete the interaction process by clicking on the controls on the touch screen instead of touch gestures. For example, you can click on the parameter menu item on the touch screen to adjust the parameters of the touch screen or the host. You can also click on the configuration menu item on the touch screen to switch or edit at least one of the instrument configuration file, the lighting configuration file, and the touch configuration file (the instrument configuration file is used to determine the instrument displayed on the touch screen, the lighting configuration file is used to determine the lighting displayed on the touch screen, and the touch configuration file is used to determine the preset correspondence between touch gestures and interaction information in the host interaction mode and the preset correspondence between touch gestures and instrument operations in the instrument interaction mode).
[0042] For example, the aforementioned instrument profile, lighting profile, and touch profile can be customized by the user to meet personalized customization needs.
[0043] In one example, the working mode corresponds to the display page of the touch screen. Therefore, once the current display page of the touch screen is determined, the current working mode of the touch screen can be determined based on the correspondence between the working mode and the display page.
[0044] Step S4: If the working mode is host interaction mode, determine the interaction information corresponding to the target touch gesture according to the preset correspondence between touch gesture and interaction information in host interaction mode, and send the interaction information to the host so that the host generates the corresponding system input event.
[0045] As mentioned above, the host interaction mode may further include: mouse emulation mode and keyboard emulation mode.
[0046] In one example, the aforementioned interactive information may specifically include at least one of simulated input operation commands (mouse operation commands and keyboard operation commands) and rich events. Rich events will be described in more detail later in this embodiment.
[0047] If the current working mode is specifically mouse simulation mode, in step S4, the target mouse operation corresponding to the target touch gesture can be determined according to the first preset correspondence between touch gestures and mouse operations in mouse simulation mode. Then, the corresponding mouse operation command is sent to the host via the HID protocol (Human Interface Device, a standardized human-computer interaction device protocol used to transmit hardware events such as mouse and keyboard to the host). After parsing the corresponding mouse operation command, the host generates native system mouse events at the system level to complete mouse-related functions such as cursor movement and left-click.
[0048] For example, when the target touch gesture is a single-finger drag gesture (single-finger drag is a continuous gesture, and the touch process will continuously collect multiple frames of touch point data, and each frame will generate a set of single-frame coordinate data), the reference coordinates and real-time coordinates can be extracted based on the single-frame coordinate data of the starting frame and the real-time frame of the single-finger drag gesture, respectively (the starting frame corresponds to the starting touch point with the smallest sampling time in the touch point coordinate set, and the real-time frame corresponds to the ending touch point with the largest sampling time in the touch point coordinate set). The horizontal X-axis displacement increment and the vertical Y-axis displacement increment are calculated based on the reference coordinates and the real-time coordinates. If the target mouse operation is determined to be mouse cursor movement according to the first preset correspondence, the touch screen sends a cursor displacement mouse operation instruction to the host (carrying the horizontal X-axis displacement increment and the vertical Y-axis displacement increment). After the host receives the cursor displacement mouse operation instruction, it can generate a native system mouse event at the system bottom layer according to the two sets of displacement increment values it carries. Taking a game as an example, the game software moves the in-game mouse cursor synchronously based on the native system mouse event.
[0049] For example, when the target touch gesture is a single click, if the target mouse operation is determined to be a left-click based on the first preset correspondence, the touchscreen sends a left-click mouse operation command to the host. The left-click mouse operation command includes: a mouse device identifier, a left-click state identifier, and the touch point coordinates at the moment of click (optional). The left-click state identifier is a marker value used to distinguish the stages of the left-click action, generally divided into two types: a left-click pressed identifier (representing the left button being pressed) and a left-click released identifier (representing the left button being released). Therefore, the touchscreen will send two left-click mouse operation commands to the host sequentially (each containing a left-click pressed identifier and a left-click released identifier), with a very short interval between the two commands to simulate a complete mouse click action. After receiving and parsing each left-click mouse operation command, the host generates the corresponding native system mouse event at the system level. Taking a game as an example, the game software captures the native system mouse event to complete the mouse click response.
[0050] Similarly, if the current working mode is specifically keyboard emulation mode, in step S4, the target keyboard operation corresponding to the target touch gesture can be determined according to the second preset correspondence between touch gestures and keyboard operations in keyboard emulation mode, and the corresponding keyboard operation command is sent to the host via the HID protocol. After the host parses the corresponding keyboard operation command, it generates native system keyboard events at the system level to complete the keyboard-related functions.
[0051] For example, when the target touch gesture is a two-finger tap, if the target keyboard operation is determined to be triggered by the Enter key according to the second preset correspondence, the touch screen sends an Enter key keyboard operation instruction to the host (the Enter key keyboard operation instruction contains the standard Enter key code, and may also include a key press or key release status indicator). After receiving and parsing the Enter key keyboard operation instruction, the host generates the corresponding native system keyboard event at the system level according to the standard Enter key code, and completes operations such as confirmation and pop-up submission based on the native system keyboard event.
[0052] For example, when the target touch gesture is a three-finger long press gesture, if the target keyboard operation is determined to be the ESC exit key operation according to the second preset correspondence, the touch screen sends an ESC key trigger keyboard operation instruction to the host (the ESC key trigger keyboard operation instruction carries an ESC key code identifier). After the host receives and parses the ESC key trigger keyboard operation instruction, it generates the corresponding native system keyboard event at the system bottom layer according to the ESC key code identifier, and performs the action of exiting the current settings interface based on the native system keyboard event.
[0053] Step S5: If the working mode is the instrument interaction mode, determine and execute the instrument operation corresponding to the target touch gesture according to the (third) preset correspondence between touch gestures and instrument operations in the instrument interaction mode.
[0054] For example, if the current working mode is the instrument interaction mode, and the user performs a two-finger left and right swipe gesture on the touch screen, according to the third preset correspondence, if it is determined that the instrument operation corresponding to the two-finger left and right swipe gesture is an instrument page switching operation, then the touch screen performs the instrument page switching operation to update the display content of the touch screen.
[0055] By implementing steps S1 to S5 above, this embodiment can deeply couple the touch gesture recognition process with the telemetry data of the racing game, improve the accuracy of touch gesture recognition, better enable the instrument panel to function as both a mouse and a keyboard, facilitate interaction with the host, improve convenience, and at the same time, better enable the instrument panel to perform corresponding instrument operations.
[0056] In addition, the inventors discovered that existing instrument panels also have the following problems: (1) Limited display functions and lack of deep personalization capabilities: Existing instrument panels cannot provide real-time and intuitive feedback on all vehicle statuses. Vehicle statuses can be parameters that represent the vehicle status in the game, such as current speed and current tire pressure. When the driver wants to know the status of a particular vehicle or several types of vehicles, they need to observe other devices to obtain vehicle status information. Attention often needs to be switched between the instrument panel and other devices, which distracts the user and results in insufficient user immersion. If deep personalization capabilities can be introduced, allowing users to design the vehicle status that the touch screen on the steering wheel can display, i.e., design their own instrument configuration file, this problem is expected to be solved.
[0057] (2) Inability to achieve complex lighting effect linkage with telemetry data: The existing instrument panel lighting system can only set basic speed indicators, which cannot meet the user's need to express various driving states through lighting language. Personalized configuration is limited. If a lighting engine (an editable lighting effect system that can export and bind telemetry data, i.e., a software or firmware system used to define the animation effects of the driving speed lights and side lights, supporting binding telemetry data and exporting configurations) can be introduced, allowing users to design the lights that can be displayed on the touch screen on the steering wheel, i.e., design their own lighting configuration files, this problem is expected to be solved. The above-mentioned lighting engine can be deployed in the host.
[0058] In summary, the solutions for personalized instrument and lighting displays are still incomplete.
[0059] Clearly, no instrument panel currently available can simultaneously achieve personalized instrument display, personalized lighting display, and mouse / keyboard interaction. Users would need to combine multiple devices, increasing cost and complexity, requiring additional space, and resulting in low integration. To address this issue, this embodiment provides a steering wheel with an integrated touchscreen. In steering wheel form, the integrated touchscreen enables mouse and keyboard operation functions for interaction with the host computer. Simultaneously, the touchscreen allows for operation of the instrument panel, such as switching between the displayed instruments and lights (belonging to instrument profiles and light profiles, which can be customized by the user), thus achieving personalized instrument and lighting displays. Therefore, this embodiment can achieve personalized instrument and lighting displays, meeting the needs of personalized customization. Meanwhile, the touchscreen of the racing simulator designed in this embodiment not only has an instrument interaction mode (operating the touchscreen itself, such as operating the instrument display and light display), but can also be switched to a mouse simulation mode (simulating operations performed on the host via a mouse) and a keyboard simulation mode (simulating operations performed on the host via a keyboard). This avoids the need to connect an additional mouse and keyboard, and integrates instrument operation, mouse operation and keyboard operation into one through touch, making operation convenient, reducing costs and complexity, saving space, improving integration, and enhancing immersion.
[0060] In summary, this embodiment incorporates multiple working modes into the touchscreen of the racing simulator: (1) Instrument interaction mode: The touch gesture is converted into the working mode of instrument operation. The touch screen itself can be operated. For example, the instrument page can be switched, the equipment parameters can be adjusted and the equipment preset configuration can be switched.
[0061] The instrument panel page switching example can include quick switching by swiping up, down, left, and right: swiping up and down switches between loaded instrument files (instrument files refer to the instrument layout or page definition file used, including the mapping of display elements and data sources, such as Formula One World Championship F1 or FIA World Championship GT); swiping left and right switches between pages in the same instrument file (main interface, tire page, or engine page), achieving an intuitive switching experience.
[0062] Examples of equipment parameter adjustments include, but are not limited to: adjusting the power parameters, 100-meter acceleration, color, and other equipment parameters of the simulated racing car.
[0063] Examples of device preset configuration switching include, but are not limited to: switching or editing (e.g., modifying) instrument configuration files (including multiple aforementioned instrument files), lighting configuration files, and touch configuration files, etc., and avoiding accidental touches by using explicit gesture switching.
[0064] (2) Mouse simulation mode: Converts touch gestures into native system mouse events working mode, simulates the operation of the host through the mouse, and can simulate complete HID mouse operations, such as moving, clicking, double-clicking, right-clicking, scrolling, etc., and avoids accidental touches by clearly switching gestures.
[0065] (3) Keyboard Simulation Mode: Converts touch gestures into native system keyboard events and simulates operations performed on the host via the keyboard. It can simulate complete HID keyboard operations and realize the functions of the keys on the keyboard, such as ESC, ENTER, etc. It avoids accidental touches by clearly switching gestures.
[0066] The following is a detailed description of a touch screen interaction method provided in this embodiment.
[0067] (a) Preliminary gesture recognition In this embodiment, the set of touch point coordinates generated by the user touching the touch screen is obtained, and gesture recognition is performed based on the set of touch point coordinates to obtain the initial touch gesture.
[0068] The touchscreen in this embodiment can be exemplarily a 4-inch touch-sensitive instrument display device, which may include a touch panel. This touch panel can be a capacitive multi-touch panel, supporting at least 5 concurrent touches. The resolution and sampling rate should meet the requirements for smooth mouse and keyboard interaction; the specific values can be determined by the user. Of course, the touch sensor of this touch panel can be not only capacitive, but also surface acoustic wave (SAW) or infrared. Capacitive sensors are suitable for multi-point and sensitive interaction, while infrared sensors are glove-friendly.
[0069] The touch screen in this embodiment may also include a touch controller, which is responsible for low-level sampling and preliminary filtering, and communicates with the main SoC (System on Chip) or MCU (Microcontroller Unit) in the touch screen via I2C (Inter-Integrated Circuit) or SPI (Serial Peripheral Interface).
[0070] Low-level sampling refers to sampling the user's touch on the touchscreen and collecting touch point data (horizontal coordinate x, vertical coordinate y, touchMajor length, pressure, and contactArea). It can collect multiple touch point data generated by a complete touch process to form a set of touch point coordinates. The sampling rate is recommended to be 100Hz–200Hz, which can be configured by the user.
[0071] Preliminary filtering refers to filtering and transforming the set of touch point coordinates for jitter reduction and coordinate normalization (from physical touch area resolution to screen resolution), converting physical size coordinates into screen pixel coordinates. Filtering can use EMA (Exponential Moving Average, used for signal filtering and jitter suppression) or Kalman filtering.
[0072] At this point, the set of touch point coordinates generated by the user touching the touch screen is obtained, specifically including: obtaining the initial set of touch point coordinates generated by the user touching the touch screen, filtering and transforming the initial set of touch point coordinates to obtain the set of touch point coordinates.
[0073] The main SoC (which can also be replaced by MCU) mentioned above is the control component in the touch screen, which is at least responsible for gesture recognition.
[0074] When performing gesture recognition, for example, the initial touch gesture can be obtained by processing the set of touch point coordinates using a gesture recognition algorithm. The initial touch gesture can be tap (single click), double tap (double click), long press (long press), swipe (swipe), pinch (zoom or pinch), and two-finger scroll, etc. In one example, the specific gesture type can be designed by the user. The gesture recognition algorithm can use a rule engine (threshold or state machine) or a lightweight ML (Machine Learning) model (running on the touch screen or the host) to improve the robustness of recognition.
[0075] The aforementioned set of touch point coordinates includes, for example, a touch point identifier, sampling time, coordinates, pressure, and contact area for each touch point. The touch point identifier uniquely identifies the finger touching the touchscreen. When one or more fingers touch the touchscreen simultaneously, the touch controller assigns an independent touch point identifier to each finger. The touch point identifier remains unchanged throughout the entire touch process of a finger, from pressing to releasing.
[0076] Therefore, when multiple fingers touch simultaneously, multiple independent sets of touch point coordinates are generated, each corresponding to the complete touch trajectory of each finger.
[0077] At this point, gesture recognition is performed based on the set of touch point coordinates to obtain the initial touch gesture, which specifically includes: (1) Calculate the distance between the starting and ending touch points based on the set of touch point coordinates, and calculate the touch movement speed.
[0078] From the set of touch point coordinates, select the touch point with the shortest sampling time (under the same touch point identifier) as the starting touch point and the touch point with the longest sampling time as the ending touch point. Calculate the distance (e.g., Euclidean distance) between the starting and ending touch points based on their coordinates. Then, calculate the absolute value of the difference between the sampling time of the starting and ending touch points to obtain the time difference. Calculate the ratio of the distance to the time difference to obtain the (average) touch movement speed.
[0079] (2) If the distance is less than the preset distance, the touch movement speed is less than the preset speed, the pressure of each touch point is less than the preset pressure, or the contact area of each touch point is greater than the preset area, then delete the touch point coordinate set.
[0080] After deleting the touch point coordinate set, if there is no touch point coordinate set in the current round, gesture recognition will no longer be performed, and the process will wait for the next round of touch point coordinate set collection.
[0081] (3) Otherwise, perform gesture recognition based on the set of touch point coordinates to obtain the initial touch gesture.
[0082] When performing gesture recognition, for example, the gesture recognition algorithm described above can be used to perform gesture recognition based on the distance, touch movement speed, and various touch point data in the touch point coordinate set calculated in step (1) to obtain the initial touch gesture. If there are multiple touch point coordinate sets, the relative position and spacing change features of multiple finger trajectories are combined simultaneously for comprehensive judgment, and finally the matching initial touch gesture is identified.
[0083] (II) Final Gesture Recognition In this embodiment, the gesture priority of each initial touch gesture is dynamically determined based on the telemetry data of the racing game, and the final target touch gesture is determined from all the initial touch gestures based on the dynamically determined gesture priority.
[0084] As mentioned above, gesture recognition can be handled by the main SoC. In this embodiment, the main SoC is also responsible for determining gesture priority and target touch gesture. It dynamically adjusts gesture priority based on the competition context (i.e., telemetry data provided by an external API (Application Programming Interface)) to further determine the target touch gesture. Specifically, it acquires telemetry data (such as game state, current vehicle speed, and current vehicle steering angular velocity), dynamically determines the gesture priority of each initial touch gesture based on the telemetry data, and determines the target touch gesture based on the gesture priority of each initial touch gesture. This enables real-time intelligent switching between high-reliability anti-mistouch and high-precision parameter adjustment in extreme competitive environments.
[0085] That is, in this embodiment, the telemetry data includes the game state, current vehicle speed, and current vehicle steering angular velocity. At this time, the gesture priority of each initial touch gesture is dynamically determined based on the telemetry data of the racing game, specifically including: (1) If the game state indicates that the current vehicle is in a race (e.g., driving on a track), the current vehicle speed is greater than the preset vehicle speed, and the current vehicle steering angular velocity is greater than the preset angular velocity (which can be called the first working condition), then the category priority of the first type of touch gesture is determined to be higher than the category priority of other types of touch gestures, so as to obtain the category priority of each initial touch gesture. The first type of touch gesture includes touch gestures related to vehicle control.
[0086] In one example, for any initial touch gesture, under the first working condition, if it belongs to the first type of touch gesture, its category priority is the same as that of the first type of touch gesture; if it belongs to another type of touch gesture besides the first type, its category priority is the same as that of the other type of touch gesture. It can be designed that under the first working condition, the category priorities of all other types of touch gestures are the same and lower than the category priority of the first type of touch gesture. The category priority can be represented numerically, with a larger value indicating a higher category priority. For example, the category priority of the first type of touch gesture is 2, and the category priority of all other types of touch gestures is 1. This value is merely an example and should not be construed as a limitation of this embodiment.
[0087] Of course, in another example, different priority categories for other types of touch gestures can also be designed. For example, in the first operating condition, the priority category of the first type of touch gesture can be 2, the priority category of the second type of touch gesture can be 1, and the priority category of the third type of touch gesture can be 0.5, etc. Those skilled in the art can design flexibly according to their needs, which will not be elaborated here.
[0088] (2) If the game state indicates that the game is currently in the process of entering the station and parking (which can be called the second working state), then the category priority of the second type of touch gesture is determined to be higher than the category priority of other types of touch gestures, so as to obtain the category priority of each initial touch gesture. The second type of touch gesture includes touch gestures related to the strategy configuration.
[0089] In one example, for each initial touch gesture, if it belongs to the second type of touch gesture, its category priority is the same as that of the second type of touch gesture; if it belongs to another type of touch gesture, its category priority is the same as that of the other type of touch gesture. It can be designed that in the second operating condition, the category priorities of all other types of touch gestures are the same and lower than the category priority of the second type of touch gesture. The category priority can be represented numerically, with a larger value indicating a higher category priority. For example, the category priority of the second type of touch gesture is 2, and the category priority of all other types of touch gestures is 1. This value is merely an example and should not be construed as a limitation of this embodiment.
[0090] Of course, in another example, different priority categories for other types of touch gestures can also be designed. For example, in the second operating condition, the priority category for the second type of touch gesture can be set to 2, the priority category for the first type of touch gesture can be set to 1, and the priority category for the third type of touch gesture can be set to 0.5, etc. Those skilled in the art can design flexibly according to their needs, which will not be elaborated here.
[0091] (3) For each initial touch gesture, a comprehensive priority is calculated based on its category priority and its own priority. The comprehensive priority is used as the gesture priority of the initial touch gesture, thereby dynamically determining its gesture priority.
[0092] In addition to category priority, this embodiment designs its own priority for each touch gesture. The value of the self-priority can range from 0 to 1. This value is only an example and should not be construed as a limitation of this embodiment. In this case, the initial touch gesture itself has a priority (i.e., its own priority), and its final comprehensive priority is the sum of the category priority and the self-priority. For example, in the first working condition, the category priority of the first type of touch gesture is 10, and the category priority of other types of touch gestures is 5. Assuming that an initial touch gesture belongs to the first type of touch gesture, its category priority is 10, and its self-priority is 0.5, then its comprehensive priority is 10.5.
[0093] In one example, the self-priority can be customized by the user. This example introduces category priority and self-priority. Category priority is determined based on telemetry data, while self-priority is determined based on user experience. By combining telemetry data and user experience, gesture priority can be determined more accurately, which helps to improve gesture recognition accuracy.
[0094] In addition, its own priority can also be the default value.
[0095] Optionally, the priority of the same touch gesture can be designed to differ under different working modes. That is, the priority of the same touch gesture will have different values in different working modes. For example, taking a single-click gesture as an example, its priority can be set to 0.3 by default or by the user, 0.8 in mouse simulation mode, and 0.6 in keyboard simulation mode. Assuming that the single-click gesture belongs to the first type of touch gesture, its category priority is 10 in the first working condition. Therefore, the overall priority of the single-click gesture is 10 + 0.3 = 10.3 in the instrument interaction mode, 10 + 0.8 = 10.8 in the mouse simulation mode, and 10 + 0.6 = 10.6 in the keyboard simulation mode. In the second working condition, the category priority of the single-click gesture is 5. Therefore, the overall priority is 5 + 0.3 = 5.3 in the instrument interaction mode, 5 + 0.8 = 5.8 in the mouse simulation mode, and 5 + 0.6 = 5.6 in the keyboard simulation mode.
[0096] In this design, the initial touch gesture has different priorities in different working modes.
[0097] It should be noted that if the first condition (i.e., the game state indicating that the vehicle is currently in a race, the current vehicle speed is greater than the preset speed, or the current vehicle steering angular velocity is greater than the preset angular velocity) and the second condition (i.e., the game state indicating that the vehicle is currently entering a parking station) are not met, then the category priority of each type of touch gesture is the same, and the overall priority is determined solely by its own priority.
[0098] In other embodiments of this application, touch features may be further incorporated to determine category priority. This will be described below.
[0099] (1) During the game When a player is cornering at high speed or accelerating to full throttle on a straightaway, with their left hand near the steering wheel paddles and their right hand resting naturally on the edge of the touchscreen, even a slight touch can easily be interpreted as a swipe to return or open settings. If all touch gestures are given the same priority and treated equally, these slight touches can lead to accidental menu access in high-speed racing scenarios. Accidental menu access can disrupt immersion and even affect performance. Therefore, this embodiment introduces dynamic prioritization (i.e., category priority) based on the racing game.
[0100] The dimensions considered for dynamic priority in the competition are shown in Table 1 below.
[0101] Table 1 Dimensions Considered in the Competition
[0102] This embodiment uses a sliding window to count the number of complete touch events on the touchscreen (which can be used to determine the number of touch events in the last second). For each touch event within the last second, it is determined whether the duration of the touch event is less than a preset duration. If so, the touch event is a short-term contact event, and the number of short-term contact events can be counted. If the number of short-term contact events is greater than a preset number, it is determined that most touch events are short-term contact events; otherwise, it is determined that most touch events are not short-term contact events. Here, a touch event refers to a complete touch process.
[0103] At this point, whether touch events are mostly short-term contact events is used as a touch feature, and the priority of various touch gestures is dynamically determined by combining telemetry data and other factors. For example, if the game state indicates that the player is currently in a race, the current vehicle speed is greater than the preset speed (which can be 80% of the maximum speed or designed by the user), the current vehicle steering angular velocity is greater than the preset angular velocity, and the touch events are mostly short-lived events, then the category priority of the first type of touch gesture is determined to be higher than the category priority of other types of touch gestures. However, if the game state indicates that the player is currently in a race but does not meet any of the conditions of "current vehicle speed greater than the preset speed," "current vehicle steering angular velocity greater than the preset angular velocity," or "touch events are mostly short-lived events," then the category priority of the first type of touch gesture is determined to be equal to the category priority of other types of touch gestures, so as to obtain the category priority of each initial touch gesture. The first type of touch gesture includes touch gestures related to vehicle control, such as touch gestures corresponding to clicking the confirmation button and long-pressing to call up voice commands. Other types of touch gestures are touch gestures other than the first type of touch gestures, such as touch gestures related to strategy configuration (i.e., the second type of touch gestures) and menu gestures, such as touch gestures corresponding to swiping to call up the sidebar and two-finger zooming of the map.
[0104] Through the above design, when driving at high speed, the system understands that the player is not trying to adjust settings but wants to concentrate on driving. Therefore, it automatically reduces the priority of menu-type gestures to prevent accidental menu touches and prioritizes vehicle control gestures such as gear shifting, greatly reducing accidental touches.
[0105] In other embodiments of this application, the interaction method of all the above embodiments may further include the following steps: If the game status indicates that the user is currently in a match, the interactive controls on the touchscreen will be placed outside the accidental touch zones (i.e., interactive controls will be prohibited from being placed within these zones). The accidental touch zones are sub-areas of the touchscreen that are prone to accidental touches. By prohibiting interactive controls from being placed within these zones, even if a user accidentally touches the area, no corresponding operation of the interactive controls (such as interface transitions) will be triggered, thus preventing accidental touches from interfering with driving operations. These interactive controls include clickable graphical interface elements such as buttons, sliders, menu items, and icons.
[0106] The following are some examples of ways to prevent interactive controls from being placed in areas prone to accidental touch: Scenario 1: The above interactive controls are a self-contained UI for the touchscreen: If the content displayed on the touchscreen is rendered directly by the touchscreen's own control components, then "disallowing interactive controls in areas prone to accidental touches" only needs to be handled in the UI rendering logic on the touchscreen side: The touchscreen's control components acquire telemetry data (such as game status) from the host computer. When the telemetry data determines that the game is currently in progress, the coordinate range of the area prone to accidental touches is marked as a non-interactive zone during UI rendering, and no interactive controls such as buttons, sliders, or menu items are rendered within this coordinate range. When the game is no longer in progress, the rendering of interactive controls in the area prone to accidental touches is restored.
[0107] Scenario 2: Interactive controls are provided by the host application: If the content displayed on the touchscreen is controlled by the host computer software (SimPro, which is Simmagic's instrument management and game adaptation platform, refers to the host computer software that interacts with the touchscreen, and can be used to refer to the host computer software of similar devices) – for example, if the host computer software transmits the UI interface to the touchscreen display via video streaming or secondary screen extension – then the implementation of "disallowing interactive controls in areas prone to accidental touches" requires host computer involvement. This can be achieved, for example, in the following way: The host computer software dynamically adjusts the UI layout of the touchscreen based on telemetry data (match status). When it is determined that the current state is a match, the host computer software actively hides or removes the interactive controls corresponding to the coordinate range of the easily accidentally touched area when generating the UI interface, so as to avoid any interactive controls appearing in the easily accidentally touched area.
[0108] The aforementioned areas prone to accidental touch can be identified in at least the following ways: Method 1: Factory default configuration: When a touchscreen leaves the factory, the coordinates of one or more default accidental touch zones are preset based on the grip posture and natural finger placement in typical driving scenarios, and stored in the touchscreen's control unit. These default accidental touch zones are usually located at the screen edge or the area where fingers naturally rest, serving as the initial anti-accidental touch configuration.
[0109] Method 2: Dynamic notification from the host: The host computer software in the host computer sends the coordinates of the areas prone to accidental touches to the touchscreen via a communication protocol, based on the currently running game scene or UI layout. The distribution of these accidental touch areas may differ across different games or UI interfaces, and the host computer dynamically instructs the touchscreen to adjust the area configuration. For example, if in a game's current UI interface, interactive controls are mainly distributed in the center and right side of the screen (such as tire setting controls and TC (Traction Control) adjustment controls), while the left edge of the screen only displays speed assist information, then the host computer instructs the touchscreen to identify the left edge of the screen as the area prone to accidental touches.
[0110] Method 3: Determined based on user feedback: Collect information on areas prone to accidental touches from players (users) after the game ends, and store these areas as personalized accidental touch zones in a local configuration file for use in subsequent games.
[0111] In one example, the touch screen or host computer software / game software can divide the touch screen into several grid areas (e.g., 3×3 or 4×4 grids). Each grid has fixed left and right, and top and bottom pixel boundaries. A mapping table between grid numbers and boundary pixel coordinates is pre-stored on the touch screen or the host computer.
[0112] The feedback interface displays a grid area diagram to the user and provides a prompt to identify areas prone to accidental touches. For example, a message may pop up saying, "Please click on the grid area you think is most likely to be accidentally touched in this game." After the user clicks on one or more grids and confirms, the touchscreen or host computer software / game software records the selected grid number. Based on the selected grid number, the system looks up the corresponding left, right, top, and bottom pixel boundaries in the mapping table and combines them to form the coordinate range of the complete area prone to accidental touches. This example can be called the grid point selection mode.
[0113] In another example, a screen diagram can be displayed in the touchscreen's feedback interface, allowing the user to directly circle or draw areas prone to accidental touches on the diagram with their finger. The touchscreen or host computer / game software receives the user-defined coordinate range and maps it to actual screen coordinates, which serve as the areas prone to accidental touches. This example can be referred to as a free selection mode.
[0114] In another example, grid selection mode and free selection mode can be used in combination: the touchscreen defaults to grid selection mode, in which a simplified grid selection interface (such as a 3×3 or 4×4 grid) is displayed in the feedback interface by default, and the user can click to select areas that are prone to accidental touches (see the previous introduction). The feedback interface also displays "Manual Selection" or "Precise Setting" buttons. If the user needs a more precise area, they can click the "Manual Selection" or "Precise Setting" button to switch to free selection mode. The subsequent process will not be described in detail.
[0115] In other embodiments of this application, to reduce accidental touches, after obtaining the set of touch point coordinates and before performing gesture recognition, all the above embodiments may also perform the following operations: The system iterates through all touch point coordinates within the set. If all touch point coordinates fall within the aforementioned area prone to accidental touches, the set is deleted, and subsequent gesture recognition is no longer performed. In this embodiment, the probability of accidental touches is extremely high when all touch point coordinates are within the aforementioned area prone to accidental touches; therefore, the corresponding set of touch point coordinates is deleted.
[0116] In other embodiments of this application, if the touchscreen hardware supports local sampling frequency adjustment, the hardware sampling frequency of the area prone to accidental touches can be lowered to reduce jitter in the sampling data. For example, the hardware sampling frequency of the area prone to accidental touches can be reduced to 50Hz, while the hardware sampling frequency of other areas can be relatively higher, such as 120Hz. Those skilled in the art can flexibly design the hardware sampling frequency values for each area within the hardware capabilities, which will not be elaborated here.
[0117] If the touchscreen hardware does not support this, the sampling frequency of areas prone to accidental touches can be reduced in the following two ways: Method 1, Local Sampling Compression: The touch screen hardware samples at a fixed frequency (e.g., 120Hz) to obtain a set of touch point coordinates. The touch point coordinates located in the area prone to accidental touch are deleted according to a first preset deletion ratio a, so that the high-density touch point coordinate data in the area prone to accidental touch is compressed, reducing the amount of invalid data caused by edge jitter.
[0118] For example, assuming a = 2 / 3, the number of contact coordinates in a certain set of contact coordinates located in the area prone to accidental touch is 15, and the total number of contact coordinates in this set of contact coordinates is 100. Then, if 10 of the above 15 contact coordinates are deleted and 5 are kept, the total number of contact coordinates in this set of contact coordinates will be 90.
[0119] Specifically, when deleting, equal intervals can be used to achieve deletion: taking the deletion of 10 out of 15 touch point coordinates as an example, the retention interval (3 in this example) is used to retain them, that is, retain the 1st, delete the 2nd to 3rd, retain the 4th (the difference between the 4th and the 1st is equal to the retention interval), delete the 5th to 6th, and so on, and finally delete 10 evenly, leaving 5.
[0120] The retention interval can be obtained as follows: 1 minus the first preset deletion ratio 'a', then take the reciprocal. If the reciprocal is not a positive integer, it can be rounded down or up to obtain the retention interval. If the reciprocal is a positive integer, it is directly used as the retention interval. For example, if a = 2 / 3, 1 - 2 / 3 = 1 / 3, and taking the reciprocal of 1 / 3 gives 3, which is used as the retention interval.
[0121] Method 2, Global Sampling Compression: The touchscreen hardware samples at a fixed frequency (e.g., 120Hz) to obtain a set of touch coordinates. It calculates the proportion of touch coordinates located in areas prone to accidental touches to the total number of touch coordinates in the set. If the proportion exceeds a threshold, touch coordinates are deleted from the set according to a second preset deletion ratio N, retaining only partially sampled coordinates. If the proportion does not exceed the threshold, all touch coordinates in the set are retained. This method uniformly downsamples the entire touch trajectory, effectively eliminating a large number of fragmented, jittery sampling points and smoothing out minor displacement features caused by hand movements. This reduces the probability of accidental touches being recognized as valid touch gestures from the data source.
[0122] For example, assuming the percentage threshold is 10%, the total number of touch coordinates in a certain touch coordinate set is 100, and the second preset deletion ratio is assumed to be 80%, then 80 touch coordinates in the touch coordinate set will be deleted as a whole, and 20 touch coordinates will be retained.
[0123] For details on how to delete it, please refer to the aforementioned records; they will not be repeated here.
[0124] The aforementioned percentage threshold, the first preset deletion ratio a, and the second preset deletion ratio N can be flexibly designed, and will not be elaborated here.
[0125] Alternatively, they can be used in combination, for example: The touchscreen hardware samples at a fixed frequency (e.g., 120Hz) to obtain a set of touch coordinates. It iterates through all touch coordinates within the set. If the proportion of touch coordinates located in the accidental touch zone falls within a first range, the entire set of touch coordinates is deleted. If the proportion falls within a second range, touch coordinates are deleted from the set according to the second preset deletion ratio N, retaining only partially sampled coordinates. If the proportion falls within a third range, touch coordinates located in the accidental touch zone are deleted according to the first preset deletion ratio a. If the proportion falls within a fourth range, the complete set of touch coordinates is retained.
[0126] The first range can be exemplarily [90%, 100%], the second range can be exemplarily [50%, 90%], the third range can be exemplarily [10%, 50%], and the fourth range can be exemplarily [0%, 10%]. Those skilled in the art can flexibly design the upper and lower limits of the first to fourth ranges, which will not be elaborated here.
[0127] That is, before obtaining the initial touch gesture by performing gesture recognition based on the set of touch point coordinates, if the proportion of touch point coordinates located in the area prone to accidental touch is greater than 90%, the set of touch point coordinates is directly deleted. If it is between 50% and 90%, the above-mentioned global sampling compression is performed. If it is between 10% and 50%, the above-mentioned local sampling compression is performed. If it is less than 10%, the complete set of touch point coordinates is retained.
[0128] The above measures can filter out invalid touches in areas prone to accidental touches before initial touch gesture recognition: after deleting some touch point coordinates located in these areas, the amount of data remaining may not be sufficient to meet the minimum frame rate requirements for gesture recognition (e.g., swiping requires multiple consecutive frames, long press requires multiple continuous frames), thus preventing the corresponding touch process from being recognized as any valid touch gesture and avoiding accidental triggering. Simultaneously, it can reduce the system's processing load to some extent. This is mainly reflected in the reduced data volume of the touch point coordinate set, which correspondingly reduces the amount of data processed by the gesture recognition algorithm, alleviating the computational burden on the touchscreen control components. Furthermore, it offers flexible adaptation without requiring hardware modification or updates: this solution can be implemented at the software level through post-processing based on the touch point coordinate set, without requiring the touch controller hardware to support region-specific sampling, resulting in stronger adaptability.
[0129] (2) Parking at the station Before the official match, when players enter the pit stop phase, they need to quickly adjust their tire selection, fuel level, and pit stop timing strategy via the touchscreen, while possibly still using steering wheel buttons for auxiliary confirmation. If all touch gestures have the same priority and are treated equally, players will need to frequently drag sliders and tap tire icons during pit stop scenarios. If the dynamic priority from the match is still applied, it will result in: sliders being difficult to drag, tapping being unresponsive, and the operation rhythm being interrupted. Therefore, this embodiment introduces dynamic priority (i.e., category priority) for pit stop scenarios.
[0130] The dimensions considered in dynamic priority during station entry parking are shown in Table 2 below.
[0131] Table 2 Dimensions Considered in Station Parking
[0132] This embodiment can use a sliding window to statistically analyze whether the touch trajectory is an approximately horizontal or vertical drag trajectory (typical slider operation) to determine the touch trajectory pattern. After determining the set of touch point coordinates, it is possible to determine whether it is an approximately horizontal or vertical drag trajectory based on the coordinates of each touch point, which will not be elaborated on in this embodiment.
[0133] If the game state indicates that the game is currently in the process of entering a parking area and the touch trajectory is a drag trajectory, then the category priority of the second type of touch gesture is determined to be higher than the category priority of other types of touch gestures. If the game state indicates that the game is currently in the process of entering a parking area but does not meet the condition of "the touch trajectory is a drag trajectory", then the category priority of the second type of touch gesture is determined to be equal to the category priority of other types of touch gestures, so as to obtain the category priority of each initial touch gesture. The second type of touch gesture includes touch gestures related to strategy configuration, such as touch gestures corresponding to slider dragging and left and right swiping to switch tire types. Other types of touch gestures are touch gestures other than the second type of touch gestures, such as touch gestures corresponding to edge swipe back.
[0134] If the player suddenly switches back to the game screen, the dynamic priority configuration in the game will be quickly restored.
[0135] Through the above design, during the crucial stage of entering the station, the system will automatically determine that the player is finely adjusting their strategy. Therefore, it will concentrate gesture recognition resources on high-value operations such as sliders and option switching, and prioritize responding to strategy configuration gestures, making the interaction more responsive and efficient.
[0136] After dynamically determining the gesture priority (overall priority), the target touch gesture can be further determined.
[0137] In other embodiments of this application, the above-mentioned determination of the final target touch gesture from all initial touch gestures based on dynamically determined gesture priorities specifically includes: The initial touch gesture with the highest overall priority is selected as the final target touch gesture.
[0138] Optionally, the main SoC design in this embodiment employs a gesture mis-touch suppression strategy: dynamically adjusting the touch recognition threshold based on telemetry data and touch gesture characteristics to significantly reduce the probability of mis-touches and prevent screen changes caused by accidental touches during the competition.
[0139] At this point, when performing gesture recognition based on the set of touch point coordinates to obtain the initial touch gesture, the touch recognition threshold can be adjusted first based on the telemetry data and touch gesture characteristics of the racing game. Then, based on the current (adjusted) touch recognition threshold, gesture recognition is performed based on the set of touch point coordinates to obtain the initial touch gesture, thereby achieving physical image stabilization.
[0140] Specifically, the following process can be executed to identify the initial touch gesture: First, detect whether the current touch gesture features are within a threshold range. Touch gesture features can be one or more of any reasonable parameters such as contact area, swipe direction, duration, touch pressure, and movement speed. The threshold range also corresponds to one or more of any reasonable threshold range such as area threshold, direction threshold, time threshold, pressure threshold, and speed threshold. If the touch gesture features are determined not to fall within the threshold range, it indicates that the current user touch operation is a mis-touch or invalid operation, and the currently acquired touch coordinate set is deleted. If the touch gesture features are determined to be within the threshold range, it indicates that the current user touch operation is a valid operation. Next, telemetry data is acquired. Telemetry data can be one or more of any reasonable state parameters such as game state (in a match, during a parking stop), current vehicle speed, and current vehicle steering angle. Based on the acquired telemetry data, the threshold range is dynamically adjusted (equivalent to adjusting the touch recognition threshold). For example, in response to the game state being in a match, the current vehicle speed being greater than a preset speed threshold, or the current vehicle steering angle being greater than a preset angle, the threshold range is increased. Then, based on the current threshold range, gesture recognition is performed according to the touch point coordinate set to obtain the initial touch gesture.
[0141] (III) Entering Working Mode and Gesture Recognition In this embodiment, the working mode of the touch screen is determined based on the display page of the touch screen.
[0142] In this embodiment, clicking different controls on the touchscreen enters different working modes, or recognizing different quick access gestures, such as swiping down with three fingers or long-pressing, enters different working modes. At this time, the working mode corresponds one-to-one with the display page. Based on the current display page of the touchscreen, the current working mode of the touchscreen can be determined.
[0143] As an example, methods for entering the touchscreen's operating mode include: The system obtains the first click command generated by the user clicking on the control of the touch screen, opens the display page according to the first click command, and enters the working mode. Different display pages correspond to different working modes.
[0144] This example corresponds to a click interaction mode (which belongs to the other modes mentioned above) and can be applied to native click states, such as "direct UI operation state" and "menu navigation state". In native click state, the touchscreen does not simulate a mouse or keyboard, nor does it respond to complex page-turning gestures; it only supports basic click control interactions and does not perform gesture recognition. The first click instruction mentioned above is an example of an underlying single-point touch signal.
[0145] For example, after powering on, the touchscreen interface displays button controls (mode menu items) for settings, tire pressure, and fuel level. Clicking the corresponding button will bring up the corresponding display page. For instance, if the user clicks the "Settings" button, a settings interface will pop up, which will correspond to a certain working mode, such as instrument interaction mode, mouse simulation mode, or keyboard simulation mode.
[0146] As another example, methods for entering the touchscreen's operating mode include: The system acquires a set of touch point coordinates generated by the user's touchscreen. Gesture recognition is then performed based on this set. If a quick access gesture is recognized, the system opens the corresponding display page to enter the working mode. Different display pages correspond to different working modes. The type of quick access gesture can be customized by the user, with different gestures corresponding to different display pages.
[0147] When entering different working modes, i.e. when switching modes, the mode switching strategy used in this application may be control switching (i.e., the aforementioned click control), touch switching (i.e., the aforementioned recognition of quick entry gestures), or automatic switching (i.e., by operating the physical buttons on the steering wheel).
[0148] After determining the current working mode, the target touch gesture can be mapped to different operations based on the current working mode. Specifically, the main SoC can also be responsible for the mapping logic, which can be called the mapping manager. It maps the recognized target touch gesture to: interactive information (target mouse operation in mouse simulation mode and target keyboard operation in keyboard simulation mode), instrument switching operation (instrument interaction mode), etc. The host can further generate corresponding system input events based on the interactive information.
[0149] This embodiment can realize low-latency touch pipeline and filtering. By combining device-side sampling, edge filtering, local prediction (coordinate transformation + gesture recognition) and host-side minimization mapping, a low-latency and smooth pointer movement experience can be achieved.
[0150] Optionally, after determining the working mode of the touch screen, the interaction method of the touch screen in this embodiment further includes: determining whether mouse events and keyboard events are allowed based on the telemetry data of the racing game; if not, then if the working mode is the host interaction mode, then the next set of touch point coordinates is collected.
[0151] Specifically, determining whether mouse and keyboard events are allowed based on telemetry data from the racing game can include: acquiring simulated racing car state parameters from the telemetry data, such as one or more reasonable state parameters like vehicle speed and steering angle; detecting whether the simulated racing car state parameters are within a threshold range, specifically including one or more of a vehicle speed threshold range and a steering angle threshold range; if the simulated racing car state parameters are within this threshold range (e.g., vehicle speed and steering angle are within the vehicle speed threshold range and steering angle threshold range, respectively), then the host interaction mode is disabled, and mouse and keyboard events are not allowed; otherwise, mouse and keyboard events are allowed.
[0152] (iv) Host Interaction In this embodiment, if the working mode is host interaction mode, the interaction information corresponding to the target touch gesture is determined according to the preset correspondence between touch gestures and interaction information in host interaction mode, and the interaction information is sent to the host so that the host generates the corresponding system input event. The host interaction mode includes mouse simulation mode and keyboard simulation mode. Mouse simulation mode is used to simulate the operation performed on the host through the mouse, and keyboard simulation mode is used to simulate the operation performed on the host through the keyboard.
[0153] In host interaction mode, the aforementioned interactive information may include simulated input operation commands. In this case, the preset correspondence between touch gestures and interactive information in host interaction mode is as follows: the preset correspondence between touch gestures and simulated input operations in host interaction mode includes a first preset correspondence between touch gestures and mouse operations in mouse simulation mode and a second preset correspondence between touch gestures and keyboard operations in keyboard simulation mode. Specifically, the first preset correspondence between touch gestures and mouse operations in mouse simulation mode includes, but is not limited to: a single short press corresponds to a left mouse button click; a single double press corresponds to a left mouse button double press or opening a shortcut menu; a single long press (duration > 400ms) corresponds to a right mouse button click or entering context or edit mode; a single finger swipe corresponds to pointer movement; two fingers vertical movement corresponds to mouse wheel scrolling; two fingers tap corresponds to a middle mouse button click or switching modes; two fingers long press corresponds to entering or exiting light editing mode, or corresponding to a mode lock switch; gestures with three or more fingers can be customized as shortcut operations (e.g., one-click save configuration, return to the main interface), etc.
[0154] Based on the preset correspondence between touch gestures and simulated input operations in the aforementioned host interaction mode, after determining the simulated input operation corresponding to the target touch gesture, the touch screen generates a simulated input operation instruction (mouse operation instruction or keyboard operation instruction), encapsulates the simulated input operation instruction to obtain an action message, and transmits the action message to the host to send the simulated input operation instruction to the host.
[0155] The host is used to parse the action message to reconstruct the simulated input operation command, and generate the corresponding system input event according to the simulated input operation command to control the input device of the racing simulator. The input device includes at least one of the following: steering wheel, touch screen, seat and pedal.
[0156] The host computer contains host computer software and game software. If the interaction information is a simulated input operation command, the operating system in the host computer parses the action message to reconstruct the simulated input operation command. Based on this simulated input operation command, it generates a corresponding system input event to control the input device of the racing simulator. Specifically, the operating system pushes the system input event (native system mouse event or native system keyboard event) to the host computer software or game software. The host computer software or game software then triggers a corresponding software response behavior (such as interface switching operation, parameter adjustment operation, or command execution operation) based on the system input event. The host computer software or game software then issues corresponding control commands to the input device according to the software response behavior to control the input device of the racing simulator.
[0157] As mentioned above, interactive information can include rich events. Rich events, also known as extended events, are user-defined commands. These commands can be generated by pressing buttons on the steering wheel or by inputting relevant information into the touchscreen. Each user-defined command carries relevant parameters. After receiving the command, the host computer can perform corresponding operations, such as opening a browser, adjusting the volume, or shutting down. Some user-defined commands can also be converted into system input events (native system mouse events or native system keyboard events) and pushed to the host computer software or game software. The host computer software or game software then triggers corresponding software response behaviors based on these system input events (such as interface switching, parameter adjustment, or command execution). Based on these software response behaviors, the host computer software or game software sends corresponding control commands to the input devices to control the racing simulator's input devices.
[0158] In other embodiments of this application, the touch screen in all the above embodiments may also include an external interface, which may be at least one of a USB interface (such as USB-C), a Bluetooth low-latency channel (i.e., a Bluetooth interface), and a network interface (such as Wi-Fi). The touch screen communicates with the host through the external interface.
[0159] The default communication protocol between the touchscreen and the host is standard USB HID, which can be replaced with Bluetooth HID.
[0160] At this time, when transmitting the standard event of simulated input operation command, the touch screen uses an external interface and communication protocol to transmit the standard event to the host. The host computer software discards the standard event (or the standard event cannot reach the host computer software due to interception by the operating system). The operating system (such as Windows) in the host executes the standard event, that is, performs the corresponding mouse and keyboard operations, thereby operating the host computer software and game software, and further controlling the input device.
[0161] When transmitting non-standard events, such as rich events, the touchscreen uses external interfaces and communication protocols to transmit the non-standard events to the host. The operating system in the host discards the non-standard events and instead, the host computer software in the host parses and executes the non-standard events. If the non-standard events are parsed into standard events, the parsed standard events can be further sent to the operating system in the host. The operating system in the host executes the parsed standard events, that is, performs the corresponding mouse and keyboard operations, thereby operating the host computer software and game software, and further controlling the input devices.
[0162] The above design ensures plug-and-play functionality by default using the standard USB HID mouse protocol, while also supporting the sending of rich events to the host computer software's proprietary interface for deeper interaction.
[0163] In one example, when the touchscreen is connected to the host via USB, the host requests a report descriptor from the touchscreen. This report descriptor internally defines multiple reports, each distinguished by a report ID. Standard events and non-standard events (rich events) correspond to different report IDs. The report descriptor also describes the report's data structure; for example, it describes the data structure for a standard mouse report and the data structure for a non-standard event. The report ID can, for example, be located in the first byte of the data packet.
[0164] Among them, the reports corresponding to standard events can be recognized by the host operating system's HID driver. After the touch screen reports standard event data packets, the operating system directly parses and generates the corresponding system input events, realizing driverless operation. The host computer software listens to the reported data packets through the HID raw interface, identifies the report ID in the data packet, captures the raw data packets whose report IDs represent non-standard events, and independently completes the parsing and subsequent operations.
[0165] In terms of parsing action messages to reconstruct interaction information, the host is specifically used to: generate candidate simulated input operation instructions based on the action messages, and determine the confidence level of the candidate simulated input operation instructions. If the candidate simulated input operation instruction is in the operation allowed list and its confidence level is greater than the preset confidence level, then the candidate simulated input operation instruction is determined to be the reconstructed interaction information.
[0166] The confidence level is determined based on the host's execution environment. For a candidate simulated input operation instruction, the host's execution environment is the set of software and hardware contexts in which the candidate simulated input operation instruction is executed when it is attempted to be executed, which determines whether the candidate simulated input operation instruction can be executed and what the execution effect is.
[0167] For example, the execution environment of the host can be determined by the user, including but not limited to: (1) Object-level context (for the target object of the operation), including but not limited to: 1) Object instance existence: Whether the target object has been instantiated and whether it is alive.
[0168] Parameters: Whether the object handle / process ID / window ID is valid; whether the process is running; whether the file exists; whether the service is started.
[0169] 2) Object capability set (supported operation set): Which actions, instructions, and messages are allowed to be received by this object type / current state.
[0170] Parameters: Object interface capabilities, supported message list, interactive properties of controls, and window message handling callbacks.
[0171] 3) Object State: The object is in a busy / suspended / disabled / ready state, etc. For example, minimizing the window or suspending the process will also cause some operations to be invalid.
[0172] (2) Permissions and security policy contexts, including but not limited to: 1) Caller identity and permissions: Current process user identity and privilege level (administrator / normal user / sandbox permissions).
[0173] 2) System security policies / allowed lists / blocked lists: system group policy, sandbox rules, whitelists and blacklists, security software blocking rules.
[0174] Parameters: Whether the operation is in the allowed list; whether the blocking policy is triggered; system API access restrictions.
[0175] (3) System global runtime context, including but not limited to: 1) Hardware environment status: CPU (Central Processing Unit), memory, I / O (Input / Output) device status; keyboard, mouse, and graphics card mode (full screen / window); whether the device is in use.
[0176] 2) Window focus and message routing context: Parameters: Handle of the currently focused window, input focus owner; foreground / background window; message dispatch target.
[0177] 3) Operating system version and API availability environment: Does the interface exist? Some APIs do not exist in specific system versions.
[0178] 4) System resource status: memory exhausted, handle exhausted, insufficient resources prevent the operation from being initiated.
[0179] In other embodiments of this application, in addition to the execution environment described above, the confidence level can also be determined by combining the context of the caller (the operation itself corresponding to the candidate simulated input operation instruction).
[0180] Examples of caller contexts include, but are not limited to: 1) Invocation mode: synchronous invocation / asynchronous invocation; 2) Calling thread context: Whether the thread is allowed to initiate the operation, and whether it is on the UI thread.
[0181] The confidence score of a candidate simulated input operation instruction characterizes the degree of matching between the operation corresponding to the candidate simulated input operation instruction (referred to as the candidate operation) and the host's execution environment. The higher the degree of matching, the higher the confidence score. In one example, a machine learning model can be pre-trained. During training, the input consists of sample values of candidate simulated input operation instructions and sample values of the host's execution environment (corresponding to the sample values of candidate simulated input operation instructions). In addition, the input may also include the caller context. The label is the true confidence score of the sample values of the candidate simulated input operation instructions. The machine learning model can use any existing model that can be used for confidence score prediction, such as random forests, XGBoost and other decision tree-based models, fully connected neural networks, etc. The training process adopts existing conventional training processes. Finally, a confidence score prediction model is trained that can output the confidence score of the candidate simulated input operation instruction when given the input candidate simulated input operation instruction and the host's execution environment (corresponding to the candidate simulated input operation instruction).
[0182] As an example, confidence level can be represented numerically; the larger the value, the higher the confidence level. The range of confidence level values can be 0-100.
[0183] The confidence level of candidate simulated input operation commands can be determined using the confidence prediction model described above.
[0184] In one example, the true confidence level mentioned above can be provided by the user.
[0185] In another example, the true confidence value can be indirectly learned from a large amount of real-world user behavior data using either the confidence prediction model described above or an independent confidence label generation model. The steps are as follows: During operation, the system (such as the host's operating system) records the execution environment parameters when each candidate operation is attempted, whether the candidate operation is allowed by the system, and the user's subsequent operation behavior within a preset time window after the candidate operation is executed or rejected.
[0186] If a candidate operation is approved by the system and the user does not perform an undo, rollback, or repetition operation within a preset time window, the record is marked as a positive sample; if the user performs an undo or rollback operation within the preset time window, the record is marked as a negative sample.
[0187] If a candidate operation is rejected by the system, and the user attempts the operation more than twice within a preset time window, the record will be marked as a positive sample; if the user does not attempt the operation again, the record will be marked as a negative sample.
[0188] Positive samples correspond to label 1, and negative samples correspond to label 0. The confidence prediction model or confidence label generation model outputs the probability that the candidate operation is a real and valid operation of the user. The probability is linearly mapped to the interval of 0 to 100 to obtain the true confidence value (i.e., confidence label) in continuous numerical form.
[0189] Through the above methods, the system continuously accumulates a large number of training samples with implicit labels, without requiring users to actively label them.
[0190] Understandably, the above method, which indirectly derives labels based on implicit user behavior, contains a small amount of noise. The system can be configured with noise reduction rules to filter invalid samples generated by random erroneous operations. For example, if a user simultaneously performs undo and retry operations within a preset time window after a candidate operation is allowed, it indicates uncertainty in user behavior, and this sample should be filtered out. Similarly, if multiple different candidate operations trigger the same user behavior pattern within the same time window, but only a single sample repeats, it can be identified as occasional noise and filtered out, retaining only samples that consistently appear across multiple runs. Furthermore, if the system records a large number of identical or similar candidate operations with the same behavior pattern, cluster analysis can be used to identify and label them as noise or valid samples.
[0191] The preset time window can be flexibly set, such as 1s, 5s, 10s, etc., which will not be elaborated here.
[0192] In one example, regarding the aspect of "if a candidate simulated input operation command is in the operation allowed list and its confidence level is greater than a preset confidence level, then the candidate simulated input operation command is determined to be the reconstructed interaction information," the aforementioned host is specifically used for: If there is only one candidate simulated input operation instruction, the candidate simulated input operation instruction is in the operation allowed list, and the confidence level of the candidate simulated input operation instruction is greater than the preset confidence level, then the candidate simulated input operation instruction is determined to be the restored interaction information, and the candidate simulated input operation instruction is executed. When there are multiple candidate simulated input operation commands in the operation allowed list, and the confidence levels of multiple candidate simulated input operation commands are all greater than the preset confidence level, if the confidence levels are different, the candidate simulated input operation command with the higher confidence level is determined as the restored interaction information and is executed. If the confidence levels are the same, a candidate simulated input operation command is randomly selected or selected according to the preset default priority order as the restored interaction information and is executed.
[0193] In other embodiments of this application, the operation allowable list in all the above embodiments can be determined based on the host's execution environment. In this embodiment, the operation allowable list is a set of operation instructions that are allowed to take effect under the current environment. For example, if the candidate simulated input operation instruction is to close the game software, but the host's execution environment indicates that the game software is not running at this time, then the operation allowable list does not include "close the game software".
[0194] Alternatively, the operation allowable list in this embodiment can be used to confirm whether the operation corresponding to the candidate simulated input operation instruction (referred to as the candidate operation) matches the execution environment.
[0195] There may be special cases: if there are zero candidate simulated input operation instructions in the allowed operation list, execution can be abandoned and an empty result returned. For example, candidate operations may not match the execution environment, including but not limited to the following situations: (1) The operation object does not exist: For example, the candidate operation is "close the game software", but the game software is not currently running. It is understandable that the operation object does not exist and the candidate operation cannot be executed.
[0196] (2) Operation object is not applicable: For example, the candidate operation is "execute the keyboard Enter operation", but the currently focused window is the game full-screen rendering window, which does not respond to Enter key input. It is understandable that the operation object does not currently support the candidate operation, so the candidate operation cannot be executed.
[0197] (3) Operation is prohibited by the system: For example, the candidate operation is "format disk", but this operation is actively prohibited by the system based on security policy, and thus excluded from the operation allowed list.
[0198] (4) Operation does not match the current scene / context: For example, the candidate operation is "save configuration file", but no configuration has changed at present, so the operation has no actual execution. Another example is that the touch screen reports multiple continuous swipe gestures, but a single click command is recognized. Yet another example is that the current situation is a racing game, but an exit track setting command is recognized.
[0199] In other embodiments of this application, before determining the confidence level of a candidate simulated input operation instruction, a pre-filter for the legality of operation parameters can be performed to filter out candidate simulated input operation instructions with invalid operation parameters. Specifically, the input parameters carried by the candidate simulated input operation instruction can be obtained, and it can be checked whether the input parameters are empty or invalid values. At the same time, it can be checked whether the input parameter values are within the valid parameter range of the target object (referring to the entity on which the candidate simulated input operation instruction acts, i.e., the execution target of the candidate simulated input operation instruction). If the input parameters are empty, invalid, or exceed the valid parameter range, the candidate simulated input operation instruction is determined to have failed the pre-filter for the legality of operation parameters (i.e., is invalid), and is directly discarded without further confidence evaluation. If the input parameters are not empty, are valid values, and are within the valid parameter range, the candidate simulated input operation instruction is determined to have passed the pre-filter for the legality of operation parameters and enters the subsequent confidence evaluation stage.
[0200] In some embodiments, the host computer software is also used to edit instrument configuration files, lighting configuration files, and touch configuration files, and synchronize them to the touch screen.
[0201] Specifically, in this embodiment, the host computer software can directly enter the instrument editing mode, lighting editing mode, and touch editing mode to design and configure the instrument configuration file, lighting configuration file, and touch configuration file, and simultaneously apply them to the touch screen. The instrument configuration file determines the instrument displayed on the touch screen, the lighting configuration file determines the lighting displayed on the touch screen, and the touch configuration file determines the preset correspondence between touch gestures and interactive information in the host interaction mode, as well as the preset correspondence between touch gestures and instrument operations in the instrument interaction mode.
[0202] Specifically, the simulated input operations and rich events that the aforementioned host computer software can bind can be compiled into a user configuration table. This table details in detail under what conditions, scenarios, and environments gestures, key presses, and other operations are mapped to which simulated input operation or rich event. Users can edit and modify these configurations according to their preferences and habits. The configurations include, but are not limited to, the following categories: (1) Instrument configuration (i.e., instrument configuration file): selection of instrument list, switching of instrument page; (2) Lighting display type (i.e. lighting configuration file): Select a speed light or side light to enter the binding interface. The user selects the binding trigger conditions (e.g., DRS (Drag Reduction System) is turned on, ABS (Anti-lock Braking System) is engaged, TC is adjusted, a certain gear reaches the RPM (Revolutions Per Minute) threshold, RPM percentage, etc.), and selects the lighting effect (constant light, gradient, flashing, pulse, color switching, etc.) and parameters (frequency, duration, phase). (3) Touch design (i.e. touch configuration file): Determine the preset correspondence between touch gestures and interactive information in host interaction mode, and the preset correspondence between touch gestures and instrument operation in instrument interaction mode; (4) Telemetry variable class: Establish multiple sets of telemetry mappings for different games or vehicles, supporting percentage mode (0%–100%) and absolute speed mode (RPM value).
[0203] This embodiment can further design the storage location of the above user configuration table, such as local EEPROM (Electrically Erasable Programmable Read-Only Memory) or flash memory, or cloud synchronization (through a SimPro account).
[0204] This embodiment allows for the configuration of instruments and lighting according to user needs, providing a high degree of personalization and solving the aforementioned problems of limited display functions, lack of deep personalization capabilities, and inability to perform complex lighting effect linkages with telemetry data.
[0205] Through the above design, this embodiment allows users to enter different working modes by clicking controls or recognizing quick access gestures. When the working mode is host interaction mode, the corresponding system input event is generated based on the correspondence between touch gestures and simulated input operations. This allows for the adjustment of parameters in the host computer software, further controlling the display screen, steering wheel, seat, and touch screen. This avoids the need for additional mouse and keyboard connections, making operation convenient, saving space, and enhancing immersion.
[0206] (v) Instrument Interaction In this embodiment, if the working mode is the instrument interaction mode, the instrument operation corresponding to the target touch gesture is determined and executed according to the (third) preset correspondence between touch gestures and instrument operations in the instrument interaction mode.
[0207] Exemplary instrument operations may include lighting control operations, instrument file switching operations, and page switching operations within the same instrument file.
[0208] Taking lighting control as an example, in this embodiment, the main SoC is also responsible for interacting with the lighting engine, supporting the import or export of lighting effect files to achieve lighting control. If lighting control is achieved through the local driver of the touch screen, the touch screen also includes a lighting driver interface. The lighting driver interface outputs a PWM (Pulse Width Modulation) wave. The PWM wave is transmitted to the driver chip in the touch screen through the PCB board traces in the touch screen, and the driver chip controls the RGB lights to achieve lighting control. If lighting control is achieved through the host driver, the host sends a driver command to the touch screen, and the touch screen further performs local driving based on the driver command. The two methods balance latency and flexibility.
[0209] The touchscreen in this embodiment can also have a configuration and editing interface, with a built-in simplified editor that allows direct editing of lighting effects on the touchscreen and export of JSON or binary preset files.
[0210] The external interface in this embodiment also includes a UART interface for debugging.
[0211] In addition to the above-mentioned control method through working modes, this embodiment also allows users to adjust parameters in the host computer software by clicking menu items on the steering wheel's touchscreen interface. The host computer software can then control the input device based on the changed parameters, enabling cross-device adjustment.
[0212] At this time, the touch screen interaction method of this embodiment further includes: Obtain the second click instruction generated by the user clicking the parameter menu item on the touch screen, and adjust the parameters of the touch screen or the host according to the second click instruction.
[0213] It should be noted that when adjusting parameters by clicking the parameter menu item, no working mode needs to be introduced. It can be applied to the native click state and does not perform gesture recognition.
[0214] In addition to operating the instrument through the instrument interaction mode, you can also operate the instrument by clicking on the menu items on the touch screen.
[0215] At this time, the touch screen interaction method of this embodiment further includes: The system obtains a third click command generated by the user clicking the configuration menu item on the touch screen, and switches or edits at least one of the instrument configuration file, lighting configuration file, and touch configuration file according to the third click command. The instrument configuration file is used to determine the instrument displayed on the touch screen, the lighting configuration file is used to determine the lighting displayed on the touch screen, and the touch configuration file is used to determine the preset correspondence between touch gestures and interactive information in the host interaction mode and the preset correspondence between touch gestures and instrument operations in the instrument interaction mode.
[0216] It should be noted that when switching or editing file working modes by clicking the configuration menu item, no working mode needs to be introduced. It can be applied to the native click state without gesture recognition.
[0217] This embodiment enables exportable configuration or scene switching: it supports one-click switching of multiple combinations of instrument, lighting and touch configurations (exemplary combinations are combinations of instrument configuration files, lighting configuration files and touch configuration files) according to game or vehicle scenes, which facilitates quick switching in competition or multiplayer environments.
[0218] For example, users can customize the combination for this scenario in a game, and when in a game, switch to the combination that applies to this scenario.
[0219] To enable touch interaction on a steering wheel with a touchscreen in a racing simulation scenario, simulating mouse and keyboard input while mimicking accidental touches and telemetry context linkage during races, this embodiment presents a collaborative touch event handling process between the device (touchscreen) and the PC (host), as follows: Figure 2 As shown, it illustrates the complete chain from touch sampling, debouncing / calibration, event segmentation, gesture recognition, pattern decision, event mapping, to instruction encapsulation and transmission, PC-side reception and parsing, configuration matching and context decision, action execution and feedback closed loop. This touch event processing flow supports both driverless mouse and keyboard simulation via USB HID and the implementation of high-level interactive commands (i.e., commands corresponding to rich events), which are linked with the touch screen's page and lighting effects.
[0220] The above touch event handling process includes: (1) Sampling: Sampling at r Hz (120Hz recommended) to obtain the set of contact coordinates.
[0221] (2) De-jittering / calibration: Based on the set distance threshold, the coordinate changes of the touch points in the touch point coordinate set are processed. For example, coordinate changes of less than 10 pixels in every 100ms are ignored. Combined with the speed threshold, the effective movement of the user and micro-jitter are distinguished.
[0222] (3) Touch event (complete touch process) segmentation: The touch sequence is segmented by time window (e.g., 20ms–50ms) and the start and end information of the touch is identified. When no touch is pressed, the sampled touch coordinate set is empty. When the touch is pressed, the touch coordinate set is not empty. When there is multi-touch, the number of touch coordinate sets corresponds to the number of fingers pressed. The touch press and release can be determined from the change in the number of touch coordinate sets, thus realizing touch event segmentation.
[0223] (4) Gesture recognition: Based on the recognized touch events and time windows, the gesture type can be identified according to the trend and range of the touch coordinates within the time window, and mapped to common touch gestures as follows: Single-point short press: Left-click with the mouse; Single click / double click: Double-click with the left mouse button or open the shortcut menu; Single-point long press: Right-click or enter context or edit mode; Single-finger swipe: In instrument interaction mode, it is mapped to switching instrument files (up and down) or turning pages (left and right); in mouse simulation mode, it is mapped to pointer movement. Two fingers moving vertically: This is used to scroll the mouse wheel. Two-finger tap: middle mouse button click or switch modes; Press and hold with two fingers: Enter or exit the light editing mode, or use it as a mode lock switch; Three-finger or higher gestures: can be customized as shortcut operations (such as saving configuration with one click, returning to the main interface).
[0224] In addition to the gesture recognition mentioned above, there are some special constraints for racing game scenarios: 1) Gesture priority rules: long press > two-finger operation > single-finger swipe > single-finger short press. That is, when two fingers are detected to be in contact, it should be recognized as scrolling rather than single-finger tap.
[0225] 2) Competition Context Suppression: Combining telemetry data obtained from racing games, when the vehicle speed or steering angle is outside the vehicle speed threshold range, the mouse sensitivity is reduced or mouse clicks are completely blocked by default. At this time, standard HID mouse messages generated by single-point touch are automatically intercepted and suppressed. Only multi-finger compound gestures that meet specific spatiotemporal characteristics (such as two-finger coordinated displacement) are extracted and allowed, thereby achieving physical isolation from the general mouse mode to the competition instrument mode. In this process, the mouse sensitivity is essentially the gain coefficient of touch displacement and coordinate mapping. By dynamically lowering the mouse sensitivity or increasing the trigger dead zone, the interaction damping can be increased to ensure that the screen cursor will not shift significantly or be accidentally clicked under the accidental touch caused by high-speed vibration of the racing car or violent driving, thereby ensuring the stability of system operation during the competition. Only important instrument switching gestures (such as swiping up and down to switch instrument files, which requires two fingers + swiping up) are allowed to avoid accidental touches.
[0226] 3) Contact morphology judgment: Distinguish between palm touch and finger operation based on contact area or pressure difference: When the contact area is greater than the area threshold, it is judged as a touch and ignored.
[0227] (5) Mode decision: Based on the current display page of the touch screen, the current working mode of the touch screen can be determined. At the same time, combined with the telemetry data in the racing game (e.g., vehicle speed, steering angle), it can be determined whether to allow mouse events or keyboard events or only allow instrument operation.
[0228] (6) Event mapping: The device converts touch gestures into specific operations (instrument operation or analog input operation), or the device generates rich events based on the user pressing the button on the steering wheel or manually inputting information to the touch screen. If it is an instrument operation, the touch screen will execute it directly. If it is an analog input operation (mouse operation or keyboard operation) or a rich event, it will be output by USB HID.
[0229] (7) Command encapsulation and transmission: The simulated input operation command or rich event is encapsulated into an action message (ActionID (Action Message ID) + parameters + timestamp or sequence number) and sent to the PC via the USB HID communication channel.
[0230] (8) PC terminal receiving and parsing: The PC terminal receives action messages, parses them into simulated input operation instructions or rich events, enqueues them, and performs deduplication or rate limiting.
[0231] (9) Configuration matching and context decision-making: If it is a simulated input operation command, the PC determines the final action and whether to allow it based on the user configuration (mapping table / layout / macro / rule) and telemetry data. Specifically, it generates candidate simulated input operation commands and compares them with the operation allow list. Only candidate simulated input operation commands that fall into the operation allow list and meet the confidence level are allowed to be executed on the bus. All other signals are truncated and discarded, thereby achieving precise filtering and release of mouse events or keyboard events according to the driving situation.
[0232] (10) Action execution and feedback loop: Execute actions (mouse / keyboard / virtual buttons / lighting effects / page control) and send back the status for screen / lighting effects / mode prompt synchronization.
[0233] Through the above process, users can directly operate the PC game software (including game startup and exit) and host computer software using the touch screen and buttons on the steering wheel without needing an external mouse and keyboard. This solves the problem of racing game players needing to frequently use a mouse and keyboard next to their racing seats but finding it inconvenient to place a table. All PC operations can be achieved using only a steering wheel with a touch screen, greatly enhancing the gaming experience for racing game players.
[0234] A traditional steering wheel is just a steering wheel, but the steering wheel with a touch screen provided in this embodiment not only fulfills the basic steering wheel functions, but also replaces the mouse and keyboard, and realizes the automatic binding configuration of touch shortcut gestures, touch virtual buttons, and physical steering wheel buttons, achieving the effect of an intelligent operating terminal, far exceeding the functional definition of a steering wheel, and bringing great convenience to users.
[0235] This embodiment has the following advantages: (1) Significantly reduce the false touch rate: Based on multiple determinations of telemetry context and touch point shape, the risk of misoperation during the competition is reduced.
[0236] (2) Improve efficiency and intuitive interaction: Swipe up, down, left and right to quickly switch between instrument files and pages. Users can complete complex operations without an external mouse and keyboard.
[0237] (3) Real-time lighting editing and preview: Edit and preview lighting effects directly on the device in real time, shorten the configuration cycle and improve the personalized experience.
[0238] (4) Plug-and-play desktop interaction capability: The standard HID protocol enables the device to directly replace the mouse and keyboard to complete desktop interaction, and is compatible with a wide range of operating systems.
[0239] (5) Low-latency smooth experience: End-to-end optimization (sampling, filtering, prediction) ensures smooth mouse movement and gesture response with low latency.
[0240] (6) Scenario-based configuration and quick switching: Multiple configurations support different games / vehicles, and one-click switching saves debugging time and supports event requirements.
[0241] This embodiment provides a touch screen, which includes a control component that operates using the touch screen interaction method described above.
[0242] This embodiment applies a touchscreen to the field of racing simulation. The touchscreen enables functions such as touchpad, virtual buttons, keyboard and mouse operation, and instrument display, avoiding the need for additional mouse and keyboard connections. It integrates game instrument operation with touch, mouse, and keyboard, making it convenient to operate, saving space, and enhancing immersion. The driver does not need to be distracted by the screen and other devices, and it avoids the problem of mouse and keyboard causing a sense of detachment for the user due to their non-realistic racing operation ecosystem.
[0243] This embodiment provides a steering wheel with an integrated touch screen. The steering wheel with an integrated touch screen includes the aforementioned touch screen, which can be integrated onto the surface of the steering wheel in any way, such as by bonding or bolting.
[0244] The above settings enable the steering wheel to integrate the operation functions of the touch screen, achieving integration. Through the steering wheel with integrated touch screen, keyboard and mouse operations can be performed on the touch screen of the steering wheel to achieve interaction with the PC. It is a steering wheel with touch screen that has touch interaction, virtual buttons and mouse simulation functions.
[0245] This embodiment provides a racing simulator, which includes a host and a steering wheel with an integrated touch screen, and the host and the control components of the touch screen of the steering wheel with the integrated touch screen are communicatively connected.
[0246] The control unit is used to encapsulate interactive information, obtain action messages, and transmit the action messages to the host.
[0247] The host is used to parse action messages to reconstruct interaction information and generate corresponding system input events based on the interaction information to control the input devices of the racing simulator. The input devices include at least one of a steering wheel, a touch screen, a seat, and pedals.
[0248] The host computer is equipped with host computer software, which is used to edit instrument configuration files, lighting configuration files, and touch configuration files, and synchronize them to the touch screen.
[0249] In other embodiments of this application, the above-mentioned interactive information includes: simulated input operation commands; In terms of parsing action messages to reconstruct interaction information, the host is specifically used to: generate candidate simulated input operation instructions based on the action messages, and determine the confidence level of the candidate simulated input operation instructions. If the candidate simulated input operation instruction is in the operation allowed list and its confidence level is greater than the preset confidence level, then the candidate simulated input operation instruction is determined to be the reconstructed interaction information.
[0250] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a control component, implements the steps in the above-described method embodiments.
[0251] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a control component, implements the steps in the above-described method embodiments.
[0252] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations and be authorized by the owner of the corresponding device.
[0253] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0254] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A touchscreen interaction method, characterized in that, The interaction methods of the touch screen include: Obtain the set of touch point coordinates generated by the user touching the touch screen, perform gesture recognition based on the set of touch point coordinates, and obtain the initial touch gesture; The gesture priority of each initial touch gesture is dynamically determined based on the telemetry data of the racing game, and the final target touch gesture is determined from all initial touch gestures based on the gesture priority. Determine the operating mode of the touchscreen based on the display page; If the working mode is host interaction mode, the interaction information corresponding to the target touch gesture is determined according to the preset correspondence between touch gestures and interaction information in host interaction mode, and the interaction information is sent to the host so that the host generates the corresponding system input event; wherein, the host interaction mode includes mouse simulation mode and keyboard simulation mode. Mouse simulation mode is used to simulate the operation of the host through the mouse, and keyboard simulation mode is used to simulate the operation of the host through the keyboard.
2. The touchscreen interaction method according to claim 1, characterized in that, Telemetry data includes game status, current vehicle speed, and current vehicle steering angular velocity; The gesture priority of each initial touch gesture is dynamically determined based on telemetry data from the racing game, specifically including: If the game state indicates that the vehicle is currently in a race, the current vehicle speed is greater than the preset vehicle speed, and the current vehicle steering angular velocity is greater than the preset angular velocity, then the category priority of the first type of touch gesture is determined to be higher than the category priority of other types of touch gestures, so as to obtain the category priority of each initial touch gesture; the first type of touch gesture includes touch gestures related to vehicle control; If the game state indicates that the game is currently in the process of entering the parking area, then the category priority of the second type of touch gesture is determined to be higher than the category priority of other types of touch gestures, so as to obtain the category priority of each initial touch gesture; the second type of touch gesture includes touch gestures related to the strategy configuration; For each initial touch gesture, a comprehensive priority is calculated based on its category priority and its own priority to dynamically determine its gesture priority.
3. The touchscreen interaction method according to claim 2, characterized in that, The initial touch gesture has different priorities depending on the working mode.
4. The touchscreen interaction method according to claim 2, characterized in that, The final target touch gesture is determined from all initial touch gestures based on the gesture priority, specifically including: The initial touch gesture with the highest overall priority is selected as the final target touch gesture.
5. The touchscreen interaction method according to claim 1, characterized in that, The interaction method of the touch screen further includes: if the working mode is the instrument interaction mode, determining and executing the instrument operation corresponding to the target touch gesture according to the preset correspondence between touch gestures and instrument operations in the instrument interaction mode.
6. The touchscreen interaction method according to claim 1, characterized in that, Methods to enter the touchscreen's working mode include: The system obtains the first click command generated by the user clicking on a control on the touchscreen, and opens the display page according to the first click command to enter the working mode; different display pages correspond to different working modes. Alternatively, obtain the set of touch point coordinates generated by the user touching the touch screen, perform gesture recognition based on the set of touch point coordinates, and if a quick access gesture is recognized, open the display page according to the recognized quick access gesture to enter the working mode; different display pages correspond to different working modes.
7. The touchscreen interaction method according to claim 1, characterized in that, The interaction method of the touch screen also includes: Obtain the second click instruction generated by the user clicking the parameter menu item on the touch screen, and adjust the parameters of the touch screen or the host according to the second click instruction.
8. The touchscreen interaction method according to claim 1, characterized in that, The interaction method of the touch screen also includes: The system obtains a third click command generated by the user clicking a configuration menu item on the touchscreen, and switches or edits at least one of the instrument configuration file, lighting configuration file, and touch configuration file according to the third click command. The instrument configuration file is used to determine the instrument displayed on the touchscreen, the lighting configuration file is used to determine the lighting displayed on the touchscreen, and the touch configuration file is used to determine the preset correspondence between touch gestures and interactive information in the host interaction mode and the preset correspondence between touch gestures and instrument operations in the instrument interaction mode.
9. The touchscreen interaction method according to claim 1, characterized in that, The contact coordinate set includes the sampling time, coordinates, pressure, and contact area of each contact point; Gesture recognition is performed based on the set of touch point coordinates to obtain the initial touch gesture, which specifically includes: Based on the set of contact point coordinates, calculate the distance between the starting and ending contact points, and calculate the touch movement speed; the distance between two contact points is calculated based on the coordinates of the two contact points. If the distance is less than the preset distance, the touch movement speed is less than the preset speed, the pressure of each touch point is less than the preset pressure, or the contact area of each touch point is greater than the preset area, then delete the touch point coordinate set. Otherwise, gesture recognition is performed based on the set of touch point coordinates to obtain the initial touch gesture.
10. A touch screen, characterized in that, The touch screen includes a control component, which operates using the interaction method of the touch screen as described in any one of claims 1-9.
11. A steering wheel with an integrated touchscreen, characterized in that, The steering wheel with integrated touch screen includes the touch screen as described in claim 10.
12. A racing simulator, characterized in that, The racing simulator includes a host and a steering wheel with an integrated touch screen as described in claim 11, wherein the host and the control components of the touch screen of the steering wheel with the integrated touch screen are communicatively connected; The control unit is used to encapsulate interactive information, obtain action messages, and transmit the action messages to the host. The host is used to parse action messages to reconstruct interaction information, and generate corresponding system input events based on the interaction information to control the input devices of the racing simulator; Input devices include at least one of a steering wheel, a touchscreen, a seat, and pedals.
13. The racing simulator according to claim 12, characterized in that, The interactive information includes: simulated input operation commands; In terms of parsing action messages to reconstruct interaction information, the host is specifically used to: generate candidate simulated input operation instructions based on the action messages, and determine the confidence level of the candidate simulated input operation instructions. If the candidate simulated input operation instructions are in the operation allowed list and their confidence level is greater than the preset confidence level, then the candidate simulated input operation instructions are determined to be the reconstructed interaction information.
