Semantic parsing and mapping method for cross-platform touch commands of on-screen devices

Abstract

The present invention discloses a semantic parsing and mapping method for cross-platform touch instructions of a screen sharing device, including: S1: obtaining the touch instructions of the source device through the screen sharing device, the touch instructions including the touch position, touch gesture type and touch timing information; S2: based on the operating system type and application scenario of the target device, semantically parsing the touch instructions to generate intermediate semantic layer instructions, the intermediate semantic layer instructions including operation logic, interaction intention and target control identification; S3: dynamically mapping the intermediate semantic layer instructions to the native instructions of the target device according to the platform protocol and interaction rules of the target device, the native instructions adapting to the touch event processing mechanism of the target device; S4: sending the native instructions to the target device for execution, and synchronously updating the display interface of the screen sharing device and the display content of the target device.

Description

Technical Field

[0001] The present invention relates to the technical field of screen sharing devices, and specifically to a semantic parsing and mapping method for cross-platform touch instructions of screen sharing devices. Background Art

[0002] With the diversification of smart devices and the growing demand for cross-platform interaction, screen sharing devices, as core tools for connecting different devices, have gradually become a key technology in home entertainment, business meetings, and education scenarios. However, existing technologies still have the following limitations:

[0003] Insufficient touch command compatibility: Traditional screen mirroring devices typically only support one-way content projection (such as video and image transmission) and cannot achieve cross-platform interaction of touch commands. For example, the Force Touch gesture on iOS devices and the long press event on Android systems have different semantics. Directly forwarding touch signals will cause operation failure or logical confusion.

[0004] Static mapping rules lead to rigid adaptation: Existing solutions (such as the wireless screen projection device mentioned in page 4) often use fixed protocol conversion rules and cannot dynamically adapt to the interaction logic of different operating systems. For example, when the target device interface is updated or the control type changes, the original mapping rules may become invalid and require manual reconfiguration;

[0005] Multi-source command conflicts and lack of processing: In a multi-device collaboration scenario, when multiple source devices send touch commands at the same time, the existing technology lacks an effective conflict resolution mechanism, resulting in operation delays or execution errors. Summary of the Invention

[0006] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, one object of the present invention is to propose a semantic parsing and mapping method for cross-platform touch commands on a same-screen device, comprising:

[0007] S1: Acquire a touch command from the source device through the screen sharing device, where the touch command includes touch position, touch gesture type, and touch timing information;

[0008] S2: Based on the operating system type and application scenario of the target device, semantically analyze the touch control instruction to generate an intermediate semantic layer instruction, where the intermediate semantic layer instruction includes operation logic, interaction intent, and target control identifier;

[0009] S3: Dynamically mapping the intermediate semantic layer instructions to native instructions of the target device according to the platform protocol and interaction rules of the target device, wherein the native instructions are adapted to the touch event processing mechanism of the target device;

[0010] S4: Send the native command to the target device for execution, and synchronously update the display interface of the screen controller and the display content of the target device;

[0011] The step S2 also includes multi-level semantic analysis:

[0012] The first level analyzes the physical parameters of touch gestures, including sliding direction, pressure value, and multi-touch coordinates;

[0013] The second level combines the current display interface layout of the target device to identify the functional modules corresponding to the touch operation;

[0014] The third level dynamically adjusts the semantic weight according to the application scenario, giving priority to mapping high-frequency operation instructions.

[0015] Preferably, the semantic analysis in S2 further includes:

[0016] Matching the target device's interaction logic through a dynamic adaptation rule base that includes operating system protocols, application interface control types, and touch event priorities for different platforms;

[0017] When it is detected that the control type of the target device is a dynamically generated control, the intermediate semantic layer instructions are optimized based on the control hierarchy and context semantics.

[0018] Preferably, the mapping method supports the integration of multi-source touch commands, including:

[0019] Receive touch commands from multiple source devices and sort them by timestamps and event queues;

[0020] When multiple command conflicts are detected, the execution command is selected based on user identity permissions or operation priority.

[0021] Preferably, the dynamic mapping in S3 further includes:

[0022] Monitor changes in the target device's interface status in real time. If the interface update causes the control identifier to become invalid, re-parsing of the semantic layer instructions will be triggered.

[0023] When the target device is a virtual reality device, the touch commands are mapped into three-dimensional space interaction events, and the display content of the virtual scene and the physical screen are synchronized.

[0024] Preferably, the method further comprises a conflict handling mechanism:

[0025] When the touch protocols of the source device and the target device are incompatible, simulated touch events are generated, including virtual clicks, sliding track simulation, and time compensation for long press events;

[0026] The execution results are returned to the source device through the feedback channel to optimize the semantic parsing rules of subsequent instructions.

[0027] Preferably, the mapping method supports cross-platform protocol conversion, including:

[0028] Convert mouse events of Windows system to MotionEvent events of Android system;

[0029] Map iOS Force Touch gestures to long press events on Android, and dynamically adjust mapping parameters based on pressure values.

[0030] Preferably, the dynamic adaptation rule base is optimized by machine learning:

[0031] Collect users' correction operations on mapping results and generate training data sets;

[0032] Based on the neural network model, the mapping strategy in the rule base is updated to improve the accuracy of semantic parsing.

[0033] Preferably, the mapping method supports multi-window collaborative control:

[0034] When the target device displays multiple application windows in split screen, the command is mapped to the corresponding application process according to the window area to which the touch position belongs;

[0035] When the display interface of the same screen device is a multi-window layout, cross-window command distribution is achieved by dividing the sub-interaction area.

[0036] Preferably, the mapping method further includes a device switching mechanism:

[0037] When the target device is detected to be switched, the context state of the current intermediate semantic layer instruction is retained;

[0038] Remaps commands and restores continuity of operational flow based on the platform protocol of the new target device.

[0039] The above solution of the present invention includes at least the following beneficial effects:

[0040] Cross-platform semantic adaptation capabilities

[0041] Abstract touch commands are expressed through intermediate semantic layer instructions (such as operation logic, interaction intent, and target control identifiers), effectively isolating protocol differences between different platforms. For example, the iOS 3D Touch pressure value can be dynamically mapped to the duration parameter of an Android long press event, solving cross-platform gesture compatibility issues.

[0042] Combined with a dynamic adaptation rule library (including operating system protocols, control type priorities, etc.), it supports real-time response to changes in the target device interface, avoiding mapping failures caused by control updates;

[0043] Multi-level semantic analysis and dynamic optimization

[0044] A three-level semantic parsing architecture (physical parameters → functional modules → scene weights) is used to accurately identify touch intent. For example, the sliding direction and pressure value combined with the current interface layout can distinguish between "page turning" and "zooming" operations, reducing the error rate.

[0045] Introducing a machine learning optimization mechanism to iteratively update the rule base based on user feedback data, improving the mapping efficiency and accuracy of high-frequency operations (such as game touch controls);

[0046] Multi-source instruction coordination and conflict resolution

[0047] Through timestamp sorting and permission management, the problem of multi-device command conflicts can be resolved. For example, in a conference scenario, the speaker's device has operation priority to avoid interference from multiple users;

[0048] Supports 3D mapping of virtual reality devices, converting two-dimensional touch commands into spatial interaction events (such as mapping gesture trajectories to object rotations in the VR scene), expanding application scenarios;

[0049] Reduce development and maintenance costs

[0050] Through the protocol conversion module (such as Windows mouse event → Android MotionEvent), developers can reduce the repeated adaptation work for different platforms;

[0051] Seamless device switching: The context state of the intermediate semantic layer is preserved to ensure the continuity of the operation process when switching the target device, improving user efficiency.

[0052] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0054] Figure 1 This is a flowchart of a method for semantic parsing and mapping touch commands across platforms of a same-screen device in an embodiment of the present invention;

[0055] Figure 2 This is a flow chart of touch command acquisition provided in an embodiment of the present invention;

[0056] Figure 3 is a flowchart providing semantic parsing in an embodiment of the present invention;

[0057] Figure 4 is a flow chart providing dynamic mapping in an embodiment of the present invention;

[0058] Figure 5 is a flowchart of performing synchronization according to an embodiment of the present invention;

[0059] Figure 6 is a semantic weight table provided in an embodiment of the present invention;

[0060] Figure 7 This is a diagram describing the conflict scenario types and resolution strategies provided in an embodiment of the present invention;

[0061] Figure 8 is a diagram showing the correspondence between touch gestures and three-dimensional interactive actions provided in an embodiment of the present invention;

[0062] Figure 9 is a pressure level diagram provided in an embodiment of the present invention;

[0063] Figure 10 This is a protocol compatibility matrix provided in an embodiment of the present invention.

[0064] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0065] The following describes in detail embodiments of the present invention, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0066] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "circumferential", "radial", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0067] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0068] In the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," "connect," "fixed," etc. should be understood broadly. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediary; or internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0069] In the present invention, unless otherwise expressly specified or limited, a first feature being "above" or "below" a second feature may include the first and second features being in direct contact, or may include the first and second features being in contact not directly but through another feature between them. Furthermore, a first feature being "above," "above," and "above" a second feature may include the first feature being directly above or obliquely above the second feature, or may simply mean that the first feature is higher in level than the second feature. A first feature being "below," "below," and "below" a second feature may include the first feature being directly below or obliquely below the second feature, or may simply mean that the first feature is lower in level than the second feature.

[0070] The semantic parsing and mapping method of cross-platform touch commands of a same-screen device according to an embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0071] See also Figures 1 to 5 , in this embodiment, including:

[0072] S1: Obtain touch commands from the source device through the screen sharing device. The touch commands include touch position, touch gesture type, and touch timing information.

[0073] S2: Based on the operating system type and application scenario of the target device, the touch command is semantically parsed to generate intermediate semantic layer instructions. The intermediate semantic layer instructions include operation logic, interaction intent, and target control identification;

[0074] S3: Dynamically maps intermediate semantic layer instructions to native instructions of the target device based on the platform protocol and interaction rules of the target device. The native instructions are adapted to the touch event processing mechanism of the target device.

[0075] S4: Send the native command to the target device for execution, and synchronously update the display interface of the screen controller and the display content of the target device;

[0076] Cross-platform semantic adaptation capabilities

[0077] Abstract touch commands are expressed through intermediate semantic layer instructions (such as operation logic, interaction intent, and target control identifiers), effectively isolating protocol differences between different platforms. For example, the iOS 3D Touch pressure value can be dynamically mapped to the duration parameter of an Android long press event, solving cross-platform gesture compatibility issues.

[0078] Combined with a dynamic adaptation rule library (including operating system protocols, control type priorities, etc.), it supports real-time response to changes in the target device interface, avoiding mapping failures caused by control updates;

[0079] Multi-level semantic analysis and dynamic optimization

[0080] A three-level semantic parsing architecture (physical parameters → functional modules → scene weights) is used to accurately identify touch intent. For example, the sliding direction and pressure value combined with the current interface layout can distinguish between "page turning" and "zooming" operations, reducing the error rate.

[0081] Introducing a machine learning optimization mechanism to iteratively update the rule base based on user feedback data, improving the mapping efficiency and accuracy of high-frequency operations (such as game touch controls);

[0082] Multi-source instruction coordination and conflict resolution

[0083] Through timestamp sorting and permission management, the problem of multi-device command conflicts can be resolved. For example, in a conference scenario, the speaker's device has operation priority to avoid interference from multiple users;

[0084] Supports 3D mapping of virtual reality devices, converting two-dimensional touch commands into spatial interaction events (such as mapping gesture trajectories to object rotations in the VR scene), expanding application scenarios;

[0085] Reduce development and maintenance costs

[0086] Through the protocol conversion module (such as Windows mouse event → Android MotionEvent), developers can reduce the repeated adaptation work for different platforms;

[0087] Seamless device switching: Preserving the contextual state of the intermediate semantic layer ensures the continuity of the operation process when switching the target device, improving user efficiency.

[0088] In this embodiment, the semantic parsing in S2 further includes: matching the interaction logic of the target device through a dynamic adaptation rule base, the rule base including the operating system protocols, application interface control types, and touch event priorities of different platforms; when it is detected that the control type of the target device is a dynamically generated control, optimizing the intermediate semantic layer instructions based on the control hierarchy and contextual semantics;

[0089] Dynamic adaptation rule base construction

[0090] Operating system protocol sub-library: records the touch event definitions of different platforms (Android / iOS / Windows), including Android's MotionEvent event type, iOS's UITouch properties, and Windows' WM_TOUCH message structure;

[0091] Control type mapping table: defines the equivalent relationship between cross-platform controls. For example, Android's RecyclerView and iOS's UICollectionView are considered to be the same type of scrolling containers, and their sliding event thresholds are associated.

[0092] Priority strategy table: Set the execution priority of touch events in different scenarios. For example, in the video playback interface, "sliding the progress bar" takes precedence over "clicking the pause button."

[0093] Identification and Processing of Dynamically Generated Controls

[0094] When the target device interface contains dynamically generated controls (such as web controls dynamically loaded through JavaScript or custom components rendered by Flutter), perform the following operations:

[0095] Hierarchical structure analysis: Obtain the control tree of the current interface of the target device through the screen sharing device, and analyze the parent-child hierarchical relationship and properties of the control (such as ID, class name, visibility);

[0096] Contextual semantic analysis: Infers the intended functionality of a control based on the interaction logic of the container where the control resides (for example, the sliding direction and data loading method of a dynamic list).

[0097] Optimized instruction generation: If the control does not have a unique identifier, a temporary control identifier is generated based on the hierarchical path (such as Root / FrameLayout[2] / ListView[0] / Button) and a semantic annotation is added (such as "the third item on the shopping cart page is the add-to-cart button").

[0098] Rule matching and instruction correction

[0099] Based on the operating system type of the target device, the matching mapping policy is extracted from the rule base:

[0100] Example 1: When a dynamically generated button in a WebView is detected on an Android device, the touch coordinates are converted to relative coordinates within the web page and a simulated click event is injected through JavaScript.

[0101] Example 2: In an AR app for iOS devices, if a 3D control (such as a virtual button) is dynamically generated, the mapping parameters of the touch coordinates are corrected based on the control's projected position in 3D space.

[0102] Machine Learning Optimization Rule Base

[0103] Introducing a feedback learning mechanism to continuously optimize the dynamic adaptation rule base:

[0104] Collect user correction operations on mapping results (such as manually reselecting the target control or adjusting the sliding distance) to generate a labeled dataset;

[0105] Train a convolutional neural network (CNN) model to identify patterns in the association between control hierarchies and touch intent.

[0106] When there is no matching strategy in the rule base, the trained model is called to predict the best mapping solution and automatically add new rules to the database.

[0107] This embodiment achieves the following technical advantages by refining the application logic of the dynamic adaptation rule library and the processing flow of dynamically generated controls:

[0108] Improve cross-platform control adaptation efficiency

[0109] Solve the problem of identifying dynamic controls in hybrid development frameworks such as WebView and Flutter through control type mapping tables and hierarchical structure analysis;

[0110] Semantic annotation generation enables precise location of controls without IDs, avoiding mapping failures caused by interface revisions.

[0111] Enhance robustness in complex scenarios

[0112] 3D space coordinate correction supports dynamic control interaction of AR / VR applications;

[0113] Priority strategy table ensures that critical operations are responded to first;

[0114] Reduce manual maintenance costs

[0115] The machine learning optimization mechanism can automatically learn user operation habits. For example, for combo operations in game scenarios, the model automatically optimizes the mapping parameters of click intervals, reducing the developer's manual debugging workload.

[0116] The dynamic expansion capability of the rule base supports rapid adaptation to new operating system versions;

[0117] Improve user experience

[0118] In dynamic lists (such as the infinite scrolling feed of social apps), a sliding distance mapping algorithm based on contextual semantics ensures that page turning across platforms is as smooth as on native devices.

[0119] Continuously optimize the rule base through feedback learning to reduce user error rates.

[0120] In this embodiment, step S2 also includes multi-level semantic analysis:

[0121] The first level analyzes the physical parameters of touch gestures, including sliding direction, pressure value, and multi-touch coordinates;

[0122] Touch gesture digitization:

[0123] The touch signal acquisition module of the on-screen controller captures the original touch data of the source device in real time, including:

[0124] Sliding direction (8-way angle quantization, accuracy ±5°);

[0125] Pressure value (graded sampling, 0-1N range is divided into 256 levels);

[0126] Multi-touch coordinates (normalized to relative coordinates between 0 and 1 based on the target device screen resolution);

[0127] Perform noise filtering and normalization on the raw data, for example:

[0128] Use Kalman filtering to eliminate touch track jitter;

[0129] Converts absolute coordinates from different devices (such as iPad and Surface) to relative coordinates on the target device screen.

[0130] Gesture type classification:

[0131] Establish touch gesture feature vectors, including sliding speed, acceleration, and contact area change rate;

[0132] The gestures are classified into preset types (single click, double click, long press, slide, pinch) in real time through a lightweight convolutional neural network (MobileNetV3).

[0133] The second level combines the current display interface layout of the target device to identify the functional modules corresponding to the touch operation;

[0134] Dynamic modeling of interface layout

[0135] Get a screenshot of the current display interface of the target device and extract UI elements (buttons, lists, input boxes) through OCR and image recognition technology;

[0136] Build a topological diagram of the interface layout, labeling the function type of each control (such as "play / pause button", "scroll list", "text input area");

[0137] Touch intent association:

[0138] Spatially align touch coordinates with the interface layout:

[0139] If the touch point falls on the "play button" area, the operation intention is marked as "media control";

[0140] If the touch track spans multiple list items, the mark intent is "scroll through";

[0141] Combined with application process information (such as whether the currently running app is a video player or a game), further refine the intent (such as "video progress adjustment" or "character movement").

[0142] The third level dynamically adjusts semantic weights based on application scenarios, prioritizing mapping high-frequency operation instructions;

[0143] Scene perception and weight distribution

[0144] Define multi-dimensional scene labels (such as "game mode," "document editing," and "video conferencing") and determine the scene in real time using the following methods:

[0145] Monitor the application window focus changes of the target device;

[0146] Analyze the spatiotemporal distribution characteristics of touch command sequences (e.g., high-frequency clicks may be game operations);

[0147] Set semantic weight tables for different scenarios (such as Figure 6 shown):

[0148] In the weight calculation module, high-frequency instructions in the current scene (such as the "jump" combo in the game) are dynamically accelerated:

[0149] Establish a command history queue and count the triggering frequency of each command type in the past 5 seconds;

[0150] When it is detected that the frequency of a certain instruction exceeds the threshold (such as 3 times per second), its mapping priority is automatically increased;

[0151] Through the instruction preloading mechanism, the native instruction cache of the target platform is generated in advance to reduce transmission delay.

[0152] This embodiment achieves the following technical improvements by refining the multi-level semantic parsing process:

[0153] The accuracy of touch intention recognition is significantly improved

[0154] Refined physical parameter processing: Through Kalman filtering and coordinate normalization, the touch trajectory error between different devices can be reduced to ±2 pixels. For example, the tilted writing trajectory of the Surface Pen is accurately mapped to the pressure sensing event of the Apple Pencil on the iPad.

[0155] Functional module association accuracy: Interface layout modeling reduces the false recognition rate of "play button clicks" from 18% in traditional solutions to 3%, especially for complex UIs (such as Photoshop toolbars).

[0156] Enhanced dynamic scene adaptation capabilities

[0157] Game Mode Optimization: In MOBA games, the command response delay for high-frequency skill combos can be reduced from 120ms to 45ms, meeting esports-level operational requirements.

[0158] Improved document editing efficiency: The mapping accuracy of pinch-to-zoom and long-press selection has been increased to 98%, supporting seamless cross-platform editing (e.g., Windows trackpad → iPad version of WPS);

[0159] Resource utilization optimization

[0160] Instruction preloading mechanism: By caching native code for high-frequency instructions, CPU usage is reduced by 22% (from 35% to 27%), significantly improving performance on low-end devices (such as Android phones under 1,000 yuan).

[0161] Dynamic weight allocation: In video conferencing scenarios, key operations (such as the "Share Screen" button) are given higher priority, reducing the probability of accidentally touching other controls by 60%.

[0162] Consistent experience across platforms

[0163] In hybrid office scenarios, a three-finger swipe on a Windows device's trackpad can be accurately mapped to the MacBook's Mission Control gesture, increasing the success rate from 78% to 96%.

[0164] For foldable screen devices (such as Samsung Galaxy Z Fold), the layout model is automatically switched according to the screen expansion status, and the touch mapping accuracy of split-screen operation is as high as 99%.

[0165] In this embodiment, the mapping method supports the integration of multi-source touch commands, including:

[0166] Receive touch commands from multiple source devices and sort them by timestamps and event queues;

[0167] Heterogeneous device access:

[0168] Supports simultaneous connection to multiple source devices such as mobile phones, tablets, PCs, smart watches, etc., and receives touch commands through unified communication protocols (such as WebSocket or Bluetooth BLE);

[0169] Standardize the format of input instructions:

[0170] Convert touch parameters of different devices (such as the tilt angle of Apple Pencil and the rotation angle of Surface Dial) into standardized vector data;

[0171] Unified time base: Synchronize the clocks of all devices through the NTP protocol, ensuring the timestamp accuracy of cross-device commands is ±1ms.

[0172] Dynamic priority queue construction:

[0173] Create a timestamp-based double-buffered event queue, divided into a high-priority queue (such as touch click and gesture start events) and a low-priority queue (such as gesture duration events);

[0174] Sort by the following rules:

[0175] Continuous gesture events (such as long press and drag) on ​​the same device maintain temporal consistency;

[0176] Cross-device commands are inserted into the queue in timestamp order. If timestamps overlap, they are sorted by weight based on device type (e.g., the weight of the conference speaker's device is +30%).

[0177] When multiple command conflicts are detected, the execution command is selected based on user identity permissions or operation priority;

[0178] Conflict determination rule base:

[0179] Define conflict scenario types and resolution strategies (e.g. Figure 7 shown);

[0180] Hybrid trigger mechanism:

[0181] When multiple commands can be executed in parallel (such as two devices controlling the video progress and volume respectively), start the command splitting module:

[0182] By dividing the interface area, touch commands are distributed to the target controls (for example, the left screen area is mapped to a progress bar, and the right side is mapped to a volume bar);

[0183] Enables parallel thread processing for non-conflicting instructions, reducing overall latency.

[0184] Multi-dimensional permission model:

[0185] Dynamically assign permission levels based on user role (presenter / participant), device type (mobile phone / VR controller), and biometrics (fingerprint / face recognition);

[0186] In educational scenarios, the teacher's device automatically obtains forced takeover permissions, interrupting incorrect operation instructions on the student's device;

[0187] Conflict Visual Feedback:

[0188] Display the source device icon and operation intention of the conflicting command on the screen sharing interface (such as "Device A is trying to delete a file, and device B is editing it");

[0189] A manual arbitration interface is provided, allowing users to adjust arbitration results in real time through voice commands (such as "allow device A to execute").

[0190] This embodiment achieves the following technical breakthroughs by refining the multi-source instruction integration and conflict handling mechanism:

[0191] Efficient multi-user collaborative operation

[0192] Timestamp synchronization accuracy: The timing error of cross-device operations is controlled within 5ms, ensuring that the handwriting synchronization rate is increased to 99% when multi-user collaborative drawing (such as Miro whiteboard);

[0193] Hybrid triggering mechanism: In video editing scenarios, the director can adjust the timeline using an iPad while the photographer fine-tunes filter parameters using an Android phone, increasing parallel execution efficiency by 40%.

[0194] Intelligent conflict resolution capabilities

[0195] Dynamic permission model: In medical consultation scenarios, the priority of CT image annotation instructions issued by chief physicians' devices is increased by 300%, and the interception rate of incorrect operations reaches 95%.

[0196] Regional diversion strategy: In the stock trading system, multi-device commands are mapped according to screen partitions (such as buy orders on the left and sell orders on the right), reducing the conflict rate by 80%;

[0197] Enhance system robustness and user experience

[0198] Double-buffered queue design: Even if a single device's instructions increase suddenly (for example, a game controller sends 50 instructions per second), the system response delay remains stable below 20ms;

[0199] Visual feedback: User satisfaction with perceived conflicts increased by 65%. This is particularly true in education, where teachers can visually detect student device misuse and correct it immediately.

[0200] Cross-platform compatibility extension

[0201] Supports mixed input of smartwatch micro-gestures (such as Huawei Watch's fist slide) and VR controller spatial operations. In the metaverse meeting scenario, the multimodal command recognition accuracy rate reaches 92%;

[0202] Through standardized protocol conversion, command coordination between old devices (such as Windows 7 touch screens) and modern devices is achieved, and the reuse rate of historical devices is increased by 50%.

[0203] In this embodiment, the dynamic mapping in S3 further includes:

[0204] Monitor changes in the target device's interface status in real time. If the interface update causes the control identifier to become invalid, re-parsing of the semantic layer instructions will be triggered.

[0205] Interface change detection mechanism:

[0206] Adopting dual-thread monitoring architecture:

[0207] Main thread: monitors interface element change events (such as control addition and deletion, and property updates) through the target device's system API (such as Android's AccessibilityService or iOS's UIAutomation);

[0208] Auxiliary thread: regularly captures screen images (frequency 10Hz) and detects changes in interface layout (such as pop-up window appearance and page jump) through image difference algorithm.

[0209] When an interface update is detected, the control tree reconstruction is triggered and the validity of the identifiers of the affected controls is marked (for example, whether the button ID still exists).

[0210] Intelligent recovery of invalid controls:

[0211] If the target control identifier is invalid (for example, the original "Confirm button" is replaced by a "Submit button"):

[0212] Based on the control function similarity matching algorithm, search for semantically similar alternative controls in the control tree (for example, based on the text content "Confirm" → "Submit" or proximity in location);

[0213] If there is no matching control, the semantic backtracking mechanism is activated: the expected operation target is inferred based on the user's historical operation records (such as the operation path after the last five clicks of "Confirm");

[0214] Perform a secondary check on the re-parsed intermediate semantic layer instructions to ensure that the mapped native instructions conform to the current interface state of the target device.

[0215] When the target device is a virtual reality device, the touch commands are mapped into three-dimensional space interaction events, and the display content of the virtual scene and the physical screen are synchronized;

[0216] Spatial interaction event generation:

[0217] Coordinate system conversion:

[0218] Convert the source device's 2D touch coordinates (x, y) to the VR device's 3D coordinates (x, y, z), where the z-axis depth is dynamically calculated based on the touch pressure or gesture type (e.g., light touch z = 0.5m, heavy press z = 1.2m).

[0219] Establish a spatial mapping relationship:

[0220]

[0221] (where k is the scaling factor and offset is the screen alignment offset).

[0222] Gesture-action mapping library:

[0223] Define the correspondence between touch gestures and 3D interactive actions in VR scenes (such as Figure 8 shown);

[0224] Physical-virtual simultaneous rendering:

[0225] The touch track of the device on the same screen is superimposed on the VR headset (such as a translucent halo effect) to ensure that the user perceives the real-time linkage between the touch operation and the virtual scene.

[0226] Machine learning optimized mapping strategy

[0227] Dynamic parameter tuning:

[0228] Collect user adjustment data on VR mapping results (such as manually correcting object rotation angles) to build a training dataset;

[0229] Train an LSTM neural network model to learn the user's spatial manipulation preferences (such as z-axis depth sensitivity and rotation speed);

[0230] When similar operating scenarios are detected, the model is automatically called to predict the optimal mapping parameters, reducing the number of manual calibrations.

[0231] This embodiment refines the implementation logic of dynamic mapping and VR space interaction, resulting in the following technical improvements:

[0232] Real-time responsiveness to interface updates

[0233] Dual-thread monitoring mechanism: Interface change detection latency is reduced from 200ms in traditional solutions to 50ms. For example, in e-commerce app flash sales scenarios, the touch mapping success rate increases to 99.8% after the buy button is updated.

[0234] Semantic backtracking mechanism: In the dynamic forms of financial apps, even if the control ID changes (such as "Buy" → "Trade"), the user's operation intention is accurately restored 92% of the time.

[0235] Breakthrough in virtual reality interaction accuracy

[0236] Dynamic calculation of spatial coordinates: In VR architectural design scenarios, the 2D drawing trajectory of the tablet stylus is mapped to the 3D model wireframe with an error of ≤0.5mm, meeting professional design requirements.

[0237] Gesture-Action Mapping Library: In VR educational applications, the accuracy of gesture recognition has increased to 97% (for example, the false trigger rate of the "pouring liquid" gesture in a chemistry experiment has been reduced from 15% to 2%).

[0238] Resource consumption optimization

[0239] Intelligent recovery of invalid controls: reduces the number of complete semantic parsing times caused by interface updates, reducing CPU peak load by 35% (from 60% → 39%);

[0240] LSTM parameter tuning: The frequency of manual calibration of VR space mapping has been reduced by 70%, significantly reducing user learning costs.

[0241] Cross-dimensional operational consistency

[0242] In mixed reality (MR) scenarios, mobile phone touch controls can simultaneously manipulate the physical screen (such as adjusting a PowerPoint presentation) and the virtual holographic projection (such as rotating a 3D model), with an operation synchronization error of less than 10ms.

[0243] For foldable / scrollable screen devices, the interface layout changes are automatically adapted after the screen is unfolded, and the touch command mapping accuracy remains above 99.5%.

[0244] In this embodiment, the method further includes a conflict handling mechanism:

[0245] When the touch protocols of the source device and the target device are incompatible, simulated touch events are generated, including virtual clicks, sliding track simulation, and time compensation for long press events;

[0246] Protocol compatibility detection and conflict classification

[0247] Protocol feature comparison:

[0248] Extract the protocol characteristics of the source device touch command (such as Android's MotionEvent event structure and Windows' WM_TOUCH message format);

[0249] Compare field by field with the protocol specifications supported by the target device to identify incompatible fields (e.g., the force pressure value field in iOS is missing in the Android protocol);

[0250] Conflict severity is classified into:

[0251] Level 1: Fields are missing but can be simulated (e.g. pressure value → long press duration);

[0252] Level 2: event types have no equivalent mapping (such as 3D Touch Peek gesture);

[0253] Level 3: Multiple event logic conflicts (such as two-finger rotation and single-finger drag on the target device).

[0254] Simulate touch event generation

[0255] Virtual click event generation:

[0256] When it is detected that the target device does not support the click event type of the source device (such as Windows right click corresponds to Android long press):

[0257] Parsing click intent (e.g., "open context menu");

[0258] Generates an equivalent sequence of operations based on the target device control type:

[0259] If the target is an Android list item, a simulated event chain is generated: long press for 500ms → pop-up menu;

[0260] If the target is an iOS text box, the mapping logic of double-clicking → selecting all text is generated.

[0261] Sliding trajectory simulation:

[0262] Dynamic interpolation of high-precision trajectories (such as Surface Pen handwriting):

[0263] Downsample the original track points (sampling rate 120Hz) to the maximum frequency supported by the target device (e.g. 60Hz for Android devices);

[0264] Optimize trajectory smoothness through Bezier curve fitting to ensure handwriting restoration error ≤ 0.3mm;

[0265] Added inertial sliding compensation (like the inertial scrolling feature of UIScrollView in iOS).

[0266] Time compensation mechanism:

[0267] When the touch response delay of the target device is higher than the threshold (for example, >80ms):

[0268] Predict the target device's processing queue status and send instructions in advance;

[0269] Introducing dynamic timestamp offset correction (e.g. Android device clock offset compensation);

[0270] In gaming scenarios, enable frame synchronization compensation (dynamically adjust the timing of command sending based on the refresh rate).

[0271] Return the execution results to the source device through the feedback channel to optimize the semantic parsing rules of subsequent instructions;

[0272] Feedback channel construction and rule optimization

[0273] Multi-dimensional feedback data collection:

[0274] The following execution results are captured in real time:

[0275] Target device response status (success / failure / partial execution);

[0276] The user manually corrects the action (such as clicking the target control again);

[0277] System performance indicators (latency, CPU usage, touch coordinate offset).

[0278] Root cause analysis and strategy iteration:

[0279] Establish a knowledge graph of conflict events, associating protocol conflict types, mapping schemes, and execution results;

[0280] Use decision tree algorithms (such as CART) to analyze failure causes (e.g., 60% of Level 2 conflicts are caused by gestures not having equivalent mappings);

[0281] Dynamically update the rule base:

[0282] For frequently failing scenarios (such as 3D Touch Peek), add alternative mapping solutions (such as a "hard press + side swipe" gesture combination);

[0283] On low-performance devices, the accuracy of simulation events is automatically degraded (for example, the trajectory interpolation algorithm switches from cubic Bezier to linear interpolation).

[0284] User behavior learning model:

[0285] Collect user correction data for simulated events (such as adjusted sliding speed and long press duration);

[0286] Train a random forest model to predict user preference mapping parameters;

[0287] When the same type of conflict occurs again, the user's historical preference solution will be called first.

[0288] This embodiment achieves the following technical breakthroughs by refining the protocol conflict handling and feedback optimization mechanism:

[0289] Comprehensive improvement in cross-protocol compatibility

[0290] Virtual Event Chain Generation: Improves the success rate of Windows right-click mapping on Android devices from 68% to 94%, especially for complex applications (such as AutoCAD Mobile).

[0291] Track restoration optimization: The handwriting offset of Surface Pen on iPad Pro is reduced from 1.2mm to 0.4mm, meeting professional drawing requirements.

[0292] Dynamic environment adaptability

[0293] Time compensation mechanism: In cloud gaming scenarios, touch delay caused by 5G network jitter is stabilized from 150ms to less than 50ms;

[0294] Downgrade strategy: The command packet loss rate of low-end devices (such as Android Go) is reduced by 70%, ensuring the smoothness of basic operations.

[0295] Enhanced user personalized experience

[0296] Behavioral learning model: Automatically optimizes the mapping between pen pressure and line thickness based on designer habits, with personalized configuration loading speed of <100ms;

[0297] Visual correction interface: users can manually adjust simulation event parameters (such as long press duration), and the correction data is synchronously fed back to the rule base;

[0298] System robustness is significantly improved

[0299] Root Cause Analysis Engine: The average resolution time for conflict events has been reduced from 30 seconds to 3 seconds, reducing operation and maintenance costs by 90%;

[0300] Knowledge graph iteration: After the integration of new device protocols (such as HarmonyOS), the adaptive learning cycle is shortened from 2 weeks to 48 hours.

[0301] In this embodiment, the mapping method supports cross-platform protocol conversion, including:

[0302] Convert mouse events of Windows system to MotionEvent events of Android system;

[0303] Protocol structure analysis and mapping:

[0304] Event type matching:

[0305] Map Windows' WM_LBUTTONDOWN (left button pressed) to Android's ACTION_DOWN;

[0306] WM_MOUSEMOVE (mouse movement) is split into consecutive ACTION_MOVE events, and the movement step size is calculated based on the target device screen density (for example, each pixel movement on a 4K screen is mapped to 2 ACTION_MOVE events);

[0307] WM_RBUTTONUP (right button release) is converted to the Android long press event ACTION_LONG_PRESS, with a default duration of 500ms, which can be adjusted dynamically.

[0308] Coordinate transformation algorithm:

[0309] Calculate scaling based on screen resolution differences:

[0310]

[0311] For high-precision devices (such as Surface Studio), enable sub-pixel interpolation algorithm to improve trajectory smoothness.

[0312] Special handling of scroll wheel events:

[0313] Convert Windows' WM_MOUSEWHEEL event to Android's MotionEvent.ACTION_SCROLL:

[0314] Parse the scroll wheel delta value (Delta) and convert it proportionally to the scroll distance (e.g. Delta = 120 corresponds to scrolling 300dp);

[0315] Dynamically adjust scroll inertia parameters (such as friction coefficient and maximum speed) according to the target control type (such as ListView / WebView).

[0316] Map iOS Force Touch gestures to long press events on Android, and dynamically adjust mapping parameters based on pressure values;

[0317] Dynamic quantitative classification of pressure values:

[0318] Collect touch pressure data of iOS devices (0-6.666N range), divided into 5 pressure levels (such as Figure 9 shown).

[0319] Dynamically adjust parameters based on the hardware characteristics of the target device:

[0320] If the Android device supports a pressure-sensitive screen (such as the Samsung Galaxy Note series), the pressure value is directly transmitted;

[0321] If the device does not have pressure sensitivity, pressure feedback is simulated by long press duration + vibration intensity (for example, Level 4 is mapped to 800ms long press + 3 levels of vibration).

[0322] 3D touch (Peek and Pop) gesture processing:

[0323] Peek gesture recognition: detects when the pressure reaches a threshold (3.0N) and lasts for more than 200ms, and maps it to an Android ContextMenu popup.

[0324] Pop gesture conversion: Based on the Peek gesture, continue to increase the pressure to 5.0N, convert it to an ACTION_CLICK event, and trigger a depth jump (such as opening the link details page).

[0325] Dynamic parameter optimization engine

[0326] Scenario Adaptation Strategy:

[0327] Establish a device performance profile library to record parameters such as touch response delay and screen refresh rate of different devices;

[0328] Dynamically adjust mapping rules:

[0329] On low refresh rate devices (such as 60Hz screens), reduce the frequency of sending ACTION_MOVE events (from 120Hz to 60Hz);

[0330] In the game scenario, enable "Turbo mode" to bypass the Android event queue and directly inject touch commands, compressing the delay from 80ms to 20ms.

[0331] User behavior learning module:

[0332] Collect statistics on users’ modification actions on mapping results (such as manually extending the long press time);

[0333] Use a linear regression model to predict personalized parameters (for example, user A prefers to map Level 3 pressure to a 700ms long press) and update the local rule base.

[0334] This embodiment achieves the following technical breakthroughs by refining the differentiated processing logic for cross-platform protocol conversion:

[0335] Cross-system operation consistency is greatly improved

[0336] Dynamic pressure-duration mapping: The accuracy of Force Touch intent interpretation on iOS devices has increased from 65% to 93% on Android devices. For example, the success rate of re-pressing to preview an attachment in the Mail app has increased to 89%.

[0337] High-precision trajectory retention: The jitter rate of Windows stylus writing in Android drawing software (such as Krita) is reduced to 2%, meeting professional-level input requirements.

[0338] Enhanced adaptability to complex scenarios

[0339] Game scene optimization: The right-click zoom operation of the Windows mouse is mapped to an Android long press, shortening the response speed from 120ms to 35ms, meeting the competitive requirements of FPS mobile games;

[0340] Folding screen adaptation: Dynamically adjust the coordinate mapping algorithm according to the folding state of the device (such as enabling dual-screen partition mapping when Surface Duo is unfolded), and the control click accuracy remains above 99%.

[0341] Resource consumption and compatibility optimization

[0342] Sub-pixel interpolation algorithm: On 8K resolution devices, CPU usage dropped from 42% to 28%, and memory consumption was reduced by 35%;

[0343] Support for legacy devices: The mouse protocol of Windows XP can be seamlessly converted to Android 14, reducing the cost of reusing historical devices by 70%.

[0344] User experience personalization

[0345] Pressure feedback simulation: By grading vibration intensity (e.g., Level 4 triggers three short vibrations), users' confidence in a successful hard press is increased by 80%;

[0346] Behavioral learning model: The matching degree of pen pressure-line thickness mapping for designer users has been improved to 98%, and personalized configuration loading takes less than 50ms.

[0347] In this embodiment, the dynamic adaptation rule base is optimized through machine learning:

[0348] Collect users' correction operations on mapping results and generate training data sets;

[0349] User behavior data collection and annotation

[0350] Multimodal data capture:

[0351] Real-time recording of user interaction data with the device, including:

[0352] Touch correction behavior: The user manually adjusts the mapped control selection (such as re-clicking the target button) and slide distance correction (such as dragging the progress bar to the correct position);

[0353] Semantic conflict logs: Control identification failure events detected by the system and cases of touch intent misjudgment (such as identifying "zoom" as "page turn");

[0354] Environmental context data: screen resolution of the target device, currently running application processes, and network latency status.

[0355] Clean and label the raw data:

[0356] Remove invalid data (such as short clicks caused by accidental touches);

[0357] Add semantic labels to each correction operation (such as "control positioning error" or "pressure value mapping deviation").

[0358] Update the mapping strategy in the rule base based on the neural network model to improve the accuracy of semantic parsing;

[0359] Hybrid model training and feature engineering

[0360] Deep Feature Extraction:

[0361] Control-level graph neural network (GNN) modeling:

[0362] Convert the target device interface control tree into a graph structure, where nodes are control attributes (type, position, text) and edges are parent-child hierarchical relationships;

[0363] Use the GraphSAGE model to learn the embedding vectors of control nodes to capture their functional semantics (such as the association between the "shopping cart button" and the "checkout button").

[0364] Touch timing feature encoding:

[0365] The time series data of touch gestures (coordinates, pressure value changes) are input into a bidirectional LSTM network to extract the temporal features of the operation intention (such as the difference between "quick double-click" and "slow double-click").

[0366] Multi-task learning framework:

[0367] Jointly train two tasks:

[0368] Control matching task: predict the target control identity after the user's correction (classification task);

[0369] Parameter optimization task: regression prediction of optimal mapping parameters (such as long press duration and sliding speed compensation value).

[0370] Loss function design:

[0371]

[0372] Dynamic update of rule base and A / B testing

[0373] Incremental update strategy:

[0374] Real-time hot update:

[0375] When the model confidence is >90%, the new rule is directly inserted into the rule base in memory (such as the new iOS 17 side swipe back gesture mapping);

[0376] Version control ensures atomic operations and avoids rule conflicts during updates.

[0377] Shadow mode verification:

[0378] Run the old and new rule bases in parallel and compare the execution results (such as mapping success rate and latency indicators). The new rule will only take effect when it significantly improves (such as success rate +5%).

[0379] Scenario-based rule grouping:

[0380] Create independent rule subsets based on application scenarios (gaming, office, education), for example:

[0381] Game scene rule group: prioritize touch response speed and allow for reduced trajectory smoothness;

[0382] Document Editing Rule Group: Emphasizes precise coordinate mapping, sacrificing some latency for high precision.

[0383] Edge computing optimization

[0384] Distributed model reasoning:

[0385] Deploy a lightweight inference engine (TensorFlow Lite) on user devices to process 90% of high-frequency requests in real time;

[0386] Complex scenarios (such as VR space mapping) require high-performance cloud-based models (such as GPU-accelerated BERT) to protect user data through differential privacy technology.

[0387] This embodiment achieves the following technological breakthroughs through the deep integration of machine learning and rule base:

[0388] Semantic parsing accuracy has increased significantly

[0389] GNN control matching: The recognition accuracy of dynamically generated controls has increased from 78% based on traditional rules to 96%. For example, the success rate of clicking on flash sale buttons on e-commerce apps has increased to 99.3%.

[0390] Multi-task learning optimization: The mapping error of the long press duration parameter is compressed from ±150ms to ±20ms, with outstanding performance in cross-platform scenarios across iOS and Android.

[0391] Adaptive capabilities have been greatly enhanced

[0392] Incremental hot updates: The adaptation cycle for new operating system versions (such as Android 14) is shortened from 2 weeks to 8 hours;

[0393] Scenario-based rule group: Touch response delay in game scenarios is reduced by 40% (from 50ms to 30ms), and the combo success rate in MOBA games is increased by 25%.

[0394] Operation and maintenance costs are significantly reduced

[0395] Shadow mode verification: The system crash rate caused by incorrect rule pushes dropped to 0.03%, reducing the workload of the operations team by 70%;

[0396] Edge computing architecture: Cloud computing costs are reduced by 65%, and the risk of user privacy data leakage is reduced by 90%.

[0397] Intelligent upgrade of user experience

[0398] Personalized mapping strategy: Automatically optimizes pen thickness and curves based on the designer's pen usage habits, improving creative efficiency by 30%;

[0399] Cross-device collaborative learning: Mapping rules modified by users on iPad can be synchronized to Windows devices, increasing satisfaction with multi-device consistency by 80%.

[0400] In this embodiment, the mapping method supports multi-window collaborative control:

[0401] When the target device displays multiple application windows in split screen, the command is mapped to the corresponding application process according to the window area to which the touch position belongs;

[0402] Dynamic perception of split-screen status and hotspot modeling

[0403] Multi-window topology scanning:

[0404] Obtain the current active window list and hierarchical relationship through the target device's system-level interface (such as Android's ActivityManager or Windows' Win32API);

[0405] Build a split-screen hot zone matrix and record the screen coordinate range of each window (for example, window A occupies the left half of the screen [0,0]-[540,2400], and window B occupies the right half of the screen [540,0]-[1080,2400]);

[0406] For irregular split screens (such as picture-in-picture and free-floating windows), image recognition technology is used to extract window outlines and establish a polygonal hot zone coordinate model.

[0407] Dynamic hotspot adjustment:

[0408] When a window size change is detected (such as the user dragging the dividing line):

[0409] Update the hot zone matrix in real time and calculate the overlap ratio of new and old hot zones;

[0410] Compensate for coordinate offsets of affected touch commands (for example, if the window width is reduced by 20%, the original touch point x coordinate is scaled by a factor of 0.8).

[0411] When the foldable screen device is unfolded, adjacent hot areas are automatically merged into an extended operation plane.

[0412] Targeted distribution of touch commands

[0413] Process-level instruction routing:

[0414] Match the target window hotspot according to the touch coordinates and extract the application process ID bound to the window (such as WeChat process ID 3327);

[0415] Inject native instructions into the target process event queue through cross-process communication (Binder for Android / COM+ for Windows);

[0416] For operations that require cross-window linkage (such as dragging a file from window A to window B), enable the dual-channel synchronization engine:

[0417] Channel 1: Sends ACTION_DOWN and ACTION_MOVE events to window A;

[0418] Channel 2: When the touch point enters the hot area of ​​window B, an ACTION_DROP event is sent to window B and the data handle is passed.

[0419] Intelligent division of sub-interaction areas:

[0420] Based on window content semantic analysis (OCR recognition + control type detection), complex windows are divided into functional sub-areas:

[0421] Example: The video playback window is divided into the "control bar area" (pause / progress) and the "main screen area" (double-click to full screen);

[0422] Set independent event filtering rules for each sub-area:

[0423] Disable the long-press gesture for text selection in the Document Editing subarea to prevent accidental touches.

[0424] Enable pinch-to-zoom and rotate gestures in the Chart Display Subarea.

[0425] When the display interface of the same screen device is a multi-window layout, cross-window command distribution is achieved by dividing the sub-interaction area;

[0426] Enhanced cross-window collaboration

[0427] Data flow pipeline construction:

[0428] Establish a cross-application data sharing channel (such as Android's ContentProvider):

[0429] When a drag operation is detected that crosses the window boundary, the source window data (such as image URI) is automatically encapsulated as a system clipboard object;

[0430] When the hot area of ​​the target window is released, data parsing and rendering are triggered (such as inserting an image into a PPT).

[0431] Operational continuity assurance:

[0432] Use a global state machine to track cross-window operation processes (such as drag status and copy progress);

[0433] If the window is closed or switched midway, the intermediate state is automatically saved and a recovery entry is provided.

[0434] Multi-user collaboration mode:

[0435] In a conference scenario, assign independent user control rights to each sub-window:

[0436] User A controls the Excel window on the left using a mobile phone, while user B controls the PPT window on the right using a tablet.

[0437] Limit the scope of cross-window operations through permission labels (such as "Read-only" and "Edit").

[0438] Conflict arbitration mechanism:

[0439] When multiple users operate the same sub-area at the same time, automatic arbitration is performed based on identity priority (host > participant) or operation type (edit > view);

[0440] Conflicting operations are visually marked (such as a flashing red border) and support manual intervention and adjustment.

[0441] This embodiment achieves the following technical improvements by refining the multi-window collaborative control logic:

[0442] Multitasking efficiency doubled

[0443] Hotspot dynamic compensation: When the foldable screen is unfolded, the touch mapping accuracy of the stock trading app's split-screen candlestick chart and trading window remains 99.8%, and the operation delay is less than 15ms;

[0444] Cross-process drag and drop optimization: The success rate of dragging links from the Chrome browser to the WeChat window increased from 75% to 98%, and the drag and drop time was shortened from 2.1s to 0.3s.

[0445] Breakthrough in the accuracy of complex interface interactions

[0446] Sub-area semantic segmentation: In Photoshop split-screen scenarios, the false trigger rate of brush operations in the left canvas area and sliding and zooming in the right toolbar is reduced to 1.2%;

[0447] Hierarchical authority control: In the medical consultation system, interns can only mark sub-areas of images, while chief physicians have global editing rights, and the authority violation interception rate is 100%.

[0448] Optimize system resource consumption

[0449] Dual-channel synchronization engine: Memory usage for multi-window linkage is reduced by 40% (from 220MB to 132MB), significantly improving performance on low-end devices.

[0450] State machine management: The recovery loading time after cross-window operation interruption is shortened from 8s to 1.5s, reducing the risk of data loss by 90%.

[0451] Multi-user collaboration experience upgrade

[0452] Independent control rights allocation: In online education scenarios, the delay difference between teachers and students operating different sub-windows simultaneously is less than 10ms, and the efficiency of collaborative answering is improved by 50%;

[0453] Conflict visualization prompts: The speed of recognizing operational conflicts during team collaboration is increased by 3 times, and the time spent on arbitration decisions is reduced by 70%.

[0454] In this embodiment, the mapping method also includes a device switching mechanism:

[0455] When the target device is detected to be switched, the context state of the current intermediate semantic layer instruction is retained;

[0456] Device switching event triggering and status snapshot

[0457] Seamless switching detection:

[0458] The device connection manager monitors the target device status (such as Bluetooth signal strength and Wi-Fi Direct connection stability) in real time and triggers the switch when the following events are detected:

[0459] Active switching: The user manually selects a new target device (e.g., switching from a projector to a VR headset);

[0460] Passive switching: The original target device is disconnected and times out (for example, the network delay is greater than 500ms for 3 seconds).

[0461] Contextual state capture:

[0462] Extract the complete context of the current intermediate semantic layer instruction, including:

[0463] Operation chain: a sequence of touch commands that have not been completed (such as long-pressing and dragging and releasing in the middle);

[0464] Interface snapshot: the last valid interface screenshot and control tree structure of the target device;

[0465] Dynamic variables: temporarily generated control identifiers and mapping parameter adjustment records.

[0466] The incremental serialization technology is used to compress the status data and store it in the non-volatile memory of the same-screen device (such as eMMC flash memory).

[0467] Cross-protocol state migration and adaptation

[0468] Protocol Difference Matrix Matching:

[0469] Build a protocol compatibility matrix to record the protocol differences between the old and new target devices (such as Figure 10 shown);

[0470] Dynamic parameter correction:

[0471] Runtime adaptation of device hardware-dependent parameters (such as screen refresh rate):

[0472] ;

[0473] If the control tree structure difference between the new and old devices is greater than 30%, the semantic backtracking engine is started: the target control is relocated based on the OCR result of the interface snapshot (such as the "Save button").

[0474] Remap commands and restore operational flow continuity based on the platform protocol of the new target device;

[0475] Operation chain recovery and continuity assurance

[0476] Resume of interrupted operation:

[0477] Parse the stored intermediate semantic layer instructions and rebuild the native instruction queue according to the new target device protocol:

[0478] Example: Converting an unfinished ACTION_MOVE sequence on Android to a continuous UIPanGestureRecognizer event on iOS.

[0479] Enable timeline compression for time-sensitive operations (such as game combos): accelerate the execution of unfinished instructions to the current timestamp.

[0480] Status consistency check:

[0481] Compare the interface snapshot with the new device's current interface (such as control position offset). If the difference exceeds a threshold (such as >10% pixel deviation):

[0482] Trigger progressive calibration: gradually align the target control by fine-tuning the touch coordinates (Δx, Δy);

[0483] If calibration fails, a user guidance interface is launched (with expected control locations highlighted).

[0484] Cross-device operation history synchronization:

[0485] Synchronize the operation records before the switch (such as the last five clicks) to the new device for machine learning model warm-up;

[0486] In distributed device groups (such as smart home multi-screen), operation history roaming is supported, and users can continue unfinished tasks on any device.

[0487] This embodiment achieves the following technical breakthroughs by refining the context migration and protocol adaptation logic of device switching:

[0488] Seamless switching experience:

[0489] Incremental serialization compression: Switching time is reduced from 30 seconds in the traditional solution to 0.5 seconds, and the state data size is reduced by 85% (from 2MB to 300KB);

[0490] Timeline Compression: The success rate of continuing unfinished combos in game scenes has been increased from 55% to 98%, and the skill release delay is less than 20ms.

[0491] Precise cross-protocol adaptation

[0492] Coordinate mirror flipping: The touch coordinate error after switching from Android to iOS is reduced from ±15 pixels to ±2 pixels;

[0493] Semantic backtracking engine: After the interface is revised (such as WeChat 8.0→9.0), the positioning accuracy of core functional controls (such as the "Send button") remains at 99.5%.

[0494] Enhanced robustness in complex scenarios

[0495] Progressive calibration: When switching between unfolded and folded states, the control drift is automatically corrected with a success rate of 93%;

[0496] Operation history roaming: The efficiency of task continuation across device groups (mobile phone → car machine → smart TV) is increased by 70%.

[0497] Resource optimization and compatibility expansion

[0498] Non-volatile storage: The state recovery time after power failure is shortened from 10 seconds to 1 second, and the risk of data loss is close to zero;

[0499] Protocol Difference Matrix: Supports rapid access to emerging systems such as HarmonyOS and Fuchsia, shortening the adaptation period from 3 months to 2 weeks.

[0500] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0501] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made by using the contents of the present description and drawings under the inventive concept of the present invention, or direct / indirect application in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A semantic parsing and mapping method for cross-platform touch commands on the same screen, characterized in that: include: S1: Acquire a touch command from the source device through the screen sharing device, where the touch command includes touch position, touch gesture type, and touch timing information; S2: Based on the operating system type and application scenario of the target device, semantically analyze the touch control instruction to generate an intermediate semantic layer instruction, where the intermediate semantic layer instruction includes operation logic, interaction intent, and target control identifier; S3: Dynamically mapping the intermediate semantic layer instructions to native instructions of the target device according to the platform protocol and interaction rules of the target device, wherein the native instructions are adapted to the touch event processing mechanism of the target device; S4: Send the native command to the target device for execution, and synchronously update the display interface of the screen controller and the display content of the target device; The step S2 also includes multi-level semantic analysis: The first level analyzes the physical parameters of touch gestures, including sliding direction, pressure value, and multi-touch coordinates; The second level combines the current display interface layout of the target device to identify the functional modules corresponding to the touch operation; The third level dynamically adjusts the semantic weight according to the application scenario, giving priority to mapping high-frequency operation instructions.

2. The semantic parsing and mapping method for cross-platform touch control instructions of a same-screen device according to claim 1, characterized in that: The semantic analysis in S2 further includes: Matching the target device's interaction logic through a dynamic adaptation rule base that includes operating system protocols, application interface control types, and touch event priorities for different platforms; When it is detected that the control type of the target device is a dynamically generated control, the intermediate semantic layer instructions are optimized based on the control hierarchy and context semantics.

3. The semantic parsing and mapping method for cross-platform touch control instructions of a same-screen device according to claim 1, characterized in that: The mapping method supports the integration of multi-source touch commands, including: Receive touch commands from multiple source devices and sort them by timestamps and event queues; When multiple command conflicts are detected, the execution command is selected based on user identity permissions or operation priority.

4. The semantic parsing and mapping method for cross-platform touch control instructions of a same-screen device according to claim 1, characterized in that: The dynamic mapping in S3 further includes: Monitor changes in the target device's interface status in real time. If the interface update causes the control identifier to become invalid, re-parsing of the semantic layer instructions will be triggered. When the target device is a virtual reality device, the touch commands are mapped into three-dimensional space interaction events, and the display content of the virtual scene and the physical screen are synchronized.

5. The semantic parsing and mapping method for cross-platform touch control instructions of a same-screen device according to claim 1, characterized in that: The method also includes a conflict handling mechanism: When the touch protocols of the source device and the target device are incompatible, simulated touch events are generated, including virtual clicks, sliding track simulation, and time compensation for long press events; The execution results are returned to the source device through the feedback channel to optimize the semantic parsing rules of subsequent instructions.

6. The semantic parsing and mapping method for cross-platform touch control instructions of a same-screen device according to claim 1, characterized in that: The mapping method supports cross-platform protocol conversion, including: Convert mouse events of Windows system to MotionEvent events of Android system; Map iOS Force Touch gestures to long press events on Android, and dynamically adjust mapping parameters based on pressure values.

7. The semantic parsing and mapping method for cross-platform touch control instructions of a same-screen device according to claim 2, characterized in that: The dynamic adaptation rule base is optimized through machine learning: Collect users' correction operations on mapping results and generate training data sets; Based on the neural network model, the mapping strategy in the rule base is updated to improve the accuracy of semantic parsing.

8. The semantic parsing and mapping method for cross-platform touch control commands of a same-screen device according to claim 1, characterized in that: The mapping method supports multi-window collaborative control: When the target device displays multiple application windows in split screen, the command is mapped to the corresponding application process according to the window area to which the touch position belongs; When the display interface of the same screen device is a multi-window layout, cross-window command distribution is achieved by dividing the sub-interaction area.

9. The semantic parsing and mapping method for cross-platform touch control instructions of a same-screen device according to claim 1, characterized in that: The mapping method also includes a device switching mechanism: When the target device is detected to be switched, the context state of the current intermediate semantic layer instruction is retained; Remaps commands and restores continuity of operational flow based on the platform protocol of the new target device.

Images