Docks-based multi-mode interaction control method, system, medium and product

Abstract

The application discloses a multi-mode interaction control method, system, medium and product based on a docking station, and relates to the field of docking stations.In the method, page layout information of a display screen of the docking station is acquired, and structured user interface metadata is generated; a preset cooperative activation area is generated based on the user interface metadata; if a cursor of a pointing device in the preset cooperative activation area meets a cooperative control triggering condition, a cooperative control mode is triggered, two-dimensional coordinates of the cursor are mapped into target coordinates in a display screen coordinate system of the docking station; the target coordinates are sent to the docking station as an instruction, so that the docking station displays a proxy cursor at the target coordinates; if it is determined that the cursor of the pointing device triggers a preset interaction triggering event, the proxy cursor is controlled to execute the preset interaction triggering event, so that the docking station executes an operation corresponding to the preset interaction triggering event, and cross-device interaction is completed.

Description

Technical Field

[0001] This application relates to the field of docking stations, specifically to multi-mode interactive control methods, systems, media, and products based on docking stations. Background Technology

[0002] With the increasing popularity of mobile work and thin and light laptops, docking stations, as key hubs for connecting external devices, are experiencing growing market demand. Traditional docking stations have limited functionality, typically only providing limited operational status feedback via LED indicators. To configure them—such as switching video output ports, adjusting resolution, or checking network status—users must navigate through the operating system's settings interface of the connected host device, such as a laptop, requiring cumbersome adjustments. This indirect control method, reliant on the host system, is not only lengthy and inefficient, but also severely impacts user experience in certain scenarios, such as when users are unfamiliar with the settings, making docking station management and use lack intuitiveness.

[0003] To address the inconvenience of configuring via the host operating system, an improved solution has emerged: integrating a display screen, typically a touchscreen, onto the docking station itself. This design allows the docking station to have an independent user interface, enabling users to directly view device connection status, network speed, output parameters, and other information on the docking station's screen. Users can also directly switch functions or set parameters via touch. This approach returns control to the docking station itself, achieving localized direct interaction. Compared to configuration via the host, it significantly improves the directness of operation and the immediacy of feedback, simplifying the user's configuration process.

[0004] However, while the aforementioned docking stations with touchscreen displays optimize the interactive experience to some extent, they also introduce a new technical problem. In typical usage scenarios, users' hands and attention are primarily focused on the keyboard and mouse of the host device—the most frequent and primary input areas. When users need to interact with the docking station's display, they must remove their hands from the host's keyboard or mouse, reach for the physically separate docking station, complete the touch operation on its limited-size screen, and then return their hands to the host's input device to continue their previous work. This process disrupts the user's continuous workflow, creating a sense of disjointed operation and efficiency loss, resulting in a poor user experience. Summary of the Invention

[0005] This application provides a multi-mode interactive control method, system, medium, and product based on a docking station, which enables cross-device interaction and improves the user experience.

[0006] The first aspect of this application provides a multi-mode interactive control method based on a docking station. The method includes: acquiring page layout information of the docking station's display screen and generating structured user interface metadata based on the page layout information; generating a preset collaborative activation area based on the user interface metadata; monitoring whether a cursor from a pointing device enters the preset collaborative activation area and determining whether the behavior of the cursor in the preset collaborative activation area satisfies collaborative control triggering conditions; if the cursor from the pointing device enters the preset collaborative activation area and its behavior in the preset collaborative activation area satisfies collaborative control triggering conditions, then triggering a collaborative control mode; in the collaborative control mode, mapping the two-dimensional coordinates of the cursor to target coordinates in the display screen coordinate system of the docking station based on the relative position of the cursor in the preset collaborative activation area and the user interface metadata; sending the target coordinates to the docking station to display a proxy cursor at the target coordinates, the movement trajectory of the proxy cursor being the same as the movement trajectory of the cursor from the pointing device in the preset collaborative activation area; if it is determined that the cursor from the pointing device triggers a preset interactive trigger event, then controlling the proxy cursor to execute the preset interactive trigger event, so that the docking station performs an operation corresponding to the preset interactive trigger event, completing cross-device interaction.

[0007] By adopting the above technical solution, the page layout information of the docking station display is obtained, and structured user interface metadata is generated based on the page layout information. This provides the necessary data foundation for subsequent collaborative activation area generation and coordinate mapping. Generating a preset collaborative activation area based on the user interface metadata provides a sensitive area for the cursor on the pointing device to trigger the collaborative control mode, reducing the user's operational difficulty. By monitoring whether the cursor on the pointing device enters the preset collaborative activation area and determining whether the cursor's behavior within the preset collaborative activation area meets the collaborative control trigger conditions, the user's cross-device interaction intent can be accurately identified. In collaborative control mode, based on the relative position of the cursor within the preset collaborative activation area and the user interface metadata, the cursor's two-dimensional coordinates are mapped to target coordinates in the docking station display coordinate system. The target coordinates are then sent to the docking station to display a proxy cursor, ensuring that the proxy cursor's movement trajectory is the same as the pointer device's cursor's movement trajectory within the preset collaborative activation area. This achieves seamless switching and synchronous control of the cursor between different devices. When the pointer device's cursor triggers a preset interaction trigger event, the proxy cursor is controlled to execute the preset interaction trigger event, causing the docking station to perform the corresponding operation, completing cross-device interaction and improving user operational efficiency and experience.

[0008] Optionally, generating structured user interface metadata based on the page layout information specifically includes: performing semantic segmentation on the page layout information to identify functional area boundaries, control types, and control hierarchy relationships, generating an initial interface tree structure; calculating the interactive weight value of each node based on the initial interface tree structure, wherein the interactive weight value is weighted according to the node's control type, node depth, number of sibling nodes, and interactive attributes of the parent node; marking nodes with interactive weight values ​​lower than a preset weight threshold as non-critical nodes, and merging any two adjacent non-critical nodes into an aggregate node to obtain an optimized interface tree structure; performing serialization processing on the optimized interface tree structure, assigning a corresponding identifier to each node, and establishing a mapping index table between the nodes to generate the user interface metadata.

[0009] By employing the aforementioned technical solution, semantic segmentation of page layout information is performed to identify functional area boundaries, control types, and control hierarchy relationships, generating an initial interface tree structure. This allows for a structured representation of the page layout at the semantic level. Based on this initial interface tree structure, the interactive weight value of each node is calculated, and weighted calculations are performed according to the node's control type, node depth, number of sibling nodes, and the interactive attributes of the parent node. This allows for a quantitative assessment of the interactive importance of each node. Nodes with interactive weight values ​​below a preset weight threshold are marked as non-critical nodes, and adjacent non-critical nodes are merged into aggregate nodes, resulting in an optimized interface tree structure. This reduces redundant information and improves data conciseness. Serialization processing of the optimized interface tree structure involves assigning a corresponding identifier to each node and establishing a mapping index table between nodes to generate user interface metadata. This transforms the tree structure into a linear structure that is easy to store and query, improving data accessibility and processing efficiency.

[0010] Optionally, generating a preset collaborative activation area based on the user interface metadata specifically includes: extracting the physical resolution and display scaling factor information of the docking station display from the user interface metadata, and calculating a rectangular area with the same aspect ratio as the docking station display based on the physical resolution and the display scaling factor, as the reference size of the preset collaborative activation area; obtaining historical movement data of the pointer device in the edge area of ​​the host screen, and calculating the historical average movement speed and historical average acceleration of the pointer device in the edge area of ​​the host screen based on the historical movement data; calculating the safety margin size of the preset collaborative activation area based on a preset safety margin adjustment formula, combined with the historical average movement speed and historical average acceleration; calibrating the reference size using the safety margin size as a calibration parameter to obtain the final size and final position coordinates of the preset collaborative activation area; when multiple docking stations are detected to be connected to the host device simultaneously, generating multiple corresponding preset collaborative activation areas based on the spatial placement relationship of the multiple docking stations in the physical world, wherein one preset collaborative activation area is generated for each docking station, and the spatial layout of each preset collaborative activation area on the host screen is the same as the spatial placement relationship of each docking station in the physical world.

[0011] By employing the above technical solution, the physical resolution and display scaling factor information of the docking station display are extracted from the user interface metadata. A rectangular area with the same aspect ratio as the docking station display is calculated based on these parameters and used as the reference size for the preset collaborative activation area. This ensures that the ratio of the preset collaborative activation area to the docking station display is consistent, improving the accuracy of cross-device interaction. Historical movement data of the pointer device in the edge area of ​​the host screen is acquired, and the historical average movement speed and acceleration of the pointer device in the edge area of ​​the host screen are calculated based on this historical movement data, objectively reflecting the user's actual operating habits. According to the preset safety margin adjustment formula, combined with the historical average movement speed and historical average acceleration, the safety margin size of the preset collaborative activation area is calculated. This safety margin size is then used as a calibration parameter to calibrate the reference size, obtaining the final size and position coordinates of the preset collaborative activation area. This approach satisfies user operating habits while avoiding accidental triggering of the collaborative control mode. When multiple docking stations are detected to be connected to the host device at the same time, multiple preset collaborative activation areas are generated according to the spatial arrangement of the multiple docking stations in the physical world. The spatial layout of each preset collaborative activation area on the host screen is the same as the spatial arrangement of the docking stations in the physical world. This provides users with an intuitive and cognitively consistent way to interact with multiple devices, reducing the cognitive burden on users.

[0012] Optionally, the step of calculating the safety margin of the preset collaborative activation area according to the preset safety margin adjustment formula, combined with the historical average movement speed and historical average acceleration, specifically includes: extracting the speed component corresponding to the historical average movement speed and calculating the speed standard deviation; calculating the braking distance of the cursor according to the historical average acceleration, wherein the braking distance is the square of the historical average movement speed divided by twice the historical average acceleration; multiplying the speed standard deviation by a preset false trigger coefficient to obtain a base margin value, multiplying the braking distance by a preset safety factor to obtain a dynamic margin value, and taking the larger value between the base margin value and the dynamic margin value as the safety margin size.

[0013] By employing the above technical solution, the velocity component corresponding to the historical average movement speed is extracted, and the velocity standard deviation is calculated. This allows for a quantitative description of the movement speed fluctuation of the cursor on the host screen edge area. The braking distance of the cursor is calculated based on the historical average acceleration, reflecting the shortest distance required for the cursor to move from a moving state to a stationary state. This distance can be used to estimate the risk of the cursor accidentally entering a preset collaborative activation area. Multiplying the velocity standard deviation by a preset false trigger coefficient yields the base margin value, and multiplying the braking distance by a preset safety factor yields the dynamic margin value. Taking the larger of the two values ​​as the safety margin size allows for a comprehensive consideration of the impact of velocity fluctuations and braking distance on the false trigger risk. This dynamically adjusts the safety margin of the preset collaborative activation area, reducing the false trigger rate while maximizing the effective interaction area, thus improving the stability and sensitivity of cross-device interaction.

[0014] Optionally, determining whether the behavior of the cursor of the pointer device in the preset collaborative activation area meets the collaborative control triggering condition specifically includes: establishing a cursor trajectory buffer queue, continuously collecting the screen coordinates of the cursor at a preset sampling frequency, calculating the cursor motion feature vector between adjacent sampling points, and storing the cursor motion feature vector in the cursor trajectory buffer queue; (the cursor motion feature vector includes displacement vector, velocity change rate, and motion direction angle); when the cursor is detected to enter the preset collaborative activation area for the first time, extracting historical trajectory features within a preset time window from the cursor trajectory buffer queue; inputting the historical trajectory features into a preset intent classification model to obtain a classification result, wherein the classification result is a fast swipe mode, an exploration mode, or a hover mode. The system selects between a stop mode and a precise positioning mode. If the classification result is the quick swipe mode, the current monitoring state is cleared and monitoring continues for the next entry event. If the classification result is the exploration hover mode or the precise positioning mode, a timer is started to record the cumulative dwell time of the cursor within the preset collaborative activation area. The cumulative dwell time is compared with a preset duration threshold. If the cumulative dwell time is greater than or equal to the preset duration threshold, the behavior of the cursor on the pointer device within the preset collaborative activation area is determined to meet the collaborative control triggering condition. If the cumulative dwell time is less than the preset duration threshold, the behavior of the cursor on the pointer device within the preset collaborative activation area does not meet the collaborative control triggering condition.

[0015] By adopting the above technical solution, a cursor trajectory buffer queue is established to continuously collect the cursor's screen coordinates at a preset sampling frequency and calculate the cursor motion feature vector between adjacent sampling points. This vector is stored in the cursor trajectory buffer queue, allowing for real-time recording of the cursor's motion trajectory information and providing data support for subsequent intent recognition. When the cursor first enters the preset collaborative activation area, historical trajectory features within a preset time window are extracted from the cursor trajectory buffer queue and input into a preset intent classification model for classification. Based on the cursor's motion characteristics before entering the preset collaborative activation area, the user's true intent can be inferred, helping to improve the triggering accuracy of the collaborative control mode. By identifying the fast swipe mode, the exploration hover mode, and the precise positioning mode, different interaction intents can be identified. The fast swipe mode may indicate that the user has no intention to interact with the docking station; the exploration hover mode may indicate that the user is considering whether to perform cross-device interaction; and the precise positioning mode may indicate that the user has a clear need for cross-device interaction. Adopting different response strategies based on the classification results can respect the user's intent while avoiding unnecessary switching of interaction modes, improving the system's usability. By recording the cumulative duration of the cursor's stay within the preset collaborative activation area and comparing it with a preset duration threshold, a more reliable criterion can be provided for triggering the collaborative control mode, avoiding judgment bias caused by simply relying on the cursor position or instantaneous state, and improving the stability of cross-device interaction.

[0016] Optionally, the step of inputting the historical trajectory features into a preset intent classification model to obtain classification results specifically includes: normalizing the historical trajectory features to obtain a standardized feature vector; inputting the standardized feature vector into a first discriminator, calculating the ratio of the average value of the velocity component in the historical trajectory features to a preset velocity threshold based on the first discriminator to obtain a velocity ratio, and calculating the ratio of the standard deviation of the direction angle change sequence in the historical trajectory features to a preset angle threshold to obtain an angle stability index; when the velocity ratio is greater than a preset value and the angle stability index is less than the preset value, outputting a first confidence level for the fast sweep mode; inputting the standardized feature vector into a second discriminator, and based on the second discriminator, classifying the historical trajectory features into a standardized feature vector; and inputting the standardized feature vector into a second discriminator to classify the historical trajectory features into a standardized feature vector. Linear regression analysis is performed on the velocity sequence in the historical trajectory features to obtain the velocity change slope, and the monotonicity index of the distance sequence from the trajectory point to the trajectory starting point in the historical trajectory features is calculated; when the velocity change slope is negative and the monotonicity index indicates that the distance sequence is decreasing, the second confidence score of the exploration hovering mode is output; the standardized feature vector is input into a third discriminator, and the trajectory shape corresponding to the historical trajectory sequence is matched with a preset micro-gesture template library based on the similarity of the third discriminator. When the similarity is greater than a preset similarity threshold, the third confidence score of the precise positioning mode is output; the first confidence score, the second confidence score, and the third confidence score are compared, and the mode with the highest confidence score is selected as the classification result.

[0017] By employing the above technical solution, the historical trajectory features are normalized to obtain standardized feature vectors, which eliminates the influence between indicators of different dimensions, facilitating unified measurement and comparison. Three discriminators are used to identify the fast swipe mode, the exploration hover mode, and the precise positioning mode, respectively, to fully extract the motion pattern information contained in the historical trajectory features, improving the comprehensiveness and accuracy of intent recognition. When identifying the fast swipe mode, by calculating the ratio of the average value of the velocity component in the historical trajectory features to a preset velocity threshold, and the ratio of the standard deviation of the direction angle change sequence to a preset angle threshold, it can be determined whether the cursor movement speed exceeds the normal range and the direction remains stable. When the velocity ratio is greater than the preset value and the angle stability index is less than the preset value, a high-confidence output for the fast swipe mode can be given. When identifying the exploration hover mode, by performing linear regression analysis on the velocity sequence in the historical trajectory features to obtain the velocity change slope, and calculating the monotonicity index of the distance sequence from the trajectory point to the trajectory starting point, it can be determined whether the cursor movement speed gradually decreases and the trajectory gradually converges. When the velocity change slope is negative and the distance sequence shows a decreasing trend, a high-confidence output for the exploration hover mode can be given. When identifying precise positioning patterns, the similarity of the trajectory shapes corresponding to historical trajectory sequences with a preset micro-gesture template library is performed. When the similarity exceeds a preset similarity threshold, a high-confidence output for the precise positioning pattern can be given. Finally, by comparing the confidence levels of the three patterns, the pattern with the highest confidence is selected as the classification result. This allows for optimization among the results of multiple discriminators, improving the reliability of the intent classification results.

[0018] Optionally, the step of mapping the two-dimensional coordinates of the cursor to target coordinates in the display coordinate system of the docking station based on the relative position of the cursor in the preset collaborative activation area and the user interface metadata specifically includes: obtaining the two-dimensional coordinates of the cursor in the preset collaborative activation area and calculating the normalized position vector of the two-dimensional coordinates relative to the boundary of the preset collaborative activation area; extracting the target interface element information and neighborhood element topology relationship corresponding to the normalized position vector in the user interface metadata; constructing a local coordinate transformation matrix based on the target interface element information; performing an affine transformation on the normalized position vector through the local coordinate transformation matrix to obtain preliminary mapped coordinates, and optimizing the preliminary mapped coordinates using a bilinear interpolation algorithm to generate an intermediate coordinate sequence; parsing the physical and logical parameters of the docking station display screen, establishing a parameterized model of the docking station coordinate system, and projecting the intermediate coordinate sequence onto the parameterized model to generate the target coordinates in the docking station coordinate system.

[0019] By employing the above technical solution, the two-dimensional coordinates of the cursor within the preset collaborative activation area are obtained, and the normalized position vector of the two-dimensional coordinates relative to the boundary of the preset collaborative activation area is calculated. This allows the cursor position to be mapped to a relative position space, eliminating the influence of the size and shape of the preset collaborative activation area. Extracting the target interface element information and the topological relationship of neighboring elements corresponding to the normalized position vector from the user interface metadata provides the local interface context information corresponding to the current cursor position. A local coordinate transformation matrix is ​​constructed based on the target interface element information. The normalized position vector is mapped to the initial mapped coordinates through affine transformation, and a bilinear interpolation algorithm is used to optimize the initial mapped coordinates, generating an intermediate coordinate sequence. This sequence can locally distort and adjust the cursor position while maintaining the relative positional relationship, making it more closely match the true position of the target interface element. The physical and logical parameters of the docking station display are analyzed, and a parameterized model of the docking station coordinate system is established. The intermediate coordinate sequence is projected onto the parameterized model to generate the target coordinates in the docking station coordinate system. This completes the mapping transformation from the host screen coordinate system to the docking station screen coordinate system, achieving precise synchronization of the cursor position across different devices and improving the coherence and consistency of cross-device interaction.

[0020] Secondly, embodiments of this application provide a multi-mode interactive control system based on a docking station. The multi-mode interactive control system based on a docking station includes: one or more processors and a memory; the memory is coupled to the one or more processors and is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the multi-mode interactive control system based on a docking station to perform the method described in the first aspect and any possible implementation thereof.

[0021] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a multi-mode interactive control system based on a docking station, cause the multi-mode interactive control system based on a docking station to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a multi-mode interactive control system based on a docking station, cause the multi-mode interactive control system based on the docking station to execute the method described in the first aspect and any possible implementation thereof.

[0023] In summary, one or more technical solutions provided in this application have at least the following technical effects or advantages:

[0024] By generating user interface metadata for the docking station display and defining the collaborative activation area on the host side, cursor behavior monitoring and coordinate mapping are achieved. A proxy cursor is then generated on the docking station screen, resolving the operational fragmentation caused by physical isolation in traditional docking station interactions. Users can remotely control the docking station display without leaving the host input device, simplifying the operation process, improving efficiency and smoothness, enhancing interactive intuitiveness, reducing hardware design dependencies, and significantly improving the user experience and cross-device interaction of the docking station. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the multi-mode interactive control method based on a docking station disclosed in an embodiment of this application;

[0026] Figure 2 This is another schematic flowchart of the multi-mode interactive control method based on a docking station disclosed in the embodiments of this application;

[0027] Figure 3 This is a schematic diagram of the structure of a system provided in an embodiment of this application.

[0028] Explanation of reference numerals in the attached drawings: 301, Central Processing Unit; 302, Read-Only Memory; 303, Random Access Memory; 304, Bus; 305, Input / Output Interface; 306, Input Section; 307, Output Section; 308, Storage Section; 309, Communication Section; 310, Driver; 311, Removable Media. Detailed Implementation

[0029] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0030] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0031] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0032] This application provides a multi-mode interactive control method based on a docking station, referring to... Figure 1 , Figure 1 This is a flowchart illustrating a method provided in an embodiment of this application. The method is applied to a system that can execute a multi-mode interactive control program based on a docking station. The method includes steps S101 to S107, as follows:

[0033] Step S101: Obtain the page layout information of the docking station display and generate structured user interface metadata based on the page layout information.

[0034] In step S101, the system first obtains the page layout information of the docking station's display screen through the docking station's display driver interface. The system calls the docking station's API interface to read the rendering tree of the currently displayed content and extracts raw layout data including window position, control size, and element type. Subsequently, the system performs structured processing on this raw data, identifying the boundaries of each functional area through a semantic segmentation algorithm, such as dividing different functional areas like the toolbar, content area, and status bar. The system also analyzes the hierarchical relationship between controls, constructing a tree structure where each node contains metadata such as the control's type attributes, coordinate information, and interactive attributes, ultimately generating complete user interface metadata.

[0035] In one possible implementation, structured user interface metadata is generated based on page layout information, specifically including steps S1011-S1014, as follows:

[0036] Step S1011: Perform semantic segmentation on the page layout information, identify the boundaries of functional areas, control types, and control hierarchy relationships, and generate an initial interface tree structure.

[0037] In step S1011, the system performs semantic segmentation processing on the acquired page layout information. The system first converts the page layout information into a pixel matrix representation and extracts continuous edge contours on the page using an edge detection algorithm (such as the Canny operator). The system analyzes the closure of these contours to identify the boundaries of independent functional areas. For example, the system can identify the menu bar area, toolbar area, main content area, and status bar area on the docking station display. For each identified area, the system determines its control type through feature matching, such as buttons, text boxes, drop-down menus, list items, etc. The system constructs a parent-child hierarchical relationship based on the containment relationship between areas, with the outermost container as the root node and the contained controls as child nodes, forming a tree-like initial interface tree structure. For example, a toolbar node may contain multiple button child nodes, and each button node may contain both icon and text child nodes.

[0038] Step S1012: Based on the initial interface tree structure, calculate the interactive weight value of each node. The interactive weight value is calculated by weighting the node's control type, node depth, number of sibling nodes, and interactive attributes of the parent node.

[0039] In step S1012, the system calculates the interactive weight value for each node in the initial interface tree structure. The system employs a multi-factor weighted algorithm. First, it assigns a base weight based on the control type; for example, buttons are assigned a weight of 0.9, plain text controls are assigned a weight of 0.2, and input box controls are assigned a weight of 0.8. The system also considers the node's depth in the tree; shallower nodes are generally more important. The system uses a depth decay factor (e.g., multiplying the weight by 0.9 for each additional layer of depth) to adjust the weight. The number of sibling nodes also affects the weight calculation. When a parent node has a large number of similar child nodes (e.g., list items), the system appropriately reduces the weight of each child node. Furthermore, if a parent node is marked as non-interactive (e.g., a purely decorative container), the system multiplies the weights of all its child nodes by a reduction factor. Finally, the system calculates the comprehensive interactive weight value for each node using the formula: Interactive Weight Value = Base Weight × Depth Decay Factor × Sibling Node Adjustment Factor × Parent Node Inheritance Factor.

[0040] Step S1013: Mark nodes with interactive weight values ​​lower than the preset weight threshold as non-critical nodes, and merge any two adjacent non-critical nodes into an aggregate node to obtain an optimized interface tree structure.

[0041] In step S1013, the system optimizes the interface tree to reduce redundant nodes. The system traverses all nodes, marking nodes with an interactive weight value lower than a preset weight threshold (e.g., 0.3) as non-critical nodes. These non-critical nodes are typically decorative elements, separators, or static text—interface elements that do not require interaction. The system checks adjacent non-critical nodes; if two non-critical nodes are physically adjacent on the interface and belong to the same parent node, the system merges them into a single aggregate node. For example, multiple consecutive static text labels may be merged into a single text block node. During the merging process, the system retains the boundary information of the original nodes as extended attributes of the aggregate node, ensuring that it can be restored if needed later. Through this optimization, the system generates a more streamlined optimized interface tree structure, reducing the computational overhead of subsequent processing.

[0042] Step S1014: Serialize the optimized interface tree structure, assign a corresponding identifier to each node, establish a mapping index table between nodes, and generate user interface metadata.

[0043] In step S1014, the system converts the optimized interface tree structure into final user interface metadata. The system uses a depth-first traversal algorithm to traverse the optimized interface tree structure, assigning a unique identifier to each node in the format "node type_level_serial number", such as "BTN_2_3" representing the third button node in the second level. The system creates a mapping index table to record the correspondence between each node identifier and its attribute information, including the node's coordinates, size, interactive weight value, control type, parent node identifier, and a list of child node identifiers. The system also establishes a fast lookup index, supporting quick location of the corresponding node by coordinate range and quick filtering of specific node types by control type. Finally, the system serializes this structured data into user interface metadata in JSON or XML format. This metadata not only fully describes the interface structure of the docking station display but also optimizes storage space and query efficiency, providing an efficient data foundation for subsequent collaborative control.

[0044] Step S102: Generate a preset collaborative activation area based on user interface metadata.

[0045] In step S102, the system creates a preset collaborative activation area on the host display screen based on the generated user interface metadata. The system first parses the physical parameters of the docking station display screen, including resolution, size, and aspect ratio, and then generates a virtual area at the edge of the host screen that maintains the same proportions as the docking station screen. For example, if the docking station display screen has a 1920×1080 resolution, the system will create a rectangular activation area of ​​the same proportions on the right edge of the host screen. The size of this area will be dynamically adjusted according to the user's historical operating habits.

[0046] In one possible implementation, a preset collaborative activation area is generated based on user interface metadata, specifically including steps S1021-S1027, as follows:

[0047] Step S1021: Extract the physical resolution and display scaling factor information of the docking station display from the user interface metadata, and calculate a rectangular area with the same aspect ratio as the docking station display based on the physical resolution and display scaling factor, as the reference size of the preset co-activation area.

[0048] In step S1021, the system first parses the display device configuration information from the user interface metadata. The system reads the physical resolution parameters of the docking station's display from the device description segment of the metadata, for example, extracting a resolution of 2560×1440 pixels. Simultaneously, the system obtains the display scaling factor information, which reflects the docking station's system-level display scaling settings, such as 125% scaling in Windows or 2x Retina display in macOS. The system calculates the docking station's effective display resolution using the formula: Effective Resolution = Physical Resolution / Display Scaling Factor. For example, when the physical resolution is 2560×1440 and the display scaling factor is 1.25, the effective resolution is 10248×1152.

[0049] The system then calculates the baseline size of the preset collaborative activation area. First, the system calculates the aspect ratio of the docking station display, such as 2560 ÷ 1440 ≈ 1.78. The system obtains the current resolution and available display area of ​​the host screen, taking into account the space occupied by the taskbar, system menu bar, etc. The system reserves an initial area at the edge of the host screen. The width of this area is set to a specific proportion of the host screen width (e.g., 10%), and the height is dynamically calculated based on the docking station's aspect ratio. For example, if the host screen resolution is 3840×2160, the system creates an area with a width of 384 pixels on the right edge. Based on the aspect ratio of 1.78, the calculated height should be 216 pixels. The system ensures that this rectangular area is completely within the visible range of the host screen. If the calculated height exceeds the screen boundary, the system automatically adjusts the width to fit.

[0050] Step S1022: Obtain historical movement data of the pointer device in the edge area of ​​the host screen, and calculate the historical average movement speed and historical average acceleration of the pointer device in the edge area of ​​the host screen based on the historical movement data.

[0051] In step S1022, the system activates the historical data acquisition module to obtain historical movement data of the pointer device. The system accesses the operating system's event log or its own maintained trajectory database to extract all movement records of the pointer device within a certain time period (e.g., the last 7 days) in the edge area of ​​the host screen. The edge area is defined as the area within 100 pixels of the screen boundary. Each historical record contains information such as timestamp, coordinate position, and direction of movement. The system filters out valid edge movement sequences, i.e., the cursor moving from the center area of ​​the screen to the edge area, or trajectory segments that move continuously within the edge area.

[0052] The system performs statistical analysis on the selected historical movement data. To calculate the historical average movement speed, the system first calculates the instantaneous speed between adjacent sampling points within each trajectory segment. The instantaneous speed is obtained by dividing the Euclidean distance between two adjacent points by the time interval. For example, if the cursor moves from coordinates (3700, 1000) to (3750, 1020) within 10 milliseconds, the system calculates the instantaneous speed to be 5385 pixels per second. The system then arithmetically averages the instantaneous speed values ​​of all trajectory segments to obtain the historical average movement speed, for example, 3000 pixels per second.

[0053] To calculate the historical average acceleration, the system analyzes the rate of change of velocity. The system calculates the acceleration value formed by three consecutive sampling points in each trajectory segment, obtained by dividing the velocity difference by the time interval. The system identifies typical deceleration patterns when the cursor approaches an edge; for example, users typically decelerate when approaching the screen edge to avoid the cursor going out of bounds. The system statistically analyzes the average acceleration value of these deceleration phases, typically a negative value, such as -8000 pixels / second². This historical statistical data provides an important reference benchmark for subsequent judgment of cursor behavior intent.

[0054] The system stores the calculated historical average movement speed and historical average acceleration in a configuration file and sets up a periodic update mechanism. The system recalculates these statistics every 24 hours to ensure it can adapt to changes in user operating habits. Simultaneously, the system maintains independent historical data statistics for different usage scenarios to improve prediction accuracy. This data will be used in subsequent steps to dynamically adjust the sensitivity and triggering conditions of preset collaborative activation areas, enabling more intelligent cross-device interactive control.

[0055] Step S1023: Extract the velocity component corresponding to the historical average moving speed and calculate the velocity standard deviation.

[0056] In step S1023, the system performs an in-depth analysis of the historical average movement speed, extracting speed components and calculating the speed standard deviation. The system first decomposes the historical average movement speed into horizontal and vertical speed components. For example, if the historical average movement speed for a given movement is 3000 pixels / second and the movement angle is 30 degrees, the system calculates using trigonometric functions that the horizontal speed component is 2598 pixels / second and the vertical speed component is 1500 pixels / second. The system then statistically analyzes the speed values ​​in all historical movement data to construct a speed distribution histogram.

[0057] The system uses statistical methods to calculate the speed standard deviation. First, it calculates the mean of all historical speed samples. Then, it calculates the squared difference between each speed sample and the mean. The sum of all squared differences is then divided by the number of samples, and the square root is taken to obtain the speed standard deviation. For example, if the system analyzes 1000 historical movement records with a historical average movement speed of 3000 pixels / second, the calculated speed standard deviation is 800 pixels / second. This standard deviation reflects the dispersion of user operation speed; a larger standard deviation indicates more unstable changes in user operation speed, requiring the system to set a larger safety margin to avoid false triggers.

[0058] Step S1024: Calculate the braking distance of the cursor based on the historical average acceleration. The braking distance is the square of the historical average moving speed divided by twice the historical average acceleration.

[0059] In step S1024, the system calculates the braking distance of the cursor based on a physical motion model. The system analogizes the cursor's movement to uniformly decelerated motion and applies kinematic formulas to calculate the braking distance. The system uses the formula: Braking distance = Historical average movement speed² / (2 × |Historical average acceleration|). Here, the historical average acceleration is usually a negative value, indicating a deceleration process, and the system uses its absolute value for calculation. For example, when the historical average movement speed is 3000 pixels / second and the historical average acceleration is -8000 pixels / second², the system calculates the braking distance as 3000² / (2 × 8000) = 562.5 pixels.

[0060] The braking distance is the physical distance required for the cursor to come to a complete stop from its current speed. The system uses this parameter to assess the buffer space needed when the user approaches the screen edge. If the preset collaborative activation area is too close to the screen edge, the user may not be able to stop the cursor in time and accidentally trigger the collaborative control mode. The system also adjusts the braking distance calculation based on different usage scenarios. For example, in precise operation mode, the system will appropriately increase the estimated braking distance to provide a larger operation buffer.

[0061] Step S1025: Multiply the speed standard deviation by the preset false triggering coefficient to obtain the basic margin value, multiply the braking distance by the preset safety factor to obtain the dynamic margin value, and take the larger value between the basic margin value and the dynamic margin value as the safety margin size.

[0062] In step S1025, the system comprehensively calculates the safety margin size to ensure the reasonable positioning of the preset collaborative activation area. The system first calculates the base margin value by multiplying the speed standard deviation obtained in step S1023 by a preset false trigger coefficient. The preset false trigger coefficient is an empirical value, typically set between 1.5 and 2.5, used to amplify the impact of speed fluctuations on the margin. For example, when the speed standard deviation is 800 pixels / second and the preset false trigger coefficient is 2.0, the base margin value = 800 × 2.0 = 1600 pixels. The base margin value is mainly used to address the uncertainty of user operation speed.

[0063] The system simultaneously calculates the dynamic margin value, multiplying the braking distance from step S1024 by a preset safety factor. The preset safety factor is typically set between 1.2 and 1.8 to provide additional safety margin during braking. For example, when the braking distance is 562.5 pixels and the preset safety factor is 1.5, the dynamic margin value = 562.5 × 1.5 = 843.75 pixels. The dynamic margin value ensures that the user has sufficient space to complete the deceleration maneuver.

[0064] The system compares the base margin value and the dynamic margin value, selecting the larger of the two as the final safe margin size. In the example above, the base margin value of 1600 pixels is greater than the dynamic margin value of 843.75 pixels, so the system selects 1600 pixels as the safe margin size. This safe margin defines the minimum distance that the preset co-activation area should maintain between itself and the actual edge of the screen. The system will adjust the position of the preset co-activation area according to this safe margin, for example, moving the activation area that was originally close to the right edge of the screen inward by 1600 pixels to prevent the user from accidentally triggering the co-control mode when moving the cursor quickly.

[0065] Step S1026: Using the safety margin size as a calibration parameter, calibrate the reference size to obtain the final size and final position coordinates of the preset collaborative activation area.

[0066] In step S1026, the system uses the calculated safety margin size to precisely calibrate the reference size and determine the final parameters of the preset co-activation region. The system first reads the reference size determined in step S1021, such as a rectangular region with a width of 384 pixels and a height of 216 pixels, and the safety margin size calculated in step S1025, such as 1600 pixels.

[0067] The system performs a position calibration process, determining the placement edge of the active area based on the physical position of the docking station's display relative to the host device. For example, when the docking station is located to the right of the host device, the system positions the preset co-activation area at the right edge of the host screen. When calculating the final position coordinates, the system offsets the original edge coordinates inward by a safety margin. Specifically, if the host screen resolution is 3840×2160, the system initially sets the right boundary of the active area at x=3840. After applying a 1600-pixel safety margin, the right boundary of the active area is adjusted to x=2240. The system accordingly adjusts the left boundary coordinate of the active area to x=1856 (2240-384), ensuring that the width of the active area remains unchanged at 384 pixels.

[0068] The system also performs vertical calibration. The system vertically centers the active area, calculated as follows: Top coordinates = (Screen height - Active area height) / 2. In the example above, the top coordinates = (2160 - 216) / 2 = 972 pixels. Therefore, the final coordinates of the preset co-activation area are: top left (1856, 972), bottom right (2240, 1188).

[0069] The system also makes fine-tuning based on user preferences. It analyzes historical movement data to identify hotspots where the cursor crosses the screen edges. If it detects that the user habitually performs cross-device operations from a specific height area of ​​the screen (such as the upper half), the system adjusts the vertical position of the active area accordingly, shifting it upwards to better match user habits. The system creates a configuration file for the active area parameters, recording the final size (384 pixels wide, 216 pixels high) and final position coordinates, and allows users to manually adjust these parameters through the system settings interface.

[0070] Step S1027: When it is detected that the host device is connected to multiple docking stations at the same time, multiple preset collaborative activation areas are generated according to the spatial placement relationship of the multiple docking stations in the physical world. One preset collaborative activation area is generated for each docking station. The spatial layout of each preset collaborative activation area on the host screen is the same as the spatial placement relationship of each docking station in the physical world.

[0071] In step S1027, the system handles multi-dock connection scenarios to achieve a more complex collaborative control layout. When the system detects that the host device is connected to multiple docking stations simultaneously, the system first enumerates all connected docking station devices through the device management interface to obtain the device identifier, display parameters, and connection port information of each docking station.

[0072] The system determines the spatial placement of each docking station in the physical world through multiple methods. First, the system attempts to read data from the docking station's built-in gyroscope or accelerometer to directly obtain the device's physical orientation. If the docking station lacks a sensor, the system prompts the user to manually specify the relative positions of each docking station through a configuration wizard. For example, the user can specify positions such as "Dock A is located to the left of the host," "Dock B is located to the right of the host," and "Dock C is located above the host."

[0073] Based on the identified spatial placement relationships, the system generates multiple preset collaborative activation areas at the corresponding edges of the host screen. For example, when three docking stations are detected to be located on the left, right, and top of the host, the system creates three independent preset collaborative activation areas at the left, right, and top edges of the host screen. The size of each activation area is calculated independently based on the resolution and aspect ratio of the corresponding docking station's display, ensuring that each activation area maintains the same display ratio as its corresponding docking station.

[0074] The system ensures that the preset collaborative activation areas do not overlap. When two docking stations are located adjacently (such as the top left corner and the left side) and this might cause the activation areas to overlap, the system automatically adjusts the size or position of the activation areas. The system prioritizes ensuring the integrity of the activation areas at the corners and adaptively compresses the activation areas in the middle of the edges. For example, the activation area at the top left corner maintains a complete rectangular shape, while the activation area on the left edge avoids the corner area, forming an L-shape or a shortened rectangle.

[0075] The system assigns a unique region identifier to each preset collaborative activation region and establishes a mapping table between activation regions and docking stations. When the cursor enters a specific activation region, the system can quickly identify which docking station's collaborative control mode should be activated. The system also supports setting the priority of activation regions, allowing users to specify which docking station should be activated first in the boundary areas where activation regions overlap.

[0076] The system implements an intelligent spatial awareness mechanism. When the user adjusts the physical location of the expansion dock, the system can automatically detect the change and reconfigure the layout of the preset collaborative activation area. This dynamic adaptation ensures the intuitiveness and consistency of collaborative control in a multi-dock environment, allowing users to naturally perform cross-device operations based on the physical layout of the devices.

[0077] Step S103: Monitor whether the cursor of the pointer device enters the preset collaborative activation area, and determine whether the behavior of the cursor of the pointer device in the preset collaborative activation area meets the collaborative control triggering conditions.

[0078] Step S104: If the cursor of a pointer device enters the preset collaborative activation area and the behavior in the preset collaborative activation area meets the collaborative control triggering condition, then the collaborative control mode is triggered.

[0079] In steps S103 and S104, the system continuously monitors the cursor position of the pointing device. When the system detects that the cursor enters a preset collaborative activation area, the system records the cursor's coordinate sequence at a preset sampling frequency (e.g., 60Hz) and calculates features such as the cursor's movement speed, acceleration, and direction change. The system inputs these features into a preset intent classification model for pattern recognition to determine whether it is a rapid swipe, exploration hover, or precise positioning mode. Only when it is identified as an exploration hover or precise positioning mode, and the cumulative dwell time of the cursor in the area exceeds a preset duration threshold (e.g., 500 milliseconds), will the system trigger the collaborative control mode.

[0080] In one possible implementation, determining whether the behavior of the cursor of the pointing device in the preset cooperative activation area meets the cooperative control triggering condition specifically includes steps S201-S214, as follows:

[0081] Step S201: Establish a cursor trajectory buffer queue, continuously collect the screen coordinates of the cursor at a preset sampling frequency, calculate the cursor motion feature vector between adjacent sampling points, and store the cursor motion feature vector in the cursor trajectory buffer queue.

[0082] In step S201, the cursor trajectory buffer queue refers to a data structure used to temporarily store historical cursor movement data. It operates according to the first-in, first-out principle, continuously recording the latest series of cursor movement information and automatically discarding the oldest information to ensure that the analyzed data is always recent. The preset sampling frequency represents the reciprocal of the time interval between the system's acquisition of the cursor's screen coordinate position. It determines the level of detail in the trajectory data. For example, a sampling frequency of 100 Hz means that the system will record 100 cursor position points per second. The cursor motion feature vector is a set of multiple values ​​that describe the cursor's motion state between two adjacent sampling points. The displacement vector is used to represent the cursor's position change on the two-dimensional plane of the screen, the rate of change of velocity is used to represent the trend of the cursor's movement speed, i.e., acceleration, and the motion direction angle is used to represent the direction of the cursor's movement.

[0083] Specifically, the system continuously runs a monitoring program in the background. This program continuously acquires the (x, y) coordinates of the cursor on the screen at a fixed high frequency, such as 100 times per second. Each time a new coordinate point is acquired, the program compares it with the previous coordinate point, calculates the displacement vector between them (the difference in displacement in the x and y directions), calculates the instantaneous velocity and rate of change of velocity based on the sampling time interval, and calculates the angle between the direction of motion and the horizontal axis. These quantities together form a cursor motion feature vector. The system maintains a queue of fixed length, for example, 200. Whenever a new feature vector is generated, it is pushed to the end of the queue. If the queue is full, the oldest vector at the head of the queue is removed. In this way, the queue always stores the cursor motion trajectory data for the most recent period.

[0084] Step S202: When the cursor is detected to enter the preset collaborative activation area for the first time, extract the historical trajectory features within the preset time window from the cursor trajectory buffer queue.

[0085] In step S202, the preset co-activation area refers to a specific, invisible geometric area predefined on the screen. When the cursor enters this area, the intent recognition process is triggered. For example, it can be a circular area with a radius of 30 pixels around a button. The preset time window represents a fixed time length used to extract a segment of historical data from the cursor trajectory buffer queue for analysis. For example, if it is set to 1 second, the system will extract the motion data of the last second before the cursor enters the area. The historical trajectory features refer to the set of all cursor motion feature vectors extracted from the buffer queue within the preset time window. It completely describes the movement process of the cursor for a period of time before entering the target area.

[0086] Specifically, the system monitors the cursor's current coordinates in real time. Once the system detects that the cursor has moved from outside the preset co-activation area to inside it, it immediately triggers an event. At this time, the system extracts trajectory data from the cursor trajectory buffer queue for a recent period as analysis samples. This time period is preset, for example, extracting all feature vectors generated within the last second. These vector sets constitute the historical trajectory features used for intent analysis.

[0087] Step S203: Normalize the historical trajectory features to obtain a standardized feature vector.

[0088] In step S203, the standardized feature vector refers to a set of values ​​obtained by mathematically transforming the original historical trajectory features. The purpose of this transformation is to eliminate the influence of different physical dimensions and numerical ranges, so that all features are on a uniform scale, which facilitates fair comparison and calculation by the subsequent model.

[0089] Specifically, the system performs numerical standardization on the extracted historical trajectory features. For example, the velocity component in each vector is divided by the maximum screen movement speed, scaling the velocity value to between 0 and 1; the displacement component is divided by the pixel length of the screen diagonal; and the orientation angle from 0 to 360 degrees is also mapped to the 0-1 range by dividing by 360. This process ensures the comparability of features across different dimensions, thereby generating a standardized set of feature vectors.

[0090] Step S204: Input the standardized feature vector into the first discriminator, calculate the ratio of the average value of the velocity component in the historical trajectory features to the preset velocity threshold based on the first discriminator, obtain the velocity ratio, and calculate the ratio of the standard deviation of the direction angle change sequence in the historical trajectory features to the preset angle threshold to obtain the angle stability index.

[0091] In step S204, the first discriminator refers to an algorithm module or set of logical rules specifically designed to identify fast swipe patterns; the preset speed threshold is a critical value used to determine the speed of the cursor, for example, it can be set to 800 pixels per second; the preset angle threshold is a critical value used to determine the stability of the cursor's movement direction, for example, it can be set to 15 degrees, used to measure the standard deviation of the direction change.

[0092] Step S205: When the speed ratio is greater than the preset value and the angle stability index is less than the preset value, output the first confidence level of the fast sweep mode.

[0093] In step S205, the fast swipe mode indicates that the user does not intend to stay in the co-activation area, but intends to quickly move the cursor through this area to reach other positions on the screen; the first confidence level is a numerical value used to indicate the degree of confidence that the current cursor trajectory is judged to be the fast swipe mode.

[0094] In steps S204 and S205, the first discriminator begins operation. It calculates the arithmetic mean of all velocity components in the standardized feature vector set, then divides this mean by a preset velocity threshold to obtain the velocity ratio. Simultaneously, it calculates the standard deviation of the sequence comprising all angular components, then divides this standard deviation by a preset angle threshold to obtain an angle stability index. If the calculated velocity ratio is greater than 1 and the angle stability index is less than 1, this indicates that the cursor trajectory has high speed and stable direction, consistent with the characteristics of a rapid sweep. The first discriminator then outputs a high first confidence value, for example, 0.9.

[0095] Step S206: Input the standardized feature vector into the second discriminator, perform linear regression analysis on the velocity sequence in the historical trajectory features based on the second discriminator, obtain the velocity change slope, and calculate the monotonicity index of the distance sequence from the trajectory point to the trajectory starting point in the historical trajectory features.

[0096] In step S206, the second discriminator refers to an algorithm module or set of logical rules specifically designed to identify exploration hovering patterns; the monotonicity index is a quantitative indicator used to measure whether a numerical sequence is continuously increasing, continuously decreasing, or fluctuating irregularly.

[0097] Step S207: When the slope of the velocity change is negative and the monotonicity index indicates that the distance sequence is decreasing, output the second confidence level of the exploration hovering mode.

[0098] In step S207, the exploration hover mode indicates that the user is interested in objects within the co-activated area and observes and acquires information through actions such as deceleration, small-range movement, or circling; the second confidence level is a numerical value used to indicate the degree of confidence that the current cursor trajectory is judged to be the exploration hover mode.

[0099] In steps S206 and S207, the second discriminator also begins its analysis. It first extracts all velocity components and performs a linear fit on the trends of these velocity values ​​over time, obtaining the slope of a straight line, i.e., the velocity change slope. A negative slope indicates that the cursor is decelerating. Next, it calculates the straight-line distance from each point in the trajectory to the starting point of that trajectory, forming a distance sequence, and analyzes the monotonicity of this sequence. If the distance sequence shows a trend of first increasing and then decreasing, or an overall decreasing trend, it indicates that the cursor has a tendency to zigzag or circle. When the system detects that the velocity change slope is negative and the monotonicity index also shows a decreasing distance trend, this strongly suggests that the user is hesitant or exploring, so the second discriminator outputs a high second confidence level, such as 0.95.

[0100] Step S208: Input the standardized feature vector into the third discriminator, and perform similarity matching between the trajectory shape corresponding to the historical trajectory sequence and the preset micro-gesture template library based on the third discriminator. When the similarity is greater than the preset similarity threshold, output the third confidence of the precise positioning mode.

[0101] In step S208, the third discriminator refers to an algorithm module or set of logical rules specifically designed to identify precise positioning patterns; the micro-gesture template library is a database that pre-stores various tiny trajectory shapes representing precise operational intentions, such as tiny "L"-shaped trajectories or small "hook" shapes, which may represent fine-tuning actions before the user is about to click or drag; the precise positioning pattern indicates a cursor movement pattern in which the user's intention is very clear, preparing to perform precise interaction such as clicking or dragging on a point within the co-activation area; the third confidence level is a numerical value used to indicate the degree of confidence that the current cursor trajectory is judged to be a precise positioning pattern.

[0102] Specifically, the third discriminator also performs its analysis. It compares the shape of the current trajectory with various standard gesture trajectories stored in the internal micro-gesture template library, such as a small right-angle turn or a slight arc. The comparison process uses a dynamic time warping algorithm that can tolerate time scale scaling to calculate the shape similarity between the current trajectory and each template trajectory. If the similarity between the current trajectory and a certain template is found to be very high, exceeding a preset similarity threshold, it indicates that the user may be making final fine-tuning before precise aiming, and the third discriminator outputs a high third confidence score, such as 0.88.

[0103] Step S209: Compare the first confidence level, the second confidence level, and the third confidence level, and select the pattern with the highest confidence level as the classification result.

[0104] In step S209, the system collects the first, second, and third confidence scores output by the three discriminators. For example, the system might obtain a fast swipe confidence score of 0.1, an exploratory hover confidence score of 0.95, and a precise location confidence score of 0.2. The system compares these three values ​​and selects the pattern corresponding to the largest value as the final classification result for this intent recognition. In this example, the result is the exploratory hover pattern.

[0105] Step S210: If the classification result is a quick swipe mode, clear the current monitoring state and continue monitoring the next entry event.

[0106] Step S211: If the classification result is exploration hover mode or precise positioning mode, start the timer to record the cumulative dwell time of the cursor in the preset co-activation area.

[0107] In step S211, the timer is a software timing tool used to measure the passage of time; the cumulative dwell time refers to the total time that the cursor stays continuously or discontinuously within the preset collaborative activation area after being identified as having the intention to stay.

[0108] In steps S210 and S211, the system makes a judgment. If the classification result is a quick swipe pattern, indicating that the user has no intention to linger, the system will clear all intermediate states of this analysis, reset the monitor, and wait for the next cursor entry event. Conversely, if the classification result is an exploration hover pattern or a precise positioning pattern, the system considers the user to have a potential interaction intention and immediately starts a timer to record the time the cursor stays in that area.

[0109] Step S212: Compare the cumulative stay time with the preset time threshold.

[0110] In step S212, the preset duration threshold is a time standard used to determine whether the user's intention to stay is strong enough to trigger collaborative control. For example, it can be set to 1.5 seconds.

[0111] Step S213: If the cumulative dwell time is determined to be greater than or equal to the preset duration threshold, then the behavior of the cursor of the pointer device in the preset collaborative activation area is determined to meet the collaborative control triggering condition.

[0112] In step S213, the collaborative control trigger condition refers to a final judgment result, which indicates that the system has confirmed the user's collaborative intention and is ready to start subsequent collaborative interaction functions, such as displaying an auxiliary operation panel to the user.

[0113] Step S214: If the cumulative dwell time is less than the preset duration threshold, then the behavior of the cursor of the pointer device in the preset collaborative activation area does not meet the collaborative control triggering conditions.

[0114] In steps S212, S213, and S214, as long as the cursor remains within the preset collaborative activation area, the timer continuously accumulates the dwell time. The system continuously compares this accumulated dwell time with a preset duration threshold, such as 1.5 seconds. If at any point the system detects that the accumulated dwell time has reached or exceeded 1.5 seconds, it ultimately determines that the user's behavior meets the triggering conditions for collaborative control and sends a trigger signal to the system. If the cursor leaves the area before the accumulated dwell time reaches the threshold, the timer is reset, the system determines that the behavior does not meet the triggering conditions, the entire process ends, and it waits for the next entry event.

[0115] Step S105: In collaborative control mode, based on the relative position of the cursor in the preset collaborative activation area and the user interface metadata, the two-dimensional coordinates of the cursor are mapped to the target coordinates in the display coordinate system of the docking station.

[0116] In step S105, after the system enters the collaborative control mode, it maps the coordinates of the host cursor to the coordinate system of the docking station display screen in real time. The system first calculates the relative position of the cursor within the preset collaborative activation area to obtain normalized coordinates (e.g., the relative position is 30% of the area width and 50% of the height). Then, based on the actual resolution of the docking station, the system converts the normalized coordinates into absolute pixel coordinates of the docking station screen, for example, mapping (0.3, 0.5) to the (576, 540) pixel position on the docking station.

[0117] In one possible implementation, based on the relative position of the cursor in the preset collaborative activation area and the user interface metadata, the two-dimensional coordinates of the cursor are mapped to the target coordinates in the display coordinate system of the docking station. Specifically, this includes steps S1051-S1055, as follows:

[0118] Step S1051: Obtain the two-dimensional coordinates of the cursor within the preset co-activation region, and calculate the normalized position vector of the two-dimensional coordinates relative to the boundary of the preset co-activation region.

[0119] In step S1051, the normalized position vector refers to the mathematical calculation that transforms the absolute two-dimensional coordinates of the cursor within the preset co-activation region into a relative position representation that is independent of the region size and typically ranges from 0 to 1. For example, the normalized position vector of a cursor located at the exact center of the region is (0.5, 0.5), which allows subsequent calculations to be independent of the specific pixel size of the activation region.

[0120] Specifically, when the cooperative control mode is triggered, the system captures the current two-dimensional coordinates of the pointer device cursor within the preset cooperative activation area in real time, for example, (x, y). Simultaneously, the system obtains the width W and height H of the activation area. Next, the system performs a normalization calculation, using the formulas vector X = x / W and vector Y = y / H to obtain a normalized position vector, for example, (0.75, 0.25). This vector indicates that the cursor is located at three-quarters of the horizontal direction and one-quarter of the vertical direction within the activation area.

[0121] Step S1052: Extract the target interface element information and the topological relationship of the neighboring elements corresponding to the normalized position vector from the user interface metadata.

[0122] In step S1052, the target interface element information refers to the detailed attributes of the interface element that best matches the current cursor position, found in the user interface metadata through the normalized position vector. For example, its type is a button, its text is "Confirm", and its precise boundary coordinates on the dock screen. The neighborhood element topology relationship refers to the position and layout relationship of other elements around the target interface element. For example, to the right of the "Confirm" button is the "Cancel" button, and below it is the "Help" link. This spatial arrangement information is crucial for accurate mapping.

[0123] Specifically, the system uses the normalized location vector obtained in the previous step to search within the previously generated user interface metadata tree. It searches for which interface element's logical bounding box contains this normalized location. Once found, for example, locating a "volume up" button, the system extracts all information about that button as the target interface element information. Simultaneously, the system analyzes elements adjacent to the "volume up" button in the metadata, such as the "volume down" button below it and the "mute" button to its right, thereby obtaining the topological relationships of these neighboring elements.

[0124] Step S1053: Construct a local coordinate transformation matrix based on the target interface element information.

[0125] In step S1053, the local coordinate transformation matrix is ​​a mathematical tool, specifically a matrix, which is used to accurately map the movement of the cursor on the host device within a small range to the area around the corresponding element on the docking station screen, rather than a globally uniform transformation; nonlinear distortion correction is an advanced correction technique used to adjust the local coordinate transformation matrix to compensate for geometric distortions caused by differences in the shape, pixel arrangement, or curvature of the host device and docking station screens, ensuring that straight-line movement remains a perceptible straight line after mapping, without any unnatural bending.

[0126] Specifically, the system enters a crucial calculation phase. It dynamically constructs a unique local coordinate transformation matrix for the currently locked target interface element, the "volume up" button. When constructing this matrix, the system considers multiple factors, such as the ratio of the laptop screen's resolution to the docking station's screen resolution, the difference in pixel density (DPI) between the two, and the display scaling factor of their respective operating systems. Furthermore, the system utilizes the topological relationships of neighboring elements obtained in the previous step to perform non-linear distortion correction on this matrix. The purpose of this step is to address a common problem: if simply scaling linearly, the different screen ratios of the two devices might cause two buttons that appear separate on the main device to have the cursor squeezed into the gap between the two buttons on the docking station screen after mapping. The correction process slightly "pushes" or "pulls" the mapped area to ensure the cursor accurately lands on the target element.

[0127] Step S1054: Perform an affine transformation on the normalized position vector using a local coordinate transformation matrix to obtain preliminary mapped coordinates, and then use a bilinear interpolation algorithm to optimize the preliminary mapped coordinates to generate an intermediate coordinate sequence.

[0128] In step S1054, affine transformation is a geometric transformation that, by applying a local coordinate transformation matrix, can perform linear combination operations such as scaling, rotation, and translation on the normalized position vector to obtain a preliminary mapping result. Bilinear interpolation is a smoothing technique that, when the cursor moves in the boundary area of ​​multiple interface elements, comprehensively considers the coordinate transformation effects of multiple surrounding elements and calculates a smooth transition coordinate by weighted averaging to avoid position jumps when the cursor crosses element boundaries.

[0129] Specifically, the system performs an affine transformation operation on the normalized position vector calculated in the first step and the corrected local coordinate transformation matrix generated in the third step. This operation is essentially a matrix multiplication, and the result is a preliminary mapped coordinate. To make the cursor's trajectory smoother and more natural when moving from one element to another, the system uses a bilinear interpolation algorithm. This algorithm references multiple logical anchor points around the cursor's current position, usually the center points of nearby interface elements, and performs a weighted average of their respective transformation results based on their distance, thereby optimizing the preliminary mapped coordinates and generating a series of continuous and smooth intermediate coordinate sequences.

[0130] Step S1055: Analyze the physical and logical parameters of the docking station display screen, establish a parameterized model of the docking station coordinate system, and project the intermediate coordinate sequence into the parameterized model to generate the target coordinates in the docking station coordinate system.

[0131] In step S1055, the parameterized model of the docking station coordinate system refers to a mathematical description that includes all physical and logical attributes of the docking station display screen. Physical parameters may include the physical resolution of the screen, such as 480x320 pixels, while logical parameters may include the display scaling ratio or screen rotation direction set by the operating system. Projection here is a mathematical operation. It calculates and converts the intermediate coordinate sequence obtained after a series of complex transformations in the abstract computation space into the final integer coordinate values ​​that conform to the physical pixel grid of the docking station screen and can be directly used to draw the proxy cursor.

[0132] Specifically, the system's intermediate coordinate sequence is converted into final pixel coordinates that the docking station can understand. The system calls a pre-established parametric model of the docking station's coordinate system. This model details the actual situation of the docking station's screen; for example, its physical pixel size is 800x600, but the operating system is set to 200% scaling, so the logical pixel size is 400x300. The system performs a projection operation, mapping the intermediate coordinate sequence onto this parametric model, performing final scaling and rounding calculations, and finally generating a precise, integer target coordinate, such as (350, 180). This coordinate is the position where the proxy cursor will be displayed on the docking station's screen.

[0133] Step S106: Send the target coordinates to the expansion dock so that the expansion dock displays a proxy cursor on the target coordinates. The movement trajectory of the proxy cursor is the same as the movement trajectory of the cursor on the pointer device in the preset collaborative activation area.

[0134] In step S106, the target coordinates refer to the precise position point on the docking station screen that the host device should display as a proxy cursor, obtained after coordinate system transformation and mapping calculation based on the position of the pointer device cursor on its own screen. Sending instructions means that the host device packages information containing data such as the target coordinates and transmits it to the docking station via a data connection, such as USB-C or wireless protocol. The docking station is an auxiliary screen device connected to the host device for expanding the display or providing additional interactive functions. The proxy cursor is a cursor image drawn by the docking station according to the host instructions and displayed on its screen. It is not controlled by the pointer device directly connected to the docking station, but serves as a remote representative or "shadow" of the real cursor on the host device. The movement trajectory is used to represent the path formed by the continuous movement of the cursor over a period of time. In this step, it emphasizes the consistency between the path of the proxy cursor and the path of the host cursor in a specific area. The pointer device cursor refers to the cursor displayed on the host screen that the user directly controls on the host device through physical input devices such as a mouse or touchpad. The preset collaborative activation area refers to a special area pre-defined at the edge of the host screen. When the host cursor enters this area, the logic of cross-device collaborative control is triggered.

[0135] Specifically, once the system detects that the cursor on the pointing device has entered the preset collaborative activation area, it initiates a high-frequency coordinate synchronization process. The system continuously acquires the real-time position of the pointer device cursor in the host screen coordinate system at a frequency of tens to hundreds of times per second. For each acquired position, the system executes a coordinate mapping algorithm to transform it from the host screen coordinate system to the target docking station's screen coordinate system, thereby calculating the corresponding target coordinates. This transformation algorithm considers the resolution, scaling ratio, and physical relative position of the two screens to ensure smooth and intuitive cursor movement. After calculating the target coordinates, the system immediately encapsulates this coordinate data into a standardized data packet, forming a command to update the cursor position. This command is sent to the docking station via the communication link between the host and the docking station, such as a DisplayPort auxiliary channel or a dedicated wireless data channel. Upon receiving the command, the docking station's firmware or built-in collaborative software parses the target coordinates and immediately redraws or moves the proxy cursor image at the corresponding position on its screen. Because this "get-compute-send-draw" loop is executed continuously at an extremely high frequency, when the user moves the pointer device on the host, they will see the proxy cursor on the dock reproduce the exact same movement trajectory of the host cursor in the active area with almost no delay.

[0136] Step S107: If it is determined that the cursor of the pointer device has triggered a preset interaction trigger event, then control the proxy cursor to execute the preset interaction trigger event so that the docking station can perform the operation corresponding to the preset interaction trigger event and complete the cross-device interaction.

[0137] In step S107, the cursor on the pointer device refers to the source of the interactive event, i.e., the cursor controlled by the user on the host device; the preset interactive trigger event refers to the active operation initiated by the user through the pointer device, such as mouse click, double click, right click, scroll wheel, etc. These events are predefined by the system as interactive instructions that need to be transmitted to the docking station and executed; controlling the proxy cursor to execute the preset interactive trigger event means that the host device sends an instruction to the docking station, so that it not only moves the proxy cursor, but also simulates an interactive action at the location of the proxy cursor, such as visually presenting a click animation and triggering the actual function; the docking station executing the operation corresponding to the preset interactive trigger event refers to the actual function executed by the docking station at its own operating system or application level after receiving the interactive instruction. For example, if the instruction is a click, the docking station will trigger a real click event on the button under its proxy cursor; cross-device interaction means that the user performs an operation on the host device, but the operation effect is seamlessly reflected on another independent docking station device, realizing collaborative work between devices.

[0138] Specifically, while the proxy cursor is displayed and moves on the docking station, the system on the host device continuously listens for various input events from the pointer device. The system maintains a list of preset interactive trigger events, including specific event types such as "left mouse button down," "left mouse button up," and "scroll wheel up." When the system detects an input event from the pointer device, it first determines whether the event belongs to this preset list. If it does not belong, for example, if it is simply a movement event, it proceeds according to the logic in step S106. If it determines that the event is a preset interactive trigger event, such as the user pressing the left mouse button, the system generates an interactive command. This command explicitly includes the event type, such as "click," and the target coordinates of the current proxy cursor on the docking station. Subsequently, the interactive command is sent to the docking station. Upon receiving the command, the docking station performs two levels of operations: first, it may provide the user with immediate visual feedback on the interface, such as making the proxy cursor image flash rapidly to simulate the visual effect of a click; more importantly, it injects this "click" event and coordinates into its own operating system event queue. The docking station's operating system will handle the injected event like a mouse click on a local connection. If there happens to be an application button under the agent cursor, the button will be successfully clicked, thus executing the corresponding operation and completing a full cross-device interaction.

[0139] The following describes the multi-mode interactive control system based on an expansion dock in the embodiments of this invention from the perspective of hardware processing. Please refer to [link / reference needed]. Figure 3 This is a schematic diagram of the structure of the multi-mode interactive control system based on the expansion dock in this application embodiment.

[0140] It should be noted that, Figure 3 The structure of the multi-mode interactive control system based on the expansion dock shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0141] like Figure 3As shown, the multi-mode interactive control system based on the expansion dock includes a Central Processing Unit (CPU) 301, which can perform various appropriate actions and processes according to programs stored in Read-Only Memory (ROM) 302 or programs loaded from storage section 308 into Random Access Memory (RAM) 303, such as executing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An Input / Output (I / O) interface 305 is also connected to the bus 304.

[0142] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including a hard disk, etc.; and communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0143] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the various functions defined in the present invention.

[0144] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0146] Specifically, the multi-mode interactive control system based on the expansion dock in this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the multi-mode interactive control method based on the expansion dock provided in the above embodiment.

[0147] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the dock-based multi-mode interactive control system described in the above embodiments; or it may exist independently and not assembled into the dock-based multi-mode interactive control system. The storage medium carries one or more computer programs, which, when executed by a processor of the dock-based multi-mode interactive control system, cause the dock-based multi-mode interactive control system to implement the dock-based multi-mode interactive control method based on IoT data encryption transmission provided in the above embodiments.

Claims

1. A multi-mode interactive control method based on a docking station, characterized in that, The method includes: Obtain the page layout information of the docking station display and generate structured user interface metadata based on the page layout information; A preset collaborative activation area is generated based on the user interface metadata; Monitor whether the cursor of the pointer device enters the preset collaborative activation area, and determine whether the behavior of the cursor of the pointer device in the preset collaborative activation area meets the collaborative control triggering conditions; If the cursor of the pointer device enters the preset collaborative activation area, and the behavior in the preset collaborative activation area satisfies the collaborative control triggering condition, then the collaborative control mode is triggered. In the collaborative control mode, based on the relative position of the cursor in the preset collaborative activation area and the user interface metadata, the two-dimensional coordinates of the cursor are mapped to the target coordinates in the display coordinate system of the docking station. The target coordinates are sent to the expansion dock, so that the expansion dock displays a proxy cursor on the target coordinates. The movement trajectory of the proxy cursor is the same as the movement trajectory of the cursor of the pointer device in the preset collaborative activation area. If it is determined that the cursor of the pointer device triggers a preset interaction trigger event, then the proxy cursor is controlled to execute the preset interaction trigger event, so that the docking station performs the operation corresponding to the preset interaction trigger event and completes cross-device interaction; The generation of a preset collaborative activation area based on the user interface metadata specifically includes: Extract the physical resolution and display scaling factor information of the docking station display from the user interface metadata, and calculate a rectangular area with the same aspect ratio as the docking station display based on the physical resolution and the display scaling factor, as the reference size of the preset collaborative activation area; Acquire historical movement data of the pointer device in the edge area of ​​the host screen, and calculate the historical average movement speed and historical average acceleration of the pointer device in the edge area of ​​the host screen based on the historical movement data; Based on the preset safety margin adjustment formula, and combined with the historical average moving speed and historical average acceleration, the safety margin of the preset cooperative activation area is calculated. Using the safety margin size as a calibration parameter, the reference size is calibrated to obtain the final size and final position coordinates of the preset collaborative activation region; When the host device is detected to be connected to multiple docking stations at the same time, multiple preset collaborative activation areas are generated according to the spatial placement relationship of the multiple docking stations in the physical world. One preset collaborative activation area is generated for each docking station. The spatial layout of each preset collaborative activation area on the host screen is the same as the spatial placement relationship of each docking station in the physical world. The step of determining whether the behavior of the cursor of the pointer device in the preset collaborative activation area meets the collaborative control triggering conditions specifically includes: Establish a cursor trajectory buffer queue, continuously collect the screen coordinates of the cursor at a preset sampling frequency, calculate the cursor motion feature vector between adjacent sampling points, and store the cursor motion feature vector in the cursor trajectory buffer queue; When the cursor is detected to enter the preset collaborative activation area for the first time, historical trajectory features within a preset time window are extracted from the cursor trajectory buffer queue. The historical trajectory features are input into a preset intent classification model to obtain a classification result, which is a fast swipe mode, an exploration hover mode, or a precise positioning mode. If the classification result is the fast swipe mode, then clear the current monitoring status and continue monitoring the next entry event; If the classification result is the exploration hover mode or the precise positioning mode, then start the timer to record the cumulative dwell time of the cursor in the preset collaborative activation area; Compare the cumulative dwell time with the preset duration threshold. If it is determined that the cumulative dwell time is greater than or equal to the preset duration threshold, then it is determined that the behavior of the cursor of the pointer device in the preset collaborative activation area satisfies the collaborative control triggering condition. If it is determined that the cumulative dwell time is less than the preset duration threshold, then it is determined that the behavior of the cursor of the pointer device within the preset collaborative activation area does not meet the collaborative control triggering condition.

2. The method according to claim 1, characterized in that, The step of generating structured user interface metadata based on the page layout information specifically includes: The page layout information is semantically segmented to identify functional area boundaries, control types, and control hierarchy relationships, and an initial interface tree structure is generated. Based on the initial interface tree structure, the interactive weight value of each node is calculated. The interactive weight value is calculated by weighting the node according to the control type, node depth, number of sibling nodes and interactive attributes of the parent node. Nodes with interactive weight values ​​lower than a preset weight threshold are marked as non-critical nodes, and any two adjacent non-critical nodes are merged into an aggregate node to obtain an optimized interface tree structure. The optimized interface tree structure is serialized, a corresponding identifier is assigned to each node, and a mapping index table between the nodes is established to generate the user interface metadata.

3. The method according to claim 1, characterized in that, The step of calculating the safety margin of the preset cooperative activation region based on the preset safety margin adjustment formula, combined with the historical average moving speed and historical average acceleration, specifically includes: Extract the velocity component corresponding to the historical average moving speed and calculate the velocity standard deviation; The braking distance of the cursor is calculated based on the historical average acceleration, where the braking distance is the square of the historical average moving speed divided by twice the historical average acceleration. The basic margin value is obtained by multiplying the speed standard deviation by the preset false triggering coefficient, and the dynamic margin value is obtained by multiplying the braking distance by the preset safety factor. The larger value between the basic margin value and the dynamic margin value is taken as the size of the safety margin.

4. The method according to claim 1, characterized in that, The step of inputting the historical trajectory features into a preset intent classification model to obtain the classification result specifically includes: The historical trajectory features are normalized to obtain a standardized feature vector; The standardized feature vector is input into the first discriminator. Based on the first discriminator, the ratio of the average value of the velocity component in the historical trajectory features to the preset velocity threshold is calculated to obtain the velocity ratio. The ratio of the standard deviation of the direction angle change sequence in the historical trajectory features to the preset angle threshold is also calculated to obtain the angle stability index. When the speed ratio is greater than a preset value and the angle stability index is less than the preset value, the first confidence level of the fast sweep mode is output. The standardized feature vector is input into the second discriminator. Based on the second discriminator, a linear regression analysis is performed on the velocity sequence in the historical trajectory features to obtain the velocity change slope, and the monotonicity index of the distance sequence from the trajectory point to the trajectory starting point in the historical trajectory features is calculated. When the slope of the velocity change is negative and the monotonicity index indicates that the distance sequence is decreasing, output the second confidence level of the exploration hovering mode; The standardized feature vector is input into the third discriminator. Based on the third discriminator, the trajectory shape corresponding to the historical trajectory feature is matched with the preset micro-gesture template library for similarity. When the similarity is greater than the preset similarity threshold, the third confidence level of the precise positioning mode is output. The first confidence level, the second confidence level, and the third confidence level are compared, and the pattern with the highest confidence level is selected as the classification result.

5. The method according to claim 1, characterized in that, The step of mapping the two-dimensional coordinates of the cursor to target coordinates in the display coordinate system of the docking station based on the relative position of the cursor in the preset collaborative activation area and the user interface metadata specifically includes: Obtain the two-dimensional coordinates of the cursor within the preset collaborative activation region, and calculate the normalized position vector of the two-dimensional coordinates relative to the boundary of the preset collaborative activation region; Extract the target interface element information and the topological relationship of the neighboring elements from the user interface metadata that correspond to the normalized position vector; Construct a local coordinate transformation matrix based on the target interface element information; The normalized position vector is subjected to an affine transformation through the local coordinate transformation matrix to obtain preliminary mapped coordinates. The preliminary mapped coordinates are then optimized using a bilinear interpolation algorithm to generate an intermediate coordinate sequence. The physical and logical parameters of the docking station display screen are analyzed to establish a parameterized model of the docking station coordinate system. The intermediate coordinate sequence is then projected into the parameterized model to generate the target coordinates in the docking station coordinate system.

6. A multi-mode interactive control system based on a docking station, characterized in that, The docking station-based multi-mode interactive control system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the docking station-based multi-mode interactive control system to perform the method as described in any one of claims 1-5.

7. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on a dock-based multi-mode interactive control system, the dock-based multi-mode interactive control system performs the method as described in any one of claims 1-5.

8. A computer program product, characterized in that, When the computer program product is run on a dock-based multi-mode interactive control system, the dock-based multi-mode interactive control system performs the method as described in any one of claims 1-5.

Images