An intelligent operation control system and method based on human-computer interaction

Abstract

The application provides a kind of intelligent operation control system and method based on human-computer interaction, the load state of motion disorder user in current operation process is determined by the operation behavior data on target interface, adaptive deformation processing is carried out on the control in operable state in target interface by load state, and the distribution proportion of control in target interface is obtained;The function instruction corresponding to the current control is automatically mapped to multiple alternative trigger areas in the high-frequency area of motion disorder user touch track;According to the actual triggered condition of all alternative trigger areas, the effective priority of the function instruction of the control in the target interface between different alternative trigger areas is dynamically updated;The distribution proportion and effective priority are written into the intelligent interaction model, and the predictive adjustment of the control layout of the target interface is carried out by the intelligent interaction model in subsequent interaction process. By using the scheme, the interface control trigger strategy can be dynamically optimized based on the user operation state to realize adaptive intelligent control for complex device interface.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and more specifically, to an intelligent operation control system and method based on human-computer interaction. Background Technology

[0002] Human-computer interaction (HCI) is widely used in mobile terminals, wearable devices, and touch interfaces. Users typically trigger interface controls by clicking, swiping, or long-pressing. Existing HCI interfaces generally adopt a fixed layout design, where the position, size, and trigger area of ​​controls remain unchanged during the interaction.

[0003] The core interactive pain points for people with movement disorders (such as those with Parkinson's disease, cerebral palsy, and limb disabilities) are difficulty in fine motor control and a high risk of operational errors. Furthermore, when users with movement disorders are under stress, fatigued, have decreased finger dexterity, or change their device grip posture, touch offset frequently occurs, leading to continuous accidental touches or failure to successfully trigger target functions. To reduce the accidental touch rate, some technologies attempt to improve usability by increasing the control area, adding error-tolerant zones, or providing vibration feedback. However, these methods are difficult to apply on a large scale in scenarios with limited interface space and cannot dynamically adjust according to the changing states of users with movement disorders at different stages of operation, resulting in controls that still cannot effectively adapt to the constantly changing touch behavior characteristics of users with movement disorders. At the same time, existing technologies lack a dynamic priority mechanism based on user touch behavior, failing to proactively adjust the triggering strategy of interface controls according to changes in user habits, thus reducing operational efficiency. Therefore, how to dynamically optimize the triggering strategy of interface controls based on user operation states to achieve adaptive intelligent control for complex device interfaces, thereby improving the adaptability of operation and the overall interactive experience, has become a challenge facing the industry. Summary of the Invention

[0004] This application provides an intelligent operation control system and method based on human-computer interaction, which can dynamically optimize the triggering strategy of interface controls based on the user's operation status to achieve adaptive intelligent control for complex device interfaces.

[0005] In a first aspect, this application provides an intelligent operation control method based on human-computer interaction, applied to intelligent auxiliary interaction for users with movement disorders. The method includes the following steps:

[0006] Monitor the actions of users with mobility impairments on the target interface. When a user with mobility impairment fails to perform consecutive actions on the current control, obtain the user's action behavior data on the target interface.

[0007] Based on the operational behavior data, the load status of the user with a movement disorder during the current operation is determined, and then adaptive deformation processing is performed on the operable controls in the target interface according to the load status to obtain the distribution ratio of controls in the target interface.

[0008] Automatically map the function command corresponding to the current control to multiple alternative trigger areas in the high-frequency area of ​​the touch trajectory of the user with a mobility impairment;

[0009] Dynamically update the priority of the function instructions of the controls in the target interface among different alternative trigger areas based on the actual triggering status of all alternative trigger areas;

[0010] The distribution ratio and the effective priority are written into the intelligent interaction model, which then predictively adjusts the layout of the target interface controls during subsequent interactions.

[0011] In conjunction with the first aspect, in one possible implementation, determining the load state of a user with a movement disorder during the current operation based on the operational behavior data specifically includes:

[0012] Multiple temporal features for assessing the workload of users with movement disorders are extracted from the operational behavior data;

[0013] The load status of users with movement disorders during the current operation is determined by all temporal characteristics.

[0014] In conjunction with the first aspect, in one possible implementation, adaptive deformation processing is performed on the operable controls in the target interface based on the load state to obtain the distribution ratio of controls in the target interface, specifically including:

[0015] Obtain the corresponding interface deformation parameters based on the load state;

[0016] Based on the interface deformation parameters, perform synchronous visual shape and layout adjustments on all operable controls in the target interface;

[0017] The total area of ​​all operable controls in the target interface after adjustment is used as the distribution ratio of controls in the target interface.

[0018] In conjunction with the first aspect, one possible implementation involves automatically mapping the function command corresponding to the current control to multiple alternative trigger areas in the high-frequency area of ​​the touch trajectory of a user with a mobility impairment. Specifically, this includes:

[0019] Based on the operation behavior data of users with movement disorders on the target interface, identify the spatial distribution hotspots of the touch trajectory of users with movement disorders on the screen, and delineate at least one high-frequency area of ​​touch trajectory.

[0020] Multiple alternative trigger areas with regular geometric shapes are generated within the high-frequency region of the touch trajectory;

[0021] Establish a trigger mapping relationship between the function command corresponding to the current control and all generated alternative trigger areas.

[0022] In conjunction with the first aspect, one possible implementation involves dynamically updating the priority of the function instructions of controls in the target interface across different alternative trigger areas based on the actual triggering status of all alternative trigger areas. This specifically includes:

[0023] Record the historical successful trigger events and related performance data for each alternative trigger zone;

[0024] Based on the historical successful triggering events and related performance data, determine the real-time performance score of each alternative trigger zone;

[0025] The effectiveness priority of functional instructions for controls in the target interface is updated across different alternative trigger zones based on all real-time performance scores.

[0026] In conjunction with the first aspect, in one possible implementation, the distribution ratio and the activation priority are written into the intelligent interaction model, and the intelligent interaction model predictively adjusts the layout of the target interface controls during subsequent interactions, specifically including:

[0027] The distribution ratio and the effective priority are combined with the corresponding interface identifier and the load status of users with movement disorders to form a time-series training sample.

[0028] Based on the time-series training samples, the intelligent interaction model is incrementally trained and its internal parameters are updated to learn the current interaction mode.

[0029] When a user with a mobility impairment navigates to a new interface, the updated intelligent interaction model predicts and outputs predictive layout parameters for that new interface based on the user's recent state sequence.

[0030] Based on the predictive layout parameters, the pre-deformation of controls and the pre-configuration of alternative trigger areas are completed during the initial rendering of the new interface.

[0031] In conjunction with the first aspect, in one possible implementation, the intelligent interaction model refers to a machine learning model that coordinates the operational state of users with movement disorders with the target interface adjustment strategy.

[0032] Secondly, this application provides an intelligent operation control system based on human-computer interaction, comprising:

[0033] The monitoring module is used to monitor the operations of users with mobility impairments on the target interface. When a user with mobility impairment fails to perform consecutive operations on the current control, the module acquires the user's operation behavior data on the target interface.

[0034] The processing module is used to determine the load status of the user with a movement disorder during the current operation based on the operation behavior data, and then perform adaptive deformation processing on the controls in the operable state in the target interface according to the load status to obtain the distribution ratio of controls in the target interface.

[0035] The processing module is also used to automatically map the function command corresponding to the current control to multiple alternative trigger areas in the high-frequency area of ​​the touch trajectory of the user with a mobility impairment;

[0036] The processing module is also used to dynamically update the priority of the function instructions of the controls in the target interface among different alternative trigger areas based on the actual triggering status of all alternative trigger areas.

[0037] The execution module is used to write the distribution ratio and the effective priority into the intelligent interaction model, which then predictively adjusts the layout of the target interface controls during subsequent interactions.

[0038] Thirdly, this application provides a computer device, the computer device including a memory and a processor, the memory for storing a computer program, and the processor for calling and running the computer program from the memory, so that the computer device performs the above-described intelligent operation control method based on human-computer interaction.

[0039] Fourthly, this application provides a computer-readable storage medium storing instructions or code that, when executed on a computer, cause the computer to implement the aforementioned intelligent operation control method based on human-computer interaction.

[0040] The technical solution provided in this application has the following beneficial effects:

[0041] This application's solution, firstly, performs adaptive deformation processing on the operable controls in the target interface based on the load state, obtaining the distribution ratio of controls in the target interface. This allows key controls to obtain a larger reachable area under high load conditions, thereby reducing the difficulty of fine-grained operations. In this way, the interface no longer relies on a fixed layout but dynamically reconstructs itself according to changes in the user's operational ability, improving the interface's adaptability to complex operation scenarios. Secondly, it automatically maps the function instructions corresponding to the current control to multiple alternative trigger areas in the high-frequency area of ​​the touch trajectory of users with mobility impairments. This transforms the function triggering path from interface design-oriented to user behavior-oriented, avoiding the problem of repeatedly trying the original control due to hand tremors or limited range of motion, improving the accessibility of function instructions, and enabling the interface to proactively adapt to the operation trajectory distribution of users with mobility impairments. Then, it dynamically updates the effectiveness priority of the function instructions of the controls in the target interface among different alternative trigger areas based on the actual triggering status of all alternative trigger areas. This gradually strengthens the role of high-success-rate trigger areas while weakening those unsuitable for mobility impairments. The system identifies areas that hinder user operation, thus forming a trigger preference structure that aligns with individual ability characteristics. Dynamically updated activation priorities avoid the incompatibility issues caused by one-time configurations, allowing the trigger strategy to continuously evolve with changes in the user's state, effectively improving the stability and personalization of operation strategies on complex interfaces. Finally, the distribution ratio and activation priority are written into an intelligent interaction model, which predictively adjusts the layout of target interface controls during subsequent interactions. This model proactively optimizes control layout and trigger strategies in subsequent interactions, adjusting control positions and trigger area allocation in advance, ensuring the interface is more accessible before the user operates again. This predictive adjustment mechanism enables complex device interfaces to continuously adapt, reducing the burden of repeated interface adaptation for users with disabilities, and significantly improving overall interaction continuity and user experience. In summary, this solution dynamically optimizes interface control trigger strategies based on user operation states to achieve adaptive intelligent control for complex device interfaces, thereby improving operational adaptability and overall interactive experience. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is an exemplary flowchart of an intelligent operation control method based on human-computer interaction, according to some embodiments of this application;

[0044] Figure 2 This is an exemplary flowchart illustrating the determination of distribution percentages according to some embodiments of this application;

[0045] Figure 3 This is an exemplary flowchart illustrating the update priority according to some embodiments of this application;

[0046] Figure 4 This is a schematic diagram of the structure of an intelligent operation control system based on human-computer interaction, according to some embodiments of this application;

[0047] Figure 5 This is a schematic diagram of the structure of a computer device that implements an intelligent operation control method based on human-computer interaction, according to some embodiments of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] refer to Figure 1 The figure is an exemplary flowchart of an intelligent operation control method based on human-computer interaction according to some embodiments of this application. The intelligent operation control method based on human-computer interaction mainly includes the following steps:

[0050] In step 101, the operation of the user with a motor disability on the target interface is monitored. When the user with a motor disability fails to perform continuous operation on the current control, the operation behavior data of the user with a motor disability on the target interface is obtained.

[0051] In practical implementation, the operation of the user with a disability on the target interface is monitored. When the user with a disability fails to continuously operate the current control, the operation behavior data of the user with a disability on the target interface can be obtained in the following way: Monitor the operation of the user with a disability on the target interface. For example, it can listen to and record all raw touch event streams within the target interface through the underlying interface of the operating system. Each event includes touch point coordinates, timestamp, and pressure value. For the current control that has gained focus, define its visual boundary to expand outward by a certain proportion, such as 10% of the area as the effective trigger zone, and judge each touch action from pressing to lifting: if the trajectory endpoint falls outside the effective trigger zone, it is recorded as a miss failure; if the endpoint is within the zone but the dwell time exceeds the long press threshold, such as 2 seconds, it is recorded as a dwell failure; if the touch point jitter displacement exceeds the preset ratio of the control size, such as 20... A percentage is recorded as a jitter failure, and any condition being met is counted as an operation failure. A failure counter is maintained. When the number of consecutive failures for the same control reaches a preset threshold, such as 3 times, a data acquisition process is triggered. The system will collect all original touch events from the time the control gains focus to the current moment, and extract the operation behavior data of the user with a movement disorder on the target interface. This operation behavior data includes at least the coordinate sequence of the touch trajectory, the movement speed and acceleration sequence, the pressure change sequence, the trajectory jitter amount, the distribution of each failure type, and the total duration of the operation attempt. In a preferred embodiment, the expansion ratio of the effective trigger area can be dynamically adjusted according to the user's historical operation success rate. In other embodiments, eye tracker or electromyography signal sensor data can also be used to assist in determining the operation intention and the cause of failure. This application does not limit this.

[0052] It should be noted that the operational behavior data in this application refers to multi-dimensional temporal information extracted from the original interaction events of users with movement disorders, which is used to quantitatively characterize their operational features and difficulty patterns, and is used to reflect the real-time motor control ability and operational intentions of users with movement disorders.

[0053] In step 102, the load status of the user with a movement disorder during the current operation is determined based on the operation behavior data. Then, adaptive deformation processing is performed on the controls in the operable state in the target interface according to the load status to obtain the distribution ratio of controls in the target interface.

[0054] In some embodiments, determining the load status of a user with a movement disorder during the current operation based on the operational behavior data can be achieved through the following steps:

[0055] Multiple temporal features for assessing the workload of users with movement disorders are extracted from the operational behavior data;

[0056] The load status of users with movement disorders during the current operation is determined by all temporal characteristics.

[0057] In specific implementation, extracting multiple temporal features from the operational behavior data for assessing the workload of users with mobility impairments can be achieved in the following manner: First, perform differential calculation on the touch trajectory coordinate sequence in the operational behavior data to obtain the instantaneous movement speed sequence and acceleration sequence between trajectory points; simultaneously, calculate the standard deviation of the offset of the trajectory point relative to its moving average path as the trajectory jitter amplitude feature; then, count the number of various operation failures recorded in the operational behavior data, including miss failures and dwell failures, and calculate their proportion to the total number of operation attempts as the failure mode distribution feature; finally, calculate all operation attempts, such as the total time elapsed from the first touch press to the last release, as the task duration feature; the instantaneous movement speed sequence, acceleration sequence, trajectory jitter amplitude, failure mode distribution, and task duration obtained above together constitute multiple temporal features for assessing the workload of users with mobility impairments; other methods can also be used in other embodiments, which are not limited here.

[0058] In specific implementation, determining the load status of a user with a disability during the current operation process through all temporal features can be achieved in the following way: All extracted temporal features, including speed, acceleration, jitter amplitude, failure mode distribution, and task duration, are combined into a feature vector, which is then fed into a pre-trained load assessment model. This load assessment model is a support vector machine classifier or multilayer perceptron neural network trained based on historical user interaction data. Its output layer corresponds to multiple preset discrete load levels, such as "low load," "medium load," and "high load." The load assessment model calculates and infers based on the input feature vector, outputting a specific load level label. This label represents the load status, characterizing the cognitive and operational difficulties experienced by the user with a disability during the current operation. In a preferred embodiment, the training data for the load assessment model can be obtained from large-scale interactive experiments with users of different disability types. In other embodiments, a regression model can be used to directly output a continuous load index; this application does not limit this approach.

[0059] It should be noted that the temporal characteristics in this application refer to a set of key indicators that characterize the operation patterns and difficulty levels of users with movement disorders; the load status in this application refers to indicators used to characterize the level of operational difficulty and cognitive stress faced by users with movement disorders in the current interactive task.

[0060] In some embodiments, reference Figure 2As shown, this diagram is an exemplary flowchart for determining the distribution ratio in some embodiments of this application. In this embodiment, adaptive deformation processing is performed on the controls in the operable state of the target interface according to the load state. The distribution ratio of the controls in the target interface can be obtained by the following steps:

[0061] First, in step 1021, the corresponding interface deformation parameters are obtained according to the load state;

[0062] Secondly, in step 1022, the visual shape and layout of all operable controls in the target interface are adjusted synchronously according to the interface deformation parameters.

[0063] Finally, in step 1023, the total area ratio of all operable controls in the target interface after adjustment is taken as the distribution ratio of controls in the target interface.

[0064] In specific implementation, obtaining the corresponding interface deformation parameters based on the load status can be achieved in the following way: a deformation strategy mapping table can be preset, which stores the correspondence between different load status index values ​​and a set of interface deformation parameters; when the load status is obtained, such as the "high load" level or a specific load index of 0.8, the system queries this mapping table, finds the table entry that matches the load status, and reads its corresponding interface deformation parameters; the interface deformation parameters include at least an overall scaling factor for controlling the size of the controls, for example, a scaling factor of 1.5 times for high load and a layout spacing adjustment factor for controlling the spacing between controls; wherein, as a preferred embodiment, the mapping table can be pre-configured through expert experience or user testing; in other embodiments, the parameter values ​​in the table can also be dynamically fine-tuned based on the user's historical interaction effects through a learning model, which is not limited in this application.

[0065] In specific implementation, the synchronous visual shape and layout adjustment of all operable controls in the target interface based on the interface deformation parameters can be achieved in the following way: First, the visual rendering size of each operable (i.e., non-disabled) control in the target interface can be increased simultaneously by the interface rendering engine according to the overall scaling factor in the interface deformation parameters; then, the spacing between all controls can be increased proportionally according to the layout spacing adjustment coefficient in the interface deformation parameters; if the enlargement and spacing adjustment cause the controls to exceed the screen boundary or overlap, the layout reorganization algorithm is automatically triggered, for example, adjusting the controls from a single-row arrangement to a multi-row flow layout, or from a grid layout to a list layout, to ensure that all controls can be displayed in the visible area of ​​the screen without overlap; the above-mentioned size scaling, spacing adjustment and necessary layout reorganization operations are performed synchronously to ensure the instantaneity and consistency of the interface adjustment; as a preferred embodiment, for high-load conditions, the visual appearance of all controls can be further simplified to high-contrast solid color filled shapes to reduce visual complexity; in other embodiments, an elastic grid system can also be used to automatically calculate the optimal position and size of each control, which is not limited in this application.

[0066] In specific implementation, the proportion of the total area of ​​all operable controls in the target interface after adjustment as the distribution proportion of controls in the target interface can be achieved in the following way: After completing the visual shape and layout adjustment of the interface, traverse all controls in the operable state, obtain the current screen display area of ​​each control, which is usually a rectangle or other shaped bounding rectangle; calculate the pixel area of ​​the display area of ​​each control, and sum the pixel areas of all operable controls to obtain the total display area of ​​operable controls on the screen; at the same time, obtain the total pixel area of ​​the target interface, that is, the visible area of ​​the screen currently used for interaction; finally, divide the total display area by the total pixel area, and the quotient is used as the distribution proportion of controls in the target interface; this proportion is a dimensionless value used to quantitatively represent the visual density occupied by operable elements in the current interface; other methods can also be used in other embodiments, which are not limited here.

[0067] It should be noted that the interface deformation parameters in this application refer to a set of control variables used to specifically guide the adjustment of the interface visuals and layout. They are used to transform abstract load status indicators into specific operation instructions that the rendering engine can execute. The operable controls in this application refer to the set of visual interactive elements in the target interface that can currently receive and respond to user interaction instructions. The distribution ratio in this application refers to the proportion of the area occupied by all operable controls in the visible area of ​​the screen. It is used to objectively represent the current space occupancy of interface elements and to provide key structural feedback data for the intelligent interaction model to perform predictive layout adjustment.

[0068] In step 103, the function command corresponding to the current control is automatically mapped to multiple alternative trigger areas in the high-frequency area of ​​the touch trajectory of the user with a mobility impairment.

[0069] In some embodiments, automatically mapping the function command corresponding to the current control to multiple alternative trigger areas in the high-frequency area of ​​the touch trajectory of a user with a mobility impairment can be achieved by the following steps:

[0070] Based on the operation behavior data of users with movement disorders on the target interface, identify the spatial distribution hotspots of the touch trajectory of users with movement disorders on the screen, and delineate at least one high-frequency area of ​​touch trajectory.

[0071] Multiple alternative trigger areas with regular geometric shapes are generated within the high-frequency region of the touch trajectory;

[0072] Establish a trigger mapping relationship between the function command corresponding to the current control and all generated alternative trigger areas.

[0073] In specific implementation, identifying the spatial distribution hotspots of touch trajectories on the screen based on the operational behavior data of users with mobility impairments on the target interface, and delineating at least one high-frequency region of touch trajectory, can be achieved in the following way: The coordinate sequence of touch trajectories recorded in the operational behavior data within a preset historical time period, such as the most recent 30 operations, can be collected and read; a kernel density estimation algorithm or a frequency statistics algorithm based on a fixed-size grid can be used to calculate the spatial distribution density of all touch points on the two-dimensional plane of the screen; continuous areas with density values ​​exceeding a dynamic threshold in the calculated density distribution map are identified as spatial distribution hotspots of touch trajectories of users with mobility impairments on the screen, where the dynamic threshold can be automatically calculated based on the average value and standard deviation of the overall density distribution; finally, the screen coordinate range covered by each identified hotspot is delineated as a high-frequency region of touch trajectory. In a preferred embodiment, the historical time period can be dynamically adjusted according to the frequency of user interaction; in other embodiments, the initial position of the high-frequency region can also be preset based on the physiological characteristics of the user's dominant hand, which is not limited in this application.

[0074] In specific implementation, generating multiple alternative trigger areas with regular geometric shapes within the high-frequency area of ​​the touch trajectory can be achieved in the following way: For each defined high-frequency area of ​​the touch trajectory, within its boundary range, sampling is performed according to a preset equidistant grid rule or based on identified density peak points to determine multiple center point positions; then, with each center point as the core, a region with a regular geometric shape, such as a circle or square, is generated as an alternative trigger area; the size of each alternative trigger area is determined proportionally based on the control display size after adaptive deformation processing in the aforementioned steps, for example, set to 50% to 80% of the area of ​​the deformed control, and ensuring that the edge distance between multiple alternative trigger areas generated in the same high-frequency area is greater than the minimum interval threshold to prevent accidental touch interference; wherein, as a preferred embodiment, the shape and size of the alternative trigger area can take into account the typical touch area of ​​the user's finger; in other embodiments, users can also personalize the generation rules in the settings, which is not limited in this application.

[0075] In practice, establishing the trigger mapping relationship between the function instruction corresponding to the current control and all generated alternative trigger areas can be achieved in the following way: Create a mapping table entry that associates the control that has failed continuous operations (i.e., the unique identifier of the current control and its corresponding system function instruction, such as the click event of the "Submit" button) with the definition information of all generated alternative trigger areas for that control. The definition information includes at least the shape, center coordinates, and boundary range of each alternative trigger area. Subsequently, register this mapping relationship in the system's interactive event handling layer. Specifically, register the screen coordinate range of each alternative trigger area as a system-level... The system defines a hotspot and rewrites its event handling logic so that when the system detects that a user's touch point falls into any registered alternative trigger area, it immediately intercepts the original touch event and simulates and generates and dispatches a standard function instruction event corresponding to the current control. Simultaneously, at the user interface layer, the current control can be briefly highlighted or haptic feedback can be provided to indicate to users with mobility impairments that the mapping trigger has taken effect. In a preferred embodiment, the system can maintain a global mapping table to manage the correspondence between multiple controls and multiple sets of alternative trigger areas. In other embodiments, an expiration period can be set for the mapping relationship or it can be dynamically maintained based on usage frequency; this application does not limit this.

[0076] It should be noted that, in this application, the spatial distribution hotspots refer to revealing the user's natural comfort zone and habitual operation points when interacting with the screen; the high-frequency touch trajectory area in this application refers to the specific screen coordinate range defined based on the spatial distribution hotspots to carry interactive functions, which ensures that newly generated interactive elements are within the efficient range of the operating habits of users with mobility impairments; the functional instructions corresponding to the current control in this application refer to the system-level operation commands pre-bound to controls that have failed consecutive operations, such as clicks and long presses, which are used to change the interactive triggering method while maintaining the integrity of the application functions; the trigger mapping relationship in this application refers to the one-to-many correspondence rules established between the functional instructions of the current control and all the alternative trigger areas generated for it, which are used to build an event redirection and instruction forwarding mechanism. It ensures that valid touch events occurring in any alternative trigger area can be recognized by the system and converted into precise calls to the original control's functional instructions, thereby achieving flexible expansion and personalized adaptation of the interaction path without modifying the underlying application logic; the alternative trigger area in this application refers to a screen interaction area with a regular geometric shape.

[0077] In step 104, the priority of the function instructions of the controls in the target interface among different alternative trigger areas is dynamically updated based on the actual triggering status of all alternative trigger areas.

[0078] In some embodiments, reference Figure 3 As shown, this diagram is an exemplary flowchart of updating the effectiveness priority in some embodiments of this application. In this embodiment, dynamically updating the effectiveness priority of the function instructions of the controls in the target interface among different alternative trigger areas based on the actual triggering status of all alternative trigger areas can be achieved by the following steps:

[0079] First, in step 1041, the historical successful triggering events and related performance data of each alternative trigger zone are recorded;

[0080] Secondly, in step 1042, a real-time performance score for each alternative trigger zone is determined based on the historical successful trigger events and related performance data;

[0081] Finally, in step 1043, the priority of the function instructions of the controls in the target interface is updated among different alternative trigger areas based on all real-time performance scores.

[0082] In specific implementation, recording the historical successful trigger events and related performance data of each alternative trigger zone can be achieved in the following way: when a user with a movement disorder is detected to have successfully triggered an alternative trigger zone through the aforementioned trigger mapping relationship, causing the original control function instruction mapped to it to be executed, a successful trigger event for that alternative trigger zone is recorded; each record includes at least an event timestamp, the identifier of the alternative trigger zone that triggered the event, the trigger time from the touch point entering the alternative trigger zone to the successful execution of the function instruction, and the trajectory stability index of the touch point in the alternative trigger zone during the triggering process, such as the trajectory jitter amplitude; these records are stored in chronological order in a historical database dedicated to performance analysis and are associated with the specific alternative trigger zone identifier; wherein, as a preferred embodiment, failed triggers, such as entering and quickly leaving, can also be recorded simultaneously to provide more comprehensive analysis data; in other embodiments, the recorded data can also be managed using a sliding window to retain only the data within the most recent period to control the storage scale, which is not limited in this application.

[0083] In specific implementation, determining the real-time performance score of each alternative trigger zone based on the historical successful trigger events and related performance data can be achieved in the following way: The score calculation for each alternative trigger zone can be performed periodically or whenever a new successful trigger event is recorded. The calculation process is as follows: First, count the total number of successful triggers for the alternative trigger zone in the most recent statistical period, such as the last 50 triggers or the last 10 minutes; the more triggers, the higher the base score. Then, calculate the average trigger time for all successful triggers within that period; the shorter the trigger time, the greater the positive contribution to the score. Simultaneously, calculate the average trajectory stability of all successful triggers within that period. The higher the stability of the performance index, i.e., the smaller the jitter, the greater its positive contribution to the score. Finally, a time decay factor is introduced, so that recently occurring successful triggering events have a higher weight in the calculation than earlier events. Combining the above factors, a real-time performance score reflecting the current comprehensive effectiveness of the alternative triggering area is calculated through a weighted summation or rule-based scoring function. In a preferred embodiment, the scoring function can be optimized by machine learning methods based on the user's long-term adaptation effect. In other embodiments, the number of times the user with a movement disorder actively cancels the operation can also be included as a negative factor in the scoring calculation, which is not limited in this application.

[0084] In practice, updating the priority of functional instructions of controls in the target interface across different alternative trigger areas based on all real-time performance scores can be achieved as follows: For all alternative trigger areas mapped to the same original control functional instruction, obtain their currently calculated real-time performance scores; then, sort these alternative trigger areas according to their scores from highest to lowest, generating a new ordered list, which is the updated priority list; the first element in the list represents the highest priority; this new list is then applied. For example, in interaction logic, when multiple alternative trigger areas are triggered almost simultaneously, events from higher-priority alternative trigger areas are responded to first. In the user interface, visual encoding, such as displaying the highest-priority alternative trigger area in the most prominent color or with a special identifier, can imply the effectiveness level to users with mobility impairments, guiding them to prioritize the use of high-efficiency areas. Simultaneously, alternative trigger areas with consistently low performance scores below the failure threshold can be marked or automatically disabled based on this list, achieving dynamic optimization. In a preferred embodiment, priority updates can be smooth and gradual to avoid frequent changes in interface prompts that could interfere with users with mobility impairments. In other embodiments, users with mobility impairments may also be allowed to manually adjust or fix the priority of a particular alternative trigger area; this application does not limit this.

[0085] It should be noted that the historical successful triggering events and related performance data in this application refer to a set of spatiotemporal information recorded for each alternative triggering area, quantifying its interaction efficiency and user habits. This information serves as the original factual basis for evaluating the practicality and reliability of the alternative triggering areas. The real-time efficiency score in this application refers to a numerical indicator characterizing the comprehensive interaction efficiency of each alternative triggering area at the current moment. This score serves as the core quantitative basis for dynamically comparing and ranking the merits of different alternative triggering areas. The effective priority in this application refers to the sorting rules used to define the response order and tendency of the same functional instruction among its multiple associated alternative triggering areas. This priority is used to construct a dynamic, performance-based interaction routing strategy that can proactively guide users with mobility impairments to use more efficient and reliable interaction areas and serve as a key optimization signal fed back to the intelligent interaction model to achieve continuous self-optimization of the interaction system.

[0086] In step 105, the distribution ratio and the effective priority are written into the intelligent interaction model, which then predictively adjusts the layout of the target interface controls during subsequent interactions.

[0087] In some embodiments, the distribution ratio and the effective priority are written into the intelligent interaction model, and the intelligent interaction model predictively adjusts the layout of the target interface controls during subsequent interactions. This can be achieved through the following steps:

[0088] The distribution ratio and the effective priority are combined with the corresponding interface identifier and the load status of users with movement disorders to form a time-series training sample.

[0089] Based on the time-series training samples, the intelligent interaction model is incrementally trained and its internal parameters are updated to learn the current interaction mode.

[0090] When a user with a mobility impairment navigates to a new interface, the updated intelligent interaction model predicts and outputs predictive layout parameters for that new interface based on the user's recent state sequence.

[0091] Based on the predictive layout parameters, the pre-deformation of controls and the pre-configuration of alternative trigger areas are completed during the initial rendering of the new interface.

[0092] It should be noted that the intelligent interaction model in this application refers to a machine learning model that coordinates the operational state of users with disabilities with the adjustment strategy of the target interface. It takes the sequence of user interaction states as input and the predictive layout parameters of the target interface as output. It serves as the core of prediction and decision-making for adaptive interaction. By continuously learning the personalized patterns contained in the time-series training samples, it achieves the leap from passively responding to user operations to actively predicting and pre-configuring the optimal interface layout, enabling the target interface to perform adaptive interactive control according to the current actual operational state of the user with disabilities.

[0093] In specific implementation, the distribution ratio and the effective priority, combined with the corresponding interface identifier and the user's workload status, can be integrated into a time-series training sample in the following way: At the end of the current interaction task, such as when a screen operation is completed, the control distribution ratio finally determined in this task, the effective priority list generated for each control of the current screen, are encoded into a fixed-dimensional vector, such as the priority order index of each alternative trigger area, the unique identifier of the current screen, such as the screen ID or layout template name, and the user's workload status at the end of the task, such as the "high workload" label or workload index, and packaged together; at the same time, the current timestamp is added to this data package, and it is added to the personalized interaction time-series database of the user with a disability as a new time-series training sample; wherein, as a preferred embodiment, the list corresponding to the effective priority can be vectorized by one-hot encoding or sorting numbering; in other embodiments, the set of alternative trigger area identifiers used in this interaction can also be included in the sample, which is not limited in this application.

[0094] In specific implementation, incremental training of the intelligent interaction model based on the time-series training samples to update its internal parameters to learn the current interaction mode can be achieved in the following way: the intelligent interaction model is a machine learning model suitable for sequence prediction, such as a long short-term memory network or a temporal convolutional network; during training, the most recent 100 training sample sequences of the user with the movement disorder are extracted from the time-series database as input features. This sequence includes interface identifiers, distribution proportions, priority vectors, and load states arranged by time; the training objective of the model is to learn the layout parameters applicable to predicting the next interaction interface; the training process can adopt online learning or mini-batch gradient descent, using the newly added samples to calculate the loss function, such as the gradient of the mean square error between the predicted parameters and the subsequent actual optimal parameters, and using this gradient to fine-tune and update the model's weight parameters, so that the intelligent interaction model can capture and adapt to the latest interaction preferences and ability change trends of the user with the movement disorder; wherein, as a preferred embodiment, an elastic weight consolidation algorithm can be used to prevent catastrophic forgetting in incremental learning; in other embodiments, retraining can also be performed periodically using historical full data to maintain model stability, which is not limited in this application.

[0095] In specific implementation, when a user with a mobility impairment navigates to a new interface, the updated intelligent interaction model predicts and outputs predictive layout parameters for the new interface based on the user's recent state sequence. This can be achieved in the following way: when a user with a mobility impairment triggers an interface jump, such as clicking a link or button, the updated intelligent interaction model is invoked before loading the new interface content. The model's input is the current user's recent state sequence, which consists of sample features from several recent interactions (including interface identifiers, distribution percentages, priority vectors, and load status), and the identifier of the new interface to be entered. The model infers based on the patterns it has learned and outputs a set of predictive layout parameters for the new interface. These parameters include at least the suggested basic scaling level of controls, the suggested number of initial alternative trigger zones, and the suggested trigger zone generation area offset. In a preferred embodiment, the length of the recent state sequence can be dynamically adjusted according to the prediction accuracy. In other embodiments, the model's prediction can also consider different time periods of the day or the user's current active applications, which is not limited in this application.

[0096] In specific implementation, the pre-deformation of controls and the pre-configuration of alternative trigger areas during the initial rendering of the new interface, based on the predictive layout parameters, can be achieved in the following way: After receiving the predictive layout parameters and the original layout description of the new interface, the interface rendering engine immediately applies the predictive layout parameters in the interface rendering pipeline. For example: First, based on the predicted basic scaling level of the controls, all operable controls are enlarged and rendered at once; second, based on the predicted number of initial alternative trigger areas and regional bias, a corresponding number of alternative trigger areas are pre-generated in the specified high-frequency areas, and their trigger mapping relationship with the corresponding controls is immediately established; all the above operations are completed before the interface is presented to the user with a disability, so that the layout and interaction hotspots of the user with a disability are already in a state of prediction and optimization based on their recent habits the moment they see the new interface, achieving seamless predictive adjustment; in a preferred embodiment, the system can monitor the prediction effect. If the user with a disability still frequently fails in the initial operation of the new interface, it can quickly revert to the default layout and trigger the model to relearn; in other embodiments, the pre-configuration process can use a gradient animation for a smooth transition, which is not limited in this application.

[0097] It should be noted that the temporal training samples in this application refer to structured data units encapsulated in chronological order of key indicators and corresponding interface context generated during a complete interaction. They serve as the core data carrier for recording the personalized interaction patterns and interface adaptation results of users with disabilities, and are the basic training data driving the intelligent interaction model to perform incremental learning in order to capture changes in the abilities and preferences of users with disabilities. The recent state sequence of users with disabilities in this application refers to a sequence composed of temporal training samples corresponding to the most recent interactions of users with disabilities, arranged in chronological order. It is used to reflect the latest interaction abilities, habits, and contextual environment of users with disabilities. The predictive layout parameters in this application refer to a set of pre-configured instructions used to guide the initial rendering of a new interface. They serve as an execution blueprint for transforming the model's predictive capabilities into specific interface rendering actions, enabling the visual form of the controls and the distribution of interactive hotspots to be personalized based on the user's recent patterns before the interface is presented to users with disabilities, thereby achieving predictive adjustment of the experience.

[0098] Furthermore, in another aspect of this application, in some embodiments, this application provides an intelligent operation control system based on human-computer interaction, referencing... Figure 4 The figure is a schematic diagram of the structure of an intelligent operation control system based on human-computer interaction according to some embodiments of this application. The intelligent operation control system 400 based on human-computer interaction includes: a monitoring module 401, a processing module 402, and an execution module 403, which are described below:

[0099] The monitoring module 401 in this application is mainly used to monitor the operation of the user with a motor disability on the target interface. When the user with a motor disability fails to perform continuous operation on the current control, the monitoring module 401 obtains the operation behavior data of the user with a motor disability on the target interface.

[0100] Processing module 402 in this application is mainly used to determine the load state of the user with a movement disorder in the current operation process based on the operation behavior data, and then perform adaptive deformation processing on the controls in the operable state in the target interface according to the load state to obtain the distribution ratio of controls in the target interface.

[0101] The processing module 402 described in this application is also used to automatically map the function instruction corresponding to the current control to multiple alternative trigger areas in the high-frequency area of ​​the touch trajectory of the user with a mobility impairment;

[0102] The processing module 402 described in this application is also used to dynamically update the effectiveness priority of the function instructions of the controls in the target interface among different alternative trigger areas based on the actual triggering status of all alternative trigger areas.

[0103] The execution module 403 in this application is mainly used to write the distribution ratio and the effective priority into the intelligent interaction model, so that the intelligent interaction model can predictively adjust the layout of the target interface controls in the subsequent interaction process.

[0104] The foregoing has detailed examples of intelligent operation control systems and methods based on human-computer interaction provided in the embodiments of this application. It is understood that the corresponding devices, in order to achieve the above functions, include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0105] In some embodiments, this application also provides a computer device, the computer device including a memory and a processor, the memory for storing a computer program, and the processor for calling and running the computer program from the memory, so that the computer device performs the above-described intelligent operation control method based on human-computer interaction.

[0106] In some embodiments, reference Figure 5The dashed lines in the figure indicate that the unit or module is optional. This figure is a structural schematic diagram of a computer device implementing the human-computer interaction-based intelligent operation control method of this application. The human-computer interaction-based intelligent operation control method in the above embodiments can be... Figure 5 The computer device 500 shown is used to implement this, and the computer device 500 includes at least one processor 501, a memory 502 and at least one communication unit 505. The computer device 500 may be a terminal device, a server or a chip.

[0107] The processor 501 can be a general-purpose processor or a special-purpose processor. For example, the processor 501 can be a central processing unit (CPU). The CPU can be used to control the computer device 500, execute software programs, and process data from the software programs. The computer device 500 may also include a communication unit 505 for inputting (receiving) and outputting (transmitting) signals.

[0108] For example, computer device 500 may be a chip, communication unit 505 may be the input and / or output circuit of the chip, or communication unit 505 may be the communication interface of the chip, and the chip may be a component of terminal device, network device or other device.

[0109] For example, computer device 500 may be a terminal device or a server, and communication unit 505 may be a transceiver of the terminal device or the server, or communication unit 505 may be a transceiver circuit of the terminal device or the server.

[0110] The computer device 500 may include one or more memories 502 storing a program 504. The program 504 can be executed by a processor 501 to generate instructions 503, causing the processor 501 to perform the methods described in the above method embodiments according to the instructions 503. Optionally, the memory 502 may also store data (such as a target audit model). Optionally, the processor 501 may also read data stored in the memory 502, which may be stored at the same storage address as the program 504, or the data may be stored at a different storage address than the program 504.

[0111] The processor 501 and memory 502 can be configured separately or integrated together, for example, integrated on the system-on-chip (SOC) of the terminal device.

[0112] It should be understood that each step of the above method embodiment can be completed by hardware logic circuits or software instructions in the processor 501. The processor 501 can be a CPU, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, such as discrete gate, transistor logic devices, or discrete hardware components.

[0113] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] For example, in some embodiments, this application also provides a computer-readable storage medium storing instructions or code that, when executed on a computer, cause the computer to implement the above-described intelligent operation control method based on human-computer interaction.

[0115] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0116] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A human-computer interaction-based intelligent operation control method, applied to intelligent auxiliary interaction for users with movement disorders, characterized in that, The method includes the following steps: Monitor the actions of users with mobility impairments on the target interface. When a user with mobility impairment fails to perform consecutive actions on the current control, obtain the user's action behavior data on the target interface. Based on the operational behavior data, the load status of the user with a motor disability during the current operation is determined. The load status refers to an indicator used to characterize the level of operational difficulty and cognitive stress faced by the user with a motor disability under the current interactive task. Then, based on the load status, adaptive deformation processing is performed on the controls in the operable state in the target interface to obtain the distribution ratio of controls in the target interface. The function command corresponding to the current control is automatically mapped to multiple alternative trigger areas in the high-frequency area of ​​the touch trajectory of the user with a disability. The high-frequency area of ​​the touch trajectory refers to the specific screen coordinate range used to carry interactive functions, which is defined based on the spatial distribution hotspots. Dynamically update the priority of the function instructions of the controls in the target interface among different alternative trigger areas based on the actual triggering status of all alternative trigger areas; The distribution ratio and the effective priority are written into the intelligent interaction model, which then predictively adjusts the layout of the target interface controls during subsequent interactions. Specifically, determining the load status of a user with a movement disorder during the current operation based on the operational behavior data includes: Multiple temporal features for assessing the workload of users with movement disorders are extracted from the operational behavior data; The load status of users with movement disorders during the current operation is determined by all temporal characteristics; Specifically, automatically mapping the function command corresponding to the current control to multiple alternative trigger areas in the high-frequency area of ​​the touch trajectory of users with mobility impairments includes: Based on the operation behavior data of users with movement disorders on the target interface, identify the spatial distribution hotspots of the touch trajectory of users with movement disorders on the screen, and delineate at least one high-frequency area of ​​touch trajectory. Multiple alternative trigger areas with regular geometric shapes are generated within the high-frequency region of the touch trajectory; Establish a trigger mapping relationship between the function command corresponding to the current control and all generated alternative trigger areas; Specifically, dynamically updating the priority of function commands of controls in the target interface across different alternative trigger zones based on the actual triggering status of all alternative trigger zones includes: Record the historical successful trigger events and related performance data for each alternative trigger zone; Based on the historical successful triggering events and related performance data, determine the real-time performance score of each alternative trigger zone; The effectiveness priority of functional instructions for controls in the target interface is updated across different alternative trigger zones based on all real-time performance scores.

2. The method as described in claim 1, characterized in that, Based on the load status, adaptive deformation processing is performed on the operable controls in the target interface to obtain the distribution ratio of controls in the target interface, specifically including: Obtain the corresponding interface deformation parameters based on the load state; Based on the interface deformation parameters, perform synchronous visual shape and layout adjustments on all operable controls in the target interface; The total area of ​​all operable controls in the target interface after adjustment is used as the distribution ratio of controls in the target interface.

3. The method as described in claim 1, characterized in that, The distribution ratio and the effective priority are written into the intelligent interaction model, and the intelligent interaction model predictively adjusts the layout of the target interface controls during subsequent interactions. Specifically, this includes: The distribution ratio and the effective priority are combined with the corresponding interface identifier and the load status of users with movement disorders to form a time-series training sample. Based on the time-series training samples, the intelligent interaction model is incrementally trained and its internal parameters are updated to learn the current interaction mode. When a user with a mobility impairment navigates to a new interface, the updated intelligent interaction model predicts and outputs predictive layout parameters for that new interface based on the user's recent state sequence. Based on the predictive layout parameters, the pre-deformation of controls and the pre-configuration of alternative trigger areas are completed during the initial rendering of the new interface.

4. The method as described in claim 1, characterized in that, The intelligent interaction model refers to a machine learning model that coordinates the operational status of users with movement disorders with the adjustment strategy of the target interface.

5. A human-computer interaction-based intelligent operation control system, which employs the method described in any one of claims 1 to 4 for human-computer interaction-based intelligent operation control, characterized in that, This intelligent operation control system based on human-computer interaction includes: The monitoring module is used to monitor the operations of users with mobility impairments on the target interface. When a user with mobility impairment fails to perform consecutive operations on the current control, the module acquires the user's operation behavior data on the target interface. The processing module is used to determine the load status of the user with a movement disorder during the current operation based on the operation behavior data, and then perform adaptive deformation processing on the controls in the operable state in the target interface according to the load status to obtain the distribution ratio of controls in the target interface. The processing module is also used to automatically map the function command corresponding to the current control to multiple alternative trigger areas in the high-frequency area of ​​the touch trajectory of the user with a mobility impairment; The processing module is also used to dynamically update the effective priority of the function instructions of the controls in the target interface among different alternative trigger areas based on the actual triggering status of all alternative trigger areas. The execution module is used to write the distribution ratio and the effective priority into the intelligent interaction model, which then predictively adjusts the layout of the target interface controls during subsequent interactions.

6. A computer device, characterized in that, The computer device includes a memory and a processor. The memory is used to store computer programs, and the processor is used to call and run the computer programs from the memory, so that the computer device performs the intelligent operation control method based on human-computer interaction as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions or code that, when executed on a computer, cause the computer to implement the intelligent operation control method based on human-computer interaction as described in any one of claims 1 to 4.

Images