Human-computer interaction method and device, electronic equipment and computer storage medium

Abstract

The application provides a human-computer interaction method and device, electronic equipment and computer storage medium, which determines whether a user is in an interaction confusion state through the position of a touch point and a sampling time; if the user is in the interaction confusion state, joint reasoning is performed based on an image of a current interface and user operation context information to determine a target function control most relevant to a current user operation and a semantic label of the target control, without pre-configuring a semantic label for the control by modifying a source code of a third-party native application as in the prior art, thereby fundamentally solving the problem of incomplete adaptation of mobile terminal native applications in the prior art; then, interaction prompt information is generated based on the position of the current touch point, the semantic label of the target function control and coordinate information of the target function control, the target function control is found by the user through the interaction prompt information, and the visually impaired user can obtain consistent interaction experience across applications regardless of the developer adaptation level.

Description

Technical Field

[0001] This invention relates to the field of electronic information technology, and in particular to a human-computer interaction method, device, electronic device, and computer storage medium. Background Technology

[0002] With the widespread use of smartphones in daily life, mobile applications (Apps) have become the core terminals for obtaining information and services. However, for visually impaired users, traditional graphical user interface (GUI) interactions still present significant technical barriers.

[0003] Existing accessibility technologies rely heavily on label attributes (such as contentDescription) manually configured by developers in the source code. This results in the system being unable to provide effective information to visually impaired users when faced with unlabeled or improperly labeled controls, causing a perceptual disconnect.

[0004] Current technologies primarily rely on screen readers like TalkBack to assist visually impaired users, heavily depending on developers manually configuring the `contentDescription` or `AccessibilityNodeInfo` attributes in the source code of native applications. However, in practice, many native apps have missing or inaccurate control labels, causing TalkBack to only announce "unlabeled buttons," failing to provide effective information to visually impaired users and leaving them in a "perceptual disconnect" state when faced with complex native layouts. When users experience anxiety and perform ineffective swipes or actions due to not being able to find their target in complex interfaces, existing systems cannot detect the user's "lost" state in real time and proactively intervene in navigation, resulting in extremely low interaction efficiency. Summary of the Invention

[0005] In view of this, the present invention provides a human-computer interaction method, device, electronic device, and computer storage medium that does not require modification of the source code of any third-party native application, fundamentally solving the problem of incomplete adaptation that is common in existing mobile native applications, enabling visually impaired users to obtain a consistent interactive experience across applications regardless of the developer's adaptation level.

[0006] The first aspect of this invention provides a human-computer interaction method, comprising:

[0007] Real-time acquisition of user touch point information; wherein, the touch point information includes the position of the touch point and the sampling time;

[0008] Based on the position of the touch point and the sampling time, it is determined whether the user is in an interactive disorientation state; wherein, the interactive disorientation state refers to the state when the user experiences anxiety and performs invalid operations due to not being able to find the target in a complex interface;

[0009] If the user is in an interactive maze state, the target functional control and its semantic label are determined based on the current interface image and user operation context information.

[0010] Based on the current touch point position, the semantic label of the target function control, and the coordinate information of the target function control, determine the interactive prompt information.

[0011] Optionally, determining whether the user is in an interactive maze state based on the position of the touch point and the sampling time includes:

[0012] Based on the location of the touch point and the sampling time, a trajectory indicator and an invalid dwell indicator are determined; wherein, the invalid dwell indicator refers to a user behavior metric that measures the cumulative duration of time a user spends in a non-interactive area and / or without triggering a valid control response.

[0013] Based on the trajectory index and the invalid stay index, the interactive disorientation coefficient is determined.

[0014] Optionally, determining the target functional control and its semantic label based on the image and user operation context information of the current interface includes:

[0015] The image of the current interface and the user operation context information are input into a preset model, and the target functional control and the first semantic label of the target functional control are output. The preset model identifies the visual elements in the image of the current interface, obtains the feature information of the visual elements, and determines the first semantic label and first confidence level of the candidate controls in the current interface based on the feature information of the visual elements and the user operation context information. The candidate control with the highest first confidence level is selected as the target functional control.

[0016] Optionally, before determining the target functional control based on the image of the current interface and the user operation context information, the method further includes:

[0017] Identify visual elements in the image of the current interface, and determine spatial layout information based on the position information, size information, regional affiliation relationship and adjacency relationship of all the visual elements;

[0018] Based on the spatial layout information and the visual elements, a virtual UI topology map is generated.

[0019] Optionally, determining the target functional control and its semantic label based on the image and user operation context information of the current interface includes:

[0020] The image of the current interface, the spatial layout information, and the user operation context information are input into a preset model, and the target functional control and the second semantic label of the target functional control are output. The preset model identifies the visual elements in the image of the current interface, obtains the feature information of the visual elements, and determines the second semantic label and second confidence level of the candidate controls in the current interface based on the feature information of the visual elements, the spatial layout information, and the user operation context information. The candidate control with the highest second confidence level is selected as the target functional control.

[0021] Optionally, determining the interactive prompt information based on the current touch point's position, the semantic label of the target functional control, and the coordinate information of the target functional control includes:

[0022] Based on the current position of the touch point and the coordinate information of the target function control, the spatial distance between the current position of the touch point and the target function control is determined, and vibration feedback information is determined based on the spatial distance.

[0023] If the current touch point enters the valid clickable area of ​​the target function control, a voice prompt is generated based on the semantic label of the target function control; wherein, the valid clickable area of ​​the target function control is determined based on the coordinate information of the target function control.

[0024] Optionally, determining the interactive prompt information based on the current touch point's position, the semantic label of the target functional control, and the coordinate information of the target functional control includes:

[0025] An interactive navigation path is generated based on the virtual UI topology map, the position of the current touch point, and the coordinate information of the target functional control;

[0026] Based on the interactive navigation path, the current touch point position, the semantic label of the target function control, and the coordinate information of the target function control, vibration feedback information and voice prompt information are generated.

[0027] A second aspect of the present invention provides a human-computer interaction device, comprising:

[0028] An acquisition unit is used to acquire the user's touch point information in real time; wherein, the touch point information includes the position of the touch point and the sampling time;

[0029] The navigation state monitoring unit is used to determine whether the user is in an interactive navigation state based on the position of the touch point and the sampling time; wherein, the interactive navigation state refers to the state when the user experiences anxiety and performs invalid operations in a complex interface because they cannot find the target;

[0030] The first determining unit is used to determine the target functional control and the semantic label of the target functional control based on the image of the current interface and the user operation context information if the user is in an interactive maze state.

[0031] The second determining unit is used to determine the interactive prompt information based on the current touch point position, the semantic label of the target functional control, and the coordinate information of the target functional control.

[0032] Optionally, the navigation status monitoring unit includes:

[0033] The indicator determination unit is used to determine the trajectory indicator and the invalid dwell indicator based on the position of the touch point and the sampling time; wherein, the invalid dwell indicator refers to a user behavior metric that measures the cumulative duration of the user in a non-interactive area and / or without triggering a valid control response;

[0034] The interactive disorientation coefficient determination unit is used to determine the interactive disorientation coefficient based on the trajectory index and the invalid stay index.

[0035] Optionally, the first determining unit includes:

[0036] The first input unit is used to input the image of the current interface and user operation context information into a preset model, and output the target functional control and the first semantic label of the target functional control; wherein, the preset model identifies visual elements in the image of the current interface, obtains the feature information of the visual elements, and determines the first semantic label and first confidence level of the candidate controls in the current interface based on the feature information of the visual elements and the user operation context information, and selects the candidate control with the highest first confidence level as the target functional control.

[0037] Optionally, the human-computer interaction device further includes:

[0038] The spatial layout determination unit is used to identify visual elements in the image of the current interface and determine spatial layout information based on the position information, size information, region affiliation relationship and adjacency relationship of all the visual elements.

[0039] The virtual UI topology generation unit is used to generate a virtual UI topology based on the spatial layout information and the visual elements.

[0040] Optionally, the first determining unit includes:

[0041] The second input unit is used to input the image of the current interface, the spatial layout information, and the user operation context information into a preset model, and output the target functional control and the second semantic label of the target functional control; wherein, the preset model identifies the visual elements in the image of the current interface, obtains the feature information of the visual elements, and determines the second semantic label and second confidence level of the candidate controls in the current interface based on the feature information of the visual elements, the spatial layout information, and the user operation context information, and selects the candidate control with the highest second confidence level as the target functional control.

[0042] Optionally, the second determining unit includes:

[0043] The vibration feedback information determination unit is used to determine the spatial distance between the current touch point and the target function control based on the current touch point position and the coordinate information of the target function control, and to determine vibration feedback information based on the spatial distance.

[0044] The voice prompt information generation unit is used to generate voice prompt information based on the semantic label of the target function control if the current touch point enters the valid click area of ​​the target function control; wherein the valid click area of ​​the target function control is determined based on the coordinate information of the target function control.

[0045] Optionally, the second determining unit includes:

[0046] An interactive navigation path generation unit is used to generate an interactive navigation path based on the virtual UI topology map, the position of the current touch point, and the coordinate information of the target functional control;

[0047] The generation unit is used to generate vibration feedback information and voice prompt information based on the interactive navigation path, the position of the current touch point, the semantic label of the target function control, and the coordinate information of the target function control.

[0048] A third aspect of the present invention provides an electronic device, comprising:

[0049] One or more processors;

[0050] A storage device on which one or more programs are stored;

[0051] When the one or more programs are executed by the one or more processors, the one or more processors implement the human-computer interaction method as described in any one of the first aspects.

[0052] A fourth aspect of the present invention provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the human-computer interaction method as described in any one of the first aspects.

[0053] As can be seen from the above solutions, the present invention provides a human-computer interaction method, device, electronic device, and computer storage medium. This method determines whether the user is in an interactive maze state by the position of the touch point and the sampling time. If the user is in an interactive maze state, it performs joint reasoning based on the current interface image and user operation context information to determine the target functional control most relevant to the current user operation and the semantic label of the target control. Unlike existing technologies that modify the source code of third-party native applications to pre-configure semantic labels for controls, this fundamentally solves the common problem of incomplete adaptation in existing mobile native applications. Then, based on the current touch point position, the semantic label of the target functional control, and the coordinate information of the target functional control, interactive prompts are generated to guide the user to find the target functional control. Because the present invention is based on universal visual understanding capabilities, its navigation logic does not change with differences in the internal code implementation of different apps. It constructs a standard, stable, and non-intrusive native system assistive operation specification for visually impaired users, enabling them to obtain a consistent interactive experience across applications regardless of the developer's adaptation level. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0055] Figure 1 A flowchart of a human-computer interaction method provided in an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of a human-computer interaction device provided in another embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0059] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties.

[0060] It should be noted that the concepts of "first" and "second" mentioned in this invention are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0061] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0062] This invention provides a human-computer interaction method, such as... Figure 1 As shown, the specific steps include:

[0063] S101. Obtain user touch point information in real time.

[0064] The touch point information includes the location of the touch point and the sampling time.

[0065] In the actual application of this invention, the touch point information of the user's finger on the native screen can be obtained in real time through the system's underlying accessibility service interface, but this is not limited to that.

[0066] Specifically, you can register to listen for events such as TYPE_TOUCH_INTERACTION_START and TYPE_VIEW_HOVER_ENTER through the Accessibility Service, to obtain the user's touch point information in real time.

[0067] S102. Based on the location of the touch point and the sampling time, determine whether the user is in an interactive maze state.

[0068] Interactive disorientation refers to the state in which users experience anxiety and make ineffective swipes or operations when they cannot find their target in a complex interface. Currently, existing technologies cannot detect the user's "disorientation" state in real time and actively intervene in navigation, resulting in extremely low interaction efficiency.

[0069] Optionally, in another embodiment of the present invention, one implementation of step S102 includes the following steps (step A1-step A2):

[0070] Step A1: Based on the location of the touch point and the sampling time, determine the trajectory index and the invalid dwell index.

[0071] In the practical application of this invention, the user's continuous touch point sequence (x_i, y_i, t_i) can be collected within a preset time window at a preset sampling frequency (e.g., 60Hz). x_i and y_i represent the position of the touch point at the i-th sampling moment. x_i is the horizontal coordinate of the touch point at the i-th sampling moment, y_i is the vertical coordinate of the touch point at the i-th sampling moment, and t_i represents the sampling moment.

[0072] The trajectory index, based on the touch point's position and sampling time, can be determined by determining the sliding direction based on the touch point's position at adjacent sampling times, and then dividing the sliding direction into N discrete directional states (e.g., N=8, including up, down, left, right, upper left, upper right, lower left, and lower right, a total of 8 directional states). Let n_k be the number of times the k-th directional state appears within the current time window, and M be the total number of times all directional states appear. Then the probability p_k of the k-th directional state can be expressed as:

[0073] p_k = n_k / M, where k = 1, 2, ..., N;

[0074] Then, the trajectory index H is calculated based on p_k:

[0075] H = -Σ(k=1 to N) [p_k ln(p_k)]; equivalently, it can also be written as:

[0076] H = -∑(k=1~N) p_k ln(p_k);

[0077] When a state in a certain direction does not appear, p_k = 0, and it is agreed that the corresponding term p_k ln(p_k) = 0.

[0078] Furthermore, to facilitate comparisons across different interfaces and time windows, the trajectory index H can be normalized to obtain the normalized trajectory index H_n:

[0079] H_n = H / ln(N); equivalently, it can be written as:

[0080] H_n= - [∑(k=1~N) p_k ln(p_k)] / ln(N);

[0081] The value of H_n ranges from [0, 1]. The more concentrated the user's swipe direction, the smaller H_n becomes; the more dispersed and disordered the user's swipe direction, the larger H_n becomes.

[0082] One method for determining the invalid dwell time index based on the location of the touch point and the sampling time is as follows:

[0083] First, based on the location and sampling time of consecutive touch points, determine the cumulative duration T_i of the user's time in the non-interactive area and / or without triggering a valid control response. If the preset invalid dwell time threshold is T_th, then the formula for calculating the invalid dwell index T_n can be:

[0084] T_n = min(T_i / T_th, 1); where the value of T_n is in the range of [0, 1].

[0085] Step A2: Determine the interactive disorientation coefficient based on trajectory indicators and invalid stay indicators.

[0086] Continuing with the above example, if the trajectory index is H, then the interactive navigation coefficient C can be calculated as follows:

[0087] C = αH + βT_n; where the weighting coefficients α and β must satisfy: α + β = 1, which is used to adjust the contribution ratio of the degree of trajectory disorder and the invalid stay time in the navigation determination. The weighting coefficients α and β can be preset and changed by experts, authorized technical personnel, etc., and are not limited here.

[0088] If the trajectory index is H_n, then the interactive navigation coefficient can be calculated as follows:

[0089] C = αH_n + βT_n.

[0090] Specifically, if the interaction maze coefficient is less than the maze state determination threshold C_th, that is, when C < C_th, the user is determined not to be in an interaction maze state; if the interaction maze coefficient is not less than the maze state determination threshold, that is, when C ≥ C_th, the user is determined to be in an interaction maze state.

[0091] It should be noted that the above formula is only used to illustrate one implementation of the navigation determination logic in this invention. Those skilled in the art can adjust the number of directional state divisions, time window length, weight coefficients, and threshold parameters according to the performance of the terminal device, sampling frequency, interface complexity, and user behavior characteristics, but this does not affect the basic technical idea of ​​this invention.

[0092] S103. If the user is in an interactive maze state, then based on the image of the current interface and the user operation context information, determine the target function control and the semantic label of the target function control.

[0093] The image of the current interface can be obtained by capturing the pixel frame buffer data of the current screen through, but is not limited to, calling MediaProjection or the system-level screenshot interface; this is not limited here. User operation context information includes, but is not limited to, the user's current page type, previous operation sequence, most recent touch position, swipe direction, hover area, current focus position, recent voice broadcast content, recent haptic feedback results, and the current task objective to be completed, etc., which are not limited here.

[0094] It should be noted that this invention quantifies the user's anxiety and disorientation level through an interactive disorientation coefficient C, solving the "interactive deadlock" problem where visually impaired users struggle to initiate effective help requests due to their inability to perceive the interface. When this invention determines that a user is in an interactive disorientation state, it can directly execute step S103 without requesting the user, proactively initiating interactive guidance. This transforms "passive notification" into "active navigation," overcoming the lag in traditional assistive tools that only provide feedback after the user touches controls. It reduces blind exploration in complex pages and significantly shortens the task completion path for users in complex native interfaces.

[0095] It should be noted that this invention achieves semantic reconstruction of UI components based on the images of the current interface and user operation context information. Even when native controls are completely missing labels, this invention can complete the semantic labels based on the images of the current interface and user operation context information, ensuring that visually impaired users can operate any native application without barriers. It fundamentally solves the limitation of existing accessibility tools (such as TalkBack) that heavily rely on developers manually configuring contentDescription or AccessibilityNodeInfo in the code, achieving full semantic reconstruction for any zero-adaptation app, ensuring that visually impaired users no longer experience a "perceptual disconnection" when facing unlabeled controls.

[0096] Of course, in the actual application of this invention, if the accessibility labels in the native controls, the control description information, or the semantic attributes in the system accessibility tree are complete and available, the semantic labels can be directly read and used to complete control identification, function confirmation, and subsequent navigation feedback. This part is a compatible use of the existing accessibility label reading mechanism and can also serve as the basic processing path of this invention to reduce computational overhead and improve response efficiency. When the system detects that the target functional control has missing labels, incomplete label semantics, incorrect label content, or inconsistencies between the label and the visual display of the interface, or when there are self-drawn controls, canvas controls, dynamically rendered controls, etc., in the interface that cannot be correctly parsed through conventional accessibility interfaces, the semantic labels of the target functional controls in the interface are then identified based on the image of the current interface and the user operation context information.

[0097] It is understood that the present invention may also completely avoid reading the semantic tags in the native controls, and identify the semantic tags of each functional control in the interface based on the image of the current interface and the user operation context information.

[0098] Optionally, in another embodiment of the present invention, one implementation of step S103 includes:

[0099] Input the image of the current interface and the user operation context information into the preset model, and output the target function control and the first semantic label of the target function control.

[0100] The preset model identifies visual elements in the image of the current interface, obtains the feature information of the visual elements, and determines the first semantic label and first confidence level of the candidate controls in the current interface based on the feature information of the visual elements and the user operation context information. The candidate control with the highest first confidence level is selected as the target functional control.

[0101] It should be noted that the preset model in this invention may be, but is not limited to, a Vision-Language Model (VLM), etc., and is not limited here.

[0102] In practical applications of this invention, the preset model can also be quantized and compressed, such as using INT4 or FP16 quantization, which is not limited here. The preset model can run on the SoC's dedicated NPU to ensure that the inference process does not occupy the system's main CPU resources and that data does not leave the endpoint, which is not limited here either.

[0103] In practical applications of this invention, the visual language model can also be pre-trained. Specifically, it can be trained, lightweightly fine-tuned, or its parameters adapted in conjunction with native mobile interface scenarios, visually impaired user interaction scenarios, and accessibility semantic completion tasks. This enables the model to possess specialized recognition capabilities for interface control categories, text content, spatial locations, interactive areas, and functional semantics. In this way, the model can more accurately handle semantic reconstruction tasks in areas with missing labels, self-drawn controls, dynamically rendered controls, and complex layout interfaces, thereby meeting the application requirements of this invention in real-time on-device recognition, navigation correction, and haptic navigation feedback.

[0104] The feature information of visual elements includes, but is not limited to, visual features, positional features, and layout features. In the actual application of this invention, the preset model can detect visual elements in the image of the current interface, classify visual elements into icons, buttons, text areas, input boxes, navigation bars, pop-up areas, etc., without limitation, and encode the feature information of visual elements, without limitation.

[0105] Furthermore, in the practical application of this invention, the visible text in the image of the current interface can be combined with the OCR results for text semantic encoding, which can then be used as an auxiliary feature.

[0106] Based on this, the preset model in this invention converts the user's current operation context into task constraint information or conditional prompt information, and performs multimodal fusion with the feature information of visual elements in the current interface image. In other words, the preset model in this invention not only identifies "what's on the interface," but also further infers "which components are most likely needed at this time, considering what the user is currently doing." Through this conditional constraint reasoning, the model can filter out the core functional components most relevant to the current interaction goal from multiple candidate visual elements, i.e., the target functional components.

[0107] Furthermore, the pre-defined model can output corresponding functional semantic labels and confidence scores for each candidate control. For example, for multiple buttons on a page, the model can determine whether they belong to functions such as "search," "back," "send," "confirm," "more operations," or "settings," and reorder these candidate results based on the user's current operation stage. The components that are ranked first, most relevant to the current task, and have the highest interaction priority can be identified as the core functional components of the current interface.

[0108] Furthermore, this invention can infer the page type by combining the overall characteristics of the current page, such as the homepage, search page, details page, playback page, settings page, publishing page, or pop-up page, and form a task context by combining the user's current operation stage. For example, if the user has previously entered the search process, the semantic weight of the magnifying glass icon, input box, and history area in the interface will be increased; when the user is in a playback control scenario, the pause, fast forward, and back icons at the bottom will be more likely to be identified as the current target function control.

[0109] Optionally, in another embodiment of the present invention, the output of the preset model can be further corrected by combining the spatial adjacency relationship between controls, the relationship between their respective regions, the relationship of reading order, and the historical interaction results.

[0110] For example, if an unlabeled icon is located to the right of the search box and the user has already entered keywords, that icon is more likely to be identified as a "search confirmation" button; if an icon is located in the upper left corner of the page and the user has just entered a new page, that icon is more likely to be identified as a "back" component. Thus, the system can avoid making static judgments based solely on icon appearance, and instead infer the true functional intent based on context.

[0111] Optionally, in another embodiment of the present invention, before determining the target functional control based on the image of the current interface and user operation context information, the method further includes:

[0112] Identify visual elements in the image of the current interface, and determine spatial layout information based on the position, size, region affiliation, and adjacency of all visual elements; generate a virtual UI topology map based on the spatial layout information and visual elements.

[0113] Among them, the virtual UI topology map is a structured representation of the interface that is dynamically constructed on the device side based on the visual recognition results and contextual semantic results of the current interface when it is impossible to directly rely on the native code structure or complete accessibility labels.

[0114] Specifically, a preset model can be used to detect and encode visual elements in the current interface image, extracting visual features, positional features, and layout features of visual elements such as icons, buttons, text areas, input boxes, navigation bars, and pop-up areas. Then, based on the visual features, positional features, and layout features of the visual elements, the positional information, size information, regional affiliation, and adjacency relationships of each element can be further extracted.

[0115] In the practical application of this invention, each identified visual element can be abstracted as a node, but is not limited to this. A node can correspond to a button, icon, text, input box, menu item, card container, navigation item, or other interactive object. Edges are established based on the spatial relationships, hierarchical relationships, reading order relationships, and possible interactive jump relationships between nodes to form a graph structure. Specifically, if two nodes are spatially adjacent, belong to the same functional area, are in the same row and column layout, have a parent-child containment relationship, or have a sequential access relationship in the normal interaction flow, then a relationship edge is established between the corresponding nodes. Relationship edges can include, but are not limited to, the following types: adjacent relationship edges, containment relationship edges, sorting relationship edges, same group relationship edges, and jump relationship edges; these are not limited here.

[0116] For example: multiple icon nodes in the bottom navigation bar of a page can be connected by a group relationship edge; an icon node and its text description node below it can be connected by a semantic pairing edge; a button node and its card area can be connected by an inclusion relationship edge; a "More" icon node and the menu item node that expands after being triggered can be connected by a potential jump relationship edge.

[0117] Based on this, the present invention organizes the above-mentioned nodes and edges in a unified manner to form a virtual UI topology diagram of the current interface.

[0118] It is important to emphasize that the virtual UI topology map is not derived from the original interface tree in the application development code, but is reconstructed by the system from visual pixel content and interaction semantics, hence the name "virtual UI topology map". This map can express in a structured way "what elements are in the interface", "where the elements are located", "what are the relationships between the elements", and "how the user may move between these elements".

[0119] Optionally, in another embodiment of the present invention, a runtime interaction mapping layer can be formed by mapping the semantics, coordinate range, interactive attributes, etc. of each identified interface element to the current screen pixel area based on the node information in the virtual UI topology graph. This layer emphasizes "which areas on the current screen can be located, perceived, and interacted with", and is used for subsequent touch hit detection, focus switching, navigation guidance, and haptic feedback output.

[0120] Optionally, in another embodiment of the present invention, the virtual UI topology map can also be used for path correction in a state of disorientation. When it is identified that the user is continuously sliding back and forth in a non-target area, staying in a blank area for a long time, or failing to advance the task progress, the virtual UI topology map can be used to determine which expected interaction path the user has deviated from, and the nearest interactive node to the target semantics can be automatically located. Then, correction suggestions can be output or an auxiliary feedback mechanism can be directly triggered. Specifically, this can include the following steps (steps B1-B4):

[0121] Step B1: After identifying the target functional control, generate an expected interaction path based on the node relationships in the virtual UI topology graph.

[0122] The expected interaction path is an ordered sequence of nodes from the user's current node (the current touch point) to the target node corresponding to the target function control. This sequence can be generated based on the adjacency, inclusion, sorting, grouping, and jump relationships between nodes, as well as the semantic priority of the nodes.

[0123] Step B2: Map the current position of the user's current touch point, current focus position, or the position of the most recent valid interaction to the current node in the virtual UI topology map in real time, and continuously record the sequence of nodes that the user actually passes through.

[0124] Step B3: Compare the actual sequence of nodes traversed by the user with the expected interaction path to determine whether the user has deviated from the path.

[0125] Specifically, a user can be determined to have deviated from the expected interaction path when one or more of the following situations occur:

[0126] 1. The user's current node is not on the expected interaction path;

[0127] 2. The user continuously enters node areas unrelated to the target task;

[0128] 3. Users repeatedly switch back and forth between several non-target nodes;

[0129] 4. After a user moves along the node sequence, the topological distance to the target node does not decrease but increases instead;

[0130] 5. Users spend a long time in blank areas, invalid areas, or low-relevance node areas.

[0131] Step B4: After determining that the user has deviated from the path, further search for interactive candidate nodes that are most relevant to the target semantics in the virtual UI topology map.

[0132] It should be noted that the interactive candidate node closest to the target semantics does not only refer to the node with the closest geometric distance, but is determined by a combination of the following factors:

[0133] 1. The similarity in semantic tags between interactive candidate nodes and target functional controls;

[0134] 2. The topological distance between interactive candidate nodes and the target node;

[0135] 3. The spatial distance between the interactive candidate node and the current touch point;

[0136] 4. The interactive attributes, regional affiliation, and task relevance of interactive candidate nodes.

[0137] The system can score and sort candidate nodes according to the above factors, and select the most suitable interactive node as a correction relay point or the next guiding point. This invention does not limit the specific scoring method.

[0138] Since the function of an icon cannot often be determined solely by its appearance, this invention can utilize the spatial layout information of nodes to more effectively identify the true intent of candidate controls, making semantic labels more accurate. For example, the same "three dots" icon, when located in the upper right corner of a video card, is more likely to represent "more operations," while when located in the top toolbar of the page, it is more likely to represent "menu" or "extended functions." Therefore, in another embodiment of this invention, one implementation of step S103 includes:

[0139] Input the current interface image, spatial layout information, and user operation context information into the preset model, and output the target function control and the second semantic label of the target function control.

[0140] The preset model identifies visual elements in the image of the current interface, obtains the feature information of the visual elements, and determines the second semantic label and second confidence level of the candidate controls in the current interface based on the feature information, spatial layout information and user operation context information. The candidate control with the highest second confidence level is selected as the target functional control.

[0141] By using a pre-defined model to fuse and analyze the images, spatial layout information, and user operation context information of the current interface, the true intent of candidate controls can be more effectively identified, making semantic labels more accurate.

[0142] Optionally, in another embodiment of the present invention, the virtual UI topology map can also be used for interaction result verification and semantic updates. When a user triggers a certain unlabeled icon, the true function of the icon is verified by reverse verification based on the change relationship between the new virtual UI topology map after the interface jumps and the old virtual UI topology map.

[0143] For example, if clicking the icon brings up a list of settings, the confidence level of recognizing it as a "settings" entry point increases; if clicking it brings up a sharing channel panel, its semantic meaning can be corrected to "share". Thus, the system can dynamically confirm and continuously optimize the icon's intent.

[0144] S104. Determine the interactive prompt information based on the current touch point position, the semantic label of the target function control, and the coordinate information of the target function control.

[0145] The interactive prompts include, but are not limited to, one or more combinations of vibration feedback and voice prompts, etc., and are not limited here. The coordinate information of the target functional component may be, but is not limited to, the coordinates of the center point of the target functional component.

[0146] It should be noted that the coordinate information of the target functional component can be extracted based on the image of the current interface, or provided directly by the system, etc., and is not limited here. The semantic label of the target functional control can be the first semantic label in the above embodiment, or it can be the second semantic label, and is not limited here.

[0147] Optionally, in another embodiment of the present invention, one implementation of step S104 specifically includes the following steps (steps C1-C2):

[0148] Step C1: Based on the position of the current touch point and the coordinate information of the target function control, determine the spatial distance between the current touch point and the target function control, and determine the vibration feedback information based on the spatial distance.

[0149] The spatial distance between the current touch point and the target function control can be calculated using Euclidean distance based on the coordinates of the center point of the current touch point and the target function control; this is not limited here. Vibration feedback information can be, but is not limited to, the vibration frequency and intensity of a linear resonant actuator (LRA); this is not limited here either.

[0150] In the practical application of this invention, the geometric center coordinates of the target functional control can be used as the center point coordinates of the target functional control. Assuming the bounding box of the target functional control is (x1, y1, x2, y2), its geometric center coordinates are (x0, y0), where x0=(x1+x2) / 2 and y0=(y1+y2) / 2.

[0151] The method for determining vibration feedback information f(d) based on spatial distance can be, but is not limited to, calculation using the following formula:

[0152] f(d) = k / (d+ε); where k is the vibration mapping scaling factor or gain factor, used to adjust the overall response intensity when mapping spatial distance d to the linear motor vibration frequency f(d). Its function is to calibrate the vibration frequency output range based on the linear motor performance of the terminal device, the user's tactile perception threshold, and navigation sensitivity requirements. In other words, under the same distance d, the larger k is, the higher the calculated vibration frequency f(d), and the more sensitive the tactile guidance; the smaller k is, the smoother the vibration change of f(d). Therefore, k is essentially a scaling parameter used to control the strength and sensitivity of the tactile gravitational field feedback. The closer the user's finger is to the target function control, the stronger or more rapid the vibration, thus creating a "tactile traction" effect, guiding the user to correct the sliding direction through touch, achieving precise positioning of the target function control, and effectively solving the problem of lack of interface mapping caused by simply relying on linear voice broadcasting.

[0153] The parameter ε is a smoothing constant or stabilizing term to prevent the denominator from approaching zero. Since the distance d may be close to 0 when the user's finger is very close to the target function control, writing the formula simply as f(d) = k / d would lead to an excessively large calculated vibration frequency or even mathematical singularities. To avoid this problem, a small positive number ε is introduced into the denominator, ensuring that the vibration frequency remains within a controllable range even when d is very small, thus guaranteeing the stability, continuity, and feasibility of the vibration output.

[0154] Step C2: If the current touch point enters the valid click area of ​​the target function control, then generate voice prompt information based on the semantic label of the target function control.

[0155] The effective clickable area of ​​the target function control is determined based on the coordinate information of the target function control.

[0156] Continuing with the above example, the area covered by the bounding box of the target function control, or the area expanded outward from the bounding box by a preset tolerance, is defined as the "valid click area". The valid click area can be represented as: x∈[x1-Δx, x2+Δx], y∈[y1-Δy, y2+Δy];

[0157] Where Δx and Δy are touch tolerances, used to compensate for touch deviations, model recognition errors, and screen sampling errors of visually impaired users.

[0158] If the system can only obtain the center point coordinates (x0, y0) of the target function control temporarily without obtaining the complete bounding box, a circular, elliptical, or rectangular hit area can be constructed centered at (x0, y0) based on the type of the target function control, its historical statistical size, or a preset click radius r, and this area can be used as the valid click area. For example, it can be defined as the area that satisfies (x-x0)^2+(y-y0)^2≤r^2.

[0159] This invention achieves deep decoupling and synchronization between physical vibration navigation and voice prompts in terms of timing. Voice prompts are no longer given during physical vibration (tactile) guidance; instead, a voice prompt is generated only the instant the current touch point enters the valid clickable area of ​​the target functional control. This effectively reduces the "auditory overload" caused by the frequent broadcasting of voice prompts in traditional tools, allowing visually impaired users to focus on perceiving the operation path and improving the continuity of task execution.

[0160] Of course, in the actual application of this invention, while generating voice prompts for voice broadcast, a high-intensity and short pulse vibration (Haptic Tick) can also be generated to more strongly indicate that the effective click area of ​​the target function control has been reached.

[0161] When the user's current touch point enters the valid clickable area of ​​the target function control, the system first reads the associated information of the target function control in the virtual UI topology map. The associated information includes at least: control semantic label, control type, page area, current interaction state, and suggested operation method. Subsequently, the system generates corresponding prompt statements according to the preset broadcast template and calls the terminal's local text-to-speech (TTS) engine to generate voice prompts, completing the voice broadcast.

[0162] In the practical application of this invention, the content of the voice prompt information can be generated by combining the following information, which is not limited here:

[0163] 1. The functional semantics of the target control, such as "Submit Order", "Search", "Back", "Play";

[0164] 2. Type information of the target control, such as "button", "input box", "menu item";

[0165] 3. Current status information, such as "located", "clickable", "selected", "not activated";

[0166] 4. Operation prompts, such as "Please double-click to execute", "Swipe right to proceed to the next item", "Long press to expand for more operations".

[0167] For example, when the system recognizes that the current target is the "Submit Order" button on the payment page and detects that the user's finger has entered its valid click area, it can generate a voice prompt message "Submit Order has been located, please double-click to execute" according to the template of "status words + control semantics + operation prompts", which can then be broadcast in real time by the client-side TTS engine.

[0168] Optionally, in another embodiment of the present invention, one implementation of step S104 specifically includes the following steps (steps D1-D2):

[0169] Step D1: Generate an interactive navigation path based on the virtual UI topology map, the position of the current touch point, and the coordinate information of the target functional control.

[0170] Specifically, based on the topological relationship of the user's current touch point position and the coordinate information of the target function control in the virtual UI topology map, a more reasonable exploration order and movement path are planned, namely the interactive navigation path.

[0171] In the practical application of this application, the method for generating interactive navigation paths specifically includes the following steps (steps E1-E3):

[0172] Step E1: After generating the virtual UI topology graph, each interactive object in the current interface can be abstracted into a node, and the adjacency, containment, sorting, grouping and jump relationships between nodes can be abstracted into edges, thus forming a graph structure for navigation.

[0173] Step E2: Map the current touch position (the position of the touch point) to the node corresponding to its pixel area, or determine the nearest interactive node to the current touch position as the starting node. There is no limitation here.

[0174] Step E3: Perform path search based on the topological relationships between nodes in the virtual UI topology graph to obtain the interactive navigation path from the current node to the target node (coordinate information of the target functional control).

[0175] Specifically, path costs can be set for each edge according to the spatial proximity between nodes, directional consistency, reading order relationship, regional affiliation relationship, and interactive jump priority. Then, a graph search method, such as shortest path search, heuristic search, or layer-by-layer traversal, is used to search from the starting node to the target node to obtain the node sequence with the minimum cost or the highest priority, which is used as the interactive navigation path. No restrictions are imposed here.

[0176] Optionally, in another embodiment of this application, the present invention can also convert interactive navigation paths into navigation prompts for visually impaired users.

[0177] For example, based on the positional relationship between the next node and the current node in the path, voice prompts such as "adjacent node on the left" and "second node from the bottom right" are generated, and corresponding vibration feedback information is generated by combining the distance or hierarchical changes between nodes in the path. When the system detects that the user has deviated from the currently planned path, it can also re-determine the starting node based on the updated current touch position and re-plan the interactive navigation path, thereby achieving dynamic correction.

[0178] Step D2: Based on the interactive navigation path, the current touch point position, the semantic label of the target function control, and the coordinate information of the target function control, generate vibration feedback information and voice prompt information.

[0179] Specifically, the direction of the target function control is determined based on the position of the current touch point and the coordinate information of the target function control, and the number of nodes that will be passed in this direction is determined based on the interactive navigation path in order to generate voice prompts. For example, the target function control is located at the second node to the right of the current touch point.

[0180] Furthermore, by combining the semantic tags of the target function control, voice prompts can be generated. Assuming the semantic tag of the target function control is "submit button", the voice prompt can be: the submit button is located at the second node to the right of the current touch point.

[0181] Furthermore, different vibration feedback information can be set to prompt users to move to adjacent interactive areas in the interactive navigation path, thereby reducing blind sliding and repeated exploration. This is not limited here.

[0182] In the practical application of this invention, in order to ensure the continuity of interaction and data security, the system can adopt a fully localized processing scheme: from image acquisition and model inference to the distribution of interactive prompts, everything is completed within the local SoC of the mobile phone, without involving any external network transmission.

[0183] In the practical application of this invention, a dedicated AI computing core (such as an NPU) on the edge can also be used to quantize and accelerate the preset model, effectively reducing the loop delay of "perception-reasoning-feedback". Through experiments, it can currently be controlled within 50 milliseconds, thereby ensuring real-time synchronization between haptic guidance and user gestures.

[0184] In the practical application of this invention, system-level permission management is also used to ensure that visual analysis is only used to generate navigation instructions and does not record or store any sensitive content in the user interface, thus protecting user privacy and security.

[0185] In the practical application of this invention, an Enhanced Accessibility Service can be integrated into the Android framework layer to implement this invention. This service runs silently in the background without requiring any modification to the source code of any third-party native application.

[0186] As can be seen from the above scheme, the present invention provides a human-computer interaction method that determines whether the user is in an interactive maze state by the position of the touch point and the sampling time. If the user is in an interactive maze state, it performs joint reasoning based on the image of the current interface and the user operation context information to determine the target functional control most relevant to the current user operation and the semantic label of the target control. Unlike existing technologies, it does not require modifying the source code of third-party native applications to pre-configure semantic labels for controls, fundamentally solving the problem of incomplete adaptation that is common in existing mobile native applications. Then, based on the position of the current touch point, the semantic label of the target functional control, and the coordinate information of the target functional control, interactive prompt information is generated to guide the user to find the target functional control. Since the present invention is based on universal visual understanding ability, its navigation logic does not change with the differences in the internal code implementation of different apps. It builds a standard, stable, and non-intrusive native system assistive operation specification for visually impaired users, enabling them to obtain a consistent interactive experience across applications regardless of the developer's adaptation level.

[0187] Another embodiment of the present invention provides a human-computer interaction device, such as... Figure 2 As shown, it specifically includes:

[0188] The acquisition unit 201 is used to acquire the user's touch point information in real time.

[0189] The touch point information includes the location of the touch point and the sampling time.

[0190] The maze state monitoring unit 202 is used to determine whether the user is in an interactive maze state based on the position of the touch point and the sampling time.

[0191] The first determining unit 203 is used to determine the target functional control and its semantic label based on the image of the current interface and the user operation context information if the user is in an interactive maze state.

[0192] The second determining unit 204 is used to determine interactive prompt information based on the current touch point position, the semantic label of the target functional control, and the coordinate information of the target functional control.

[0193] For details on the specific operation of the units disclosed in the above embodiments of the present invention, please refer to the corresponding method embodiments, such as... Figure 1 As shown, it will not be elaborated further here.

[0194] As can be seen from the above solutions, this invention provides a human-computer interaction device that determines whether the user is in an interactive maze state by the position of the touch point and the sampling time. If the user is in an interactive maze state, it performs joint reasoning based on the image of the current interface and the user's operation context information to determine the target functional control most relevant to the current user operation and the semantic label of the target control. Unlike existing technologies, it does not require modifying the source code of third-party native applications to pre-configure semantic labels for controls, fundamentally solving the problem of incomplete adaptation that is common in existing mobile native applications. Then, based on the position of the current touch point, the semantic label of the target functional control, and the coordinate information of the target functional control, interactive prompts are generated to guide the user to find the target functional control. Since this invention is based on universal visual understanding capabilities, its navigation logic does not change with the differences in the internal code implementation of different apps. It builds a standard, stable, and non-intrusive native system assistive operation specification for visually impaired users, enabling them to obtain a consistent interactive experience across applications regardless of the developer's adaptation level.

[0195] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field-Programmable Gate Array (FPGA), Application-Specific Integrated Circuit (ASIC), Application-Specific Standard Product (ASSP), System on a Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0196] Another embodiment of the present invention provides an electronic device, comprising:

[0197] One or more processors.

[0198] A storage device on which one or more programs are stored.

[0199] When the one or more programs are executed by the one or more processors, the one or more processors implement the human-computer interaction method as described in the above embodiments.

[0200] Another embodiment of the present invention provides a computer storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the human-computer interaction method as described in the above embodiments.

[0201] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0202] It should be noted that the computer-readable medium described above in this invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0203] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0204] Another embodiment of the present invention provides a computer program product, which, when executed, is used to perform the above-described human-computer interaction method.

[0205] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, it performs the functions defined in the methods of the embodiments of the present invention.

[0206] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in this invention is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely exemplary forms for implementing the invention.

[0207] While several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of the invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0208] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with technical features of the present invention (but not limited to) that have similar functions.

Claims

1. A human-computer interaction method, characterized in that, include: Real-time acquisition of user touch point information; wherein, the touch point information includes the position of the touch point and the sampling time; Based on the position of the touch point and the sampling time, it is determined whether the user is in an interactive disorientation state; wherein, the interactive disorientation state refers to the state when the user experiences anxiety and performs invalid operations due to not being able to find the target in a complex interface; If the user is in an interactive maze state, the target functional control and its semantic label are determined based on the current interface image and user operation context information. Based on the current touch point position, the semantic label of the target function control, and the coordinate information of the target function control, determine the interactive prompt information.

2. The human-computer interaction method according to claim 1, characterized in that, Determining whether the user is in an interactive navigation state based on the position of the touch point and the sampling time includes: Based on the location of the touch point and the sampling time, a trajectory indicator and an invalid dwell indicator are determined; wherein, the invalid dwell indicator refers to a user behavior metric that measures the cumulative duration of time a user spends in a non-interactive area and / or without triggering a valid control response. Based on the trajectory index and the invalid stay index, the interactive disorientation coefficient is determined.

3. The human-computer interaction method according to claim 1, characterized in that, The process of determining the target functional control and its semantic label based on the current interface image and user operation context information includes: The image of the current interface and the user operation context information are input into a preset model, and the target functional control and the first semantic label of the target functional control are output. The preset model identifies the visual elements in the image of the current interface, obtains the feature information of the visual elements, and determines the first semantic label and first confidence level of the candidate controls in the current interface based on the feature information of the visual elements and the user operation context information. The candidate control with the highest first confidence level is selected as the target functional control.

4. The human-computer interaction method according to claim 1, characterized in that, Before determining the target functional control based on the image and user operation context information of the current interface, the process also includes: Identify visual elements in the image of the current interface, and determine spatial layout information based on the position information, size information, regional affiliation relationship and adjacency relationship of all the visual elements; Based on the spatial layout information and the visual elements, a virtual UI topology map is generated.

5. The human-computer interaction method according to claim 4, characterized in that, The process of determining the target functional control and its semantic label based on the current interface image and user operation context information includes: The image of the current interface, the spatial layout information, and the user operation context information are input into a preset model, and the target functional control and the second semantic label of the target functional control are output. The preset model identifies the visual elements in the image of the current interface, obtains the feature information of the visual elements, and determines the second semantic label and second confidence level of the candidate controls in the current interface based on the feature information of the visual elements, the spatial layout information, and the user operation context information. The candidate control with the highest second confidence level is selected as the target functional control.

6. The human-computer interaction method according to claim 1, characterized in that, The interactive prompt information is determined based on the current touch point position, the semantic label of the target functional control, and the coordinate information of the target functional control, including: Based on the current position of the touch point and the coordinate information of the target function control, the spatial distance between the current position of the touch point and the target function control is determined, and vibration feedback information is determined based on the spatial distance. If the current touch point enters the valid clickable area of ​​the target function control, a voice prompt is generated based on the semantic label of the target function control; wherein, the valid clickable area of ​​the target function control is determined based on the coordinate information of the target function control.

7. The human-computer interaction method according to claim 4, characterized in that, The interactive prompt information is determined based on the current touch point position, the semantic label of the target functional control, and the coordinate information of the target functional control, including: An interactive navigation path is generated based on the virtual UI topology map, the position of the current touch point, and the coordinate information of the target functional control; Based on the interactive navigation path, the current touch point position, the semantic label of the target function control, and the coordinate information of the target function control, vibration feedback information and voice prompt information are generated.

8. A human-computer interaction device, characterized in that, include: An acquisition unit is used to acquire the user's touch point information in real time; wherein, the touch point information includes the position of the touch point and the sampling time; The navigation state monitoring unit is used to determine whether the user is in an interactive navigation state based on the position of the touch point and the sampling time; wherein, the interactive navigation state refers to the state when the user experiences anxiety and performs invalid operations in a complex interface because they cannot find the target; The first determining unit is used to determine the target functional control and the semantic label of the target functional control based on the image of the current interface and the user operation context information if the user is in an interactive maze state. The second determining unit is used to determine the interactive prompt information based on the current touch point position, the semantic label of the target functional control, and the coordinate information of the target functional control.

9. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the human-computer interaction method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, It stores a computer program, wherein the computer program, when executed by a processor, implements the human-computer interaction method as described in any one of claims 1 to 7.

Images