Control method

Abstract

The invention provides a control method. The method comprises the following steps: in response to triggering of a target key, displaying a first element displayed in a first effect in a target window as a second effect so as to indicate that the first element has an input focus of an input component; the first element is an element which can be clicked for interaction in the target window, and the determination of the first element is at least related to an input behavior before a target operation triggers the target key; the target key is a function key in the input component, and the target window is a display window in a display component.

Description

Technical Field

[0001] This application relates to computer technology, and particularly to a control method. Background Art

[0002] In traditional window interaction control, the switching order of the input focus is usually determined by the window structure and the default values of elements, and the switching order of the input focus is often fixed. However, the fixed switching order of the input focus often fails to meet the needs of users in complex interaction scenarios. The traditional fixed switching order of the input focus has the problem that it cannot be adjusted according to the user's input behavior, which in turn leads to a poor user experience. Summary of the Invention

[0003] An embodiment of this application provides a control method.

[0004] According to a first aspect of this application, a control method is provided, the method includes: in response to a target key being triggered, for a first element displayed with a first effect in a target window, display it with a second effect to indicate that the first element has the input focus of an input component; the first element is an element in the target window that can be clicked for interaction, and the determination of the first element is at least related to the input behavior before the target operation triggers the target key; the target key is a function key in the input component, and the target window is a display window in a display component.

[0005] According to an embodiment of this application, the method further includes: based on the input behavior, determine a first element sequence corresponding to the elements in the target window; the first element sequence represents the switching order of the input focus between the elements; the first element sequence includes at least the first element and a second element; when the first element has the input focus, trigger the target key again, and the second element has the input focus.

[0006] According to an embodiment of this application, the determination of the first element sequence corresponding to the elements in the target window includes: determine first attribute information of the elements in the target window, and sort the elements in the target window based on the first attribute information to obtain the first element sequence; the first attribute information of the elements includes at least a first type of attribute value and a second type of attribute value. In the first element sequence, the elements corresponding to the first type of attribute value are arranged after the elements corresponding to the second type of attribute value, and the elements corresponding to the second type of attribute value are sorted in ascending order of numerical value.

[0007] According to an embodiment of this application, the method further includes: input the input behavior into a first model, and input the first attribute information into the first model; obtain the first element sequence output by the first model.

[0008] According to an embodiment of the present application, the method further includes: determining a clustering element group corresponding to the target window, and detecting the first element sequence based on the clustering element group; and / or, detecting the first element sequence based on the display position relationship between the elements in the target window.

[0009] According to an embodiment of the present application, the determining the clustering element group corresponding to the target window includes: obtaining the source code of the target window; inputting the source code into a second model to enable the second model to output the logical distance between the elements in the target window; and based on the logical distance, performing clustering analysis on the elements in the target window to determine the clustering element group corresponding to the target window.

[0010] According to an embodiment of the present application, the detecting the first element sequence based on the clustering element group includes: if the storage manner of the elements in the same clustering group in the first element sequence is that there are no elements of other groups between any two elements, determining that the first element sequence meets the target condition.

[0011] According to an embodiment of the present application, the detecting the first element sequence based on the display position relationship between the elements in the target window includes: determining a plurality of minimum execution units in the first element sequence; each of the minimum execution units includes at least two elements; based on the page layout information of the target window, determining the relative display position relationship between the elements in the minimum execution unit; and in response to the order of the elements in the same minimum execution unit in the first element sequence matching the relative display position relationship, determining that the first element sequence meets the target condition.

[0012] According to an embodiment of the present application, the method further includes: in response to the second element having the input focus and receiving a return instruction for the target window, adjusting the first element sequence to obtain a second element sequence; the second element sequence includes at least the first element and a third element; the third element is located after the first element; and the third element is different from the second element.

[0013] According to an embodiment of the present application, the method further includes: outputting the second element sequence; and in response to receiving an adjustment instruction for the second element sequence, adjusting the second element sequence based on the adjustment instruction. Description of the Drawings

[0014] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features, and advantages of the exemplary embodiments of the present application will become readily understandable. In the drawings, several embodiments of the present application are shown in an exemplary rather than restrictive manner, where:

[0015] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts.

[0016] Figure 1 Shows the schematic processing flow of the control method provided by the embodiment of the present application Figure 1 ;

[0017] Figure 2 Shows the schematic processing flow of the control method provided by the embodiment of the present application Figure 2 ;

[0018] Figure 3 Shows the schematic processing flow of the control method provided by the embodiment of the present application Figure 3 ;

[0019] Figure 4 Shows the schematic processing flow of the control method provided by the embodiment of the present application Figure 4 ;

[0020] Figure 5 Shows the schematic processing flow of the control method provided by the embodiment of the present application Figure 5 ;

[0021] Figure 6 Shows the schematic processing flow of the control method provided by the embodiment of the present application Figure 6 ;

[0022] Figure 7 Shows the schematic processing flow of the control method provided by the embodiment of the present application Figure 7 ;

[0023] Figure 8 Shows an application scenario diagram of the control method provided by the embodiment of the present application;

[0024] Figure 9 Shows the schematic composition structure diagram of the electronic device provided by the embodiment of the present application. Detailed implementation manners

[0025] To make the objectives, features, and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0026] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0027] In the following description, the terms "first / second" are only used to distinguish similar objects and do not represent a specific order for the objects. It is understood that "first / second" can be interchanged in a specific order or sequence when allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0029] The processing flow in the control method provided by the embodiments of this application will be described. Refer to Figure 1 , Figure 1 which is a schematic diagram of the processing flow of the control method provided by the embodiments of this application Figure 1 , and will be described in conjunction with Figure 1 step S101 shown.

[0030] Step S101: In response to the target key being triggered, for the first element displayed with a first effect in the target window, display it with a second effect to indicate that the first element has the input focus of the input component; the first element is an element in the target window that can be clicked for interaction, and the determination of the first element is at least related to the input behavior before the target operation triggers the target key; the target key is a function key in the input component, and the target window is a display window in the display component.

[0031] In some embodiments, the target button may include: a function button in the input component. The target button can be used to trigger a set operation or function. The input component may include: a hardware device for receiving user input. For example, hardware devices such as keyboards, mice, touchscreens, and styluses. Through the user's triggering actions (such as clicking, pressing, etc.), the target button can be triggered. The target window may include: a display window in the display component. The target window can be used to display content or elements. The target window may contain multiple jumpable elements. The first element may include: an element in the target window that can be clicked for interaction. The determination of the first element is at least related to the input behavior before the target operation triggers the target button. Specifically, the first element may include elements such as buttons, text boxes, and sliders. The first element may also include other elements, and the embodiments of the present application do not limit the specific first element. The first effect may be the default display effect of the first element when it does not obtain the input focus in the target window. The first effect can be used to indicate that the element is in an inactive state. The second effect may be the display effect of the first element after it obtains the input focus. The second effect can be used to indicate that the element currently has the input focus. The input focus can be used to identify the element in the target window that can currently receive input.

[0032] Among them, the determination of the input behavior before the target operation triggers the target button may include: collecting visual information such as the user's body, actions, and gestures through a collection device, and then determining the input behavior before the target operation triggers the target button based on an image recognition algorithm. The determination of the input behavior before the target operation triggers the target button may also include: obtaining the input event generated by the input component and determining the input behavior according to the input event. Different input behaviors correspond to different determined first elements.

[0033] The method of the embodiments of the present application can flexibly implement the switching of the input focus by dynamically responding to the user's input behavior to determine the display effect and focus state of the first element in the target window. According to the user's operation habits and the input mode of the input component, it automatically adjusts the target element and switching order of the input focus to ensure that the focus switching is more in line with the user's expectations and improves the user experience.

[0034] In some embodiments, the processing flow diagram of the control method Figure 2 , such as Figure 2 shown, the control method may further include:

[0035] Step S201, based on the input behavior, determine the first element sequence corresponding to the elements in the target window.

[0036] In this embodiment, the first element sequence may represent the switching order in which the input focus switches between elements. The first element sequence may include a first element and a second element; when the first element has the input focus, triggering the target key again causes the second element to have the input focus. The determination time of the first element sequence may be before the target key is triggered, or the determination time of the first element sequence may also be after the target key is triggered.

[0037] As an example, the determined input behavior is to input from left to right. For example, in a spreadsheet software, when it is determined that the input behavior is to fill in a table in the target window, data is usually input into cells in sequence from left to right. Based on the determined input behavior, the system will determine the first element sequence corresponding to the elements in the target window. The first element sequence includes a first cell and a second cell. The first cell is before the second cell. When the user finishes inputting the content of the first cell, in response to the tab key being triggered, the input focus will automatically move to the second cell on the right. For example, if the current cell is the cell in the first row and first column of the table and the tab key on the keyboard is triggered, the cell in the first row and second column will obtain the input focus.

[0038] The method of the embodiment of the present application realizes dynamic adjustment of the input focus switching order by determining the first element sequence based on the input behavior. According to the user's operation habits and the input mode of the input component, an element switching order matching the input behavior is generated, and the target element and switching order of the input focus are automatically adjusted, so that the user does not need to manually adjust the focus during the interaction control process, ensuring that the focus switching is more in line with the user's expectations and improving the user experience.

[0039] In some embodiments, the processing flow of the control method is schematically shown Figure 3 , as Figure 3 shown, determining the first element sequence corresponding to the elements in the target window in step S201 may include:

[0040] Step S301, determining the first attribute information of the elements in the target window.

[0041] In this embodiment, the first attribute information of the element may include a first type of attribute value and a second type of attribute value. In the first element sequence, the elements corresponding to the first type of attribute value are arranged after the elements corresponding to the second type of attribute value, and the elements corresponding to the second type of attribute value are sorted in ascending order of numerical value.

[0042] Step S302, sorting the elements in the target window based on the first attribute information to obtain the first element sequence.

[0043] As an example, in a web page environment, traverse the source code of the target window of the web page to obtain the tabindex values of all jumpable elements in the target window. The tabindex values can include a first type of attribute value 0 and a second type of attribute values 1, 2, and 3. Subsequently, the elements are divided into two groups: one group is the elements with a positive integer tabindex value, sorted in ascending order according to their numerical values; the other group is the elements with a tabindex value of 0, arranged in the order of their appearance in the source code. Finally, the elements corresponding to the first type of attribute value are arranged after the elements corresponding to the second type of attribute value, and merged to form a first element sequence.

[0044] The method of the embodiment of the present application determines the first attribute information of the elements in the target window, and sorts the elements based on the first attribute information. The obtained first element sequence is more logical and in line with user expectations. According to the sorting rules of the first type of attribute value and the second type of attribute value, the elements are arranged in a certain priority and order, avoiding chaos and unreasonable order when the input focus switches. According to the determined first element sequence, the target element and the switching order of the input focus are automatically adjusted to ensure that the focus switching is more in line with user expectations and improves the user experience.

[0045] In some embodiments, the processing flow of the control method is schematically shown Figure 4 , such as Figure 4 shown, the control method may further include:

[0046] Step S401, input the input behavior into the first model, and input the first attribute information into the first model.

[0047] Step S402, obtain the first element sequence output by the first model.

[0048] In some embodiments, the first model may be a trained machine learning model. The first model can be used to comprehensively analyze the input behavior and the first attribute information, and output the optimal first element sequence based on data such as the visibility, position, and functional importance of the elements. For example, when the input behavior represents that the user is used to using the keyboard Tab key to navigate the elements in the target window of the page in the order from bottom to top, the first model will sort the elements in the order from bottom to top and in ascending order of the tabindex value according to the received input behavior corresponding to the keyboard Tab key and the tabindex values of the jumpable elements, output the first element sequence, and obtain the output first element sequence. The output first element sequence can ensure that every time the user presses the Tab key, the input focus can switch between elements in an order that conforms to the user's habits and page logic.

[0049] The method of the embodiment of the present application inputs the input behavior and the first attribute information into the first model. Through the processing and calculation of the model, the first element sequence can be generated more accurately. The model can comprehensively consider various factors, such as data on the visibility, position, and functional importance of elements, etc. The elements are arranged in a certain priority and order through the model, avoiding chaos and unreasonable order when the focus switches. According to the first element sequence output by the model, the target element and the switching order of the input focus are automatically adjusted to ensure that the focus switching more meets the user's expectations and improves the user experience.

[0050] In some embodiments, the processing flow of the control method is schematically shown Figure 5 as Figure 5 shown, and the control method may further include:

[0051] Step S501: Based on the input behavior, determine the first element sequence corresponding to the elements in the target window.

[0052] The specific explanation for step S501 is the same as that for step S201 and will not be elaborated here.

[0053] Step S502: Determine the clustering element grouping corresponding to the target window, and detect the first element sequence based on the clustering element grouping.

[0054] In this embodiment, the clustering element grouping may include: grouping the elements in the target window according to functions. For example, in the target window of a web page, all the input boxes in the target window are grouped into one group, and all the buttons are grouped into another group. Detect the first element sequence according to the clustering element grouping, and judge whether the elements in the first element sequence conform to the grouping situation of the elements in the clustering element grouping.

[0055] In some embodiments, determining the clustering element grouping corresponding to the target window in step S502 may include: obtaining the source code of the target window; inputting the source code into the second model so that the second model outputs the logical distance between the elements in the target window; based on the logical distance, perform clustering analysis on the elements in the target window to determine the clustering element grouping corresponding to the target window.

[0056] In some embodiments, the first model can be used to generate a first element sequence. The first model sorts the elements in the target window based on the input behavior and the first attribute information of the elements, and outputs the first element sequence. The second model can be used to calculate the logical distance between the elements in the target window. The second model can also be used to perform a clustering analysis on the elements in the target window based on the logical distance to determine the grouping of the elements. For example, the second model uses a clustering algorithm to group elements with similar functions into one group according to the logical distance, obtaining the clustered element grouping. Performing a clustering analysis on the elements in the target window based on the logical distance to determine the clustered element grouping corresponding to the target window may further include: the third model uses a clustering algorithm to group elements with similar functions into one group according to the logical distance, obtaining the clustered element grouping. The first model, the second model, and the third model can be different models. The first model, the second model, and the third model can also be different sub-models based on the same model. The source code may include: the DOM (Document Object Model) structure of the target window. The source code may also include other code of the target window, which is not limited in the embodiments of the present application. The logical distance can be used to represent the degree of association between the elements in the target window in terms of functions, page layout, interaction methods, etc. The clustering analysis may include the K-means algorithm and the hierarchical clustering algorithm. The clustering analysis may also include other clustering algorithms, which are not limited in the embodiments of the present application.

[0057] In some embodiments, the detection of the first element sequence based on the clustered element grouping in step S502 may include: if the storage mode of the elements within the same clustering group in the first element sequence is such that no element from other groups is included between any two elements, it is determined that the first element sequence meets the target condition.

[0058] In some embodiments, the target condition may include that the input focus switching corresponding to the first element sequence conforms to the logic. For example, the clustering element grouping of the elements in the target window includes Group 1 and Group 2. Group 1 contains Element A, Element B, and Element C. Group 2 contains Element E, Element F, and Element D. If the first element sequence is ACBEFD, it is determined that the first element sequence meets the target condition because the elements A, C, and B in Group 1 appear continuously, and the elements E, F, and D in Group 2 also appear continuously, and there are no elements of other groups between any two elements. For another example, if the first element sequence is CBAEDF, it is determined that the first element sequence meets the target condition because the elements C, B, and A in Group 1 appear continuously, and the elements E, D, and F in Group 2 also appear continuously, and there are no elements of other groups between any two elements. However, if the first element sequence is AECBDF, it is determined that the first element sequence does not meet the target condition because the elements E and D of Group 2 are inserted between Element A and Element B of Group 1, and there are elements of other groups between the two elements.

[0059] Step S503: Detect the first element sequence based on the display position relationship between the elements in the target window.

[0060] In this embodiment, the display position relationship may include: the relative position relationship between the elements in the target window. Specifically, the display position relationship may include position information such as left - right, up - down, and front - back of the elements. The display position relationship may also include other information, which is not limited in the embodiments of the present application. The display position relationship can be determined through the page layout information of the target window.

[0061] In some embodiments, detecting the first element sequence based on the display position relationship between the elements in the target window in step S503 may include: determining multiple minimum execution units in the first element sequence; each minimum execution unit includes at least two elements; based on the page layout information of the target window, determining the relative display position relationship between the elements in the minimum execution unit; in response to the order of the elements within the same minimum execution unit in the first element sequence matching the relative display position relationship, determining that the first element sequence meets the target condition.

[0062] In some embodiments, the minimum execution unit may include: a subset consisting of at least two elements in a first element sequence. The elements in the minimum execution unit have a relevance in terms of function or layout. For example, in a target window of a web page, a username input box and a password input box may form a minimum execution unit. The target condition may include that the input focus switching corresponding to the first element sequence conforms to logic. The page layout information of the target window may include: layout attributes in CSS (Cascading Style Sheets), such as flex (flexible layout) and absolute (absolute positioning). The page layout information of the target window may also include other page layout information, which is not limited in the embodiments of the present application.

[0063] As an example, in a target window of a web page, the first element sequence may be: Element A (username input box) → Element B (password input box) → Element C (remember password checkbox) → Element D (login button) → Element E (forgot password button). Determine multiple minimum execution units among the elements in the first element sequence. For example, Elements A, B, and C are minimum execution unit 1, and minimum execution unit 1 is for inputting login information; Elements D and E are minimum execution unit 2, and minimum execution unit 2 is for receiving operations related to login. According to the page layout information of the target window, determine the relative display position relationship between the elements in each minimum execution unit. For example, in minimum execution unit 1, Element A is above Element B, and Element B is above Element C; in minimum execution unit 2, Element D is to the left of Element E. Determine whether the order of the elements within the same minimum execution unit in the first element sequence matches the relative display position relationship. For example, in minimum execution unit 1, the order of Elements A → B → C matches the relative display position relationship (A is above B, and B is above C); in minimum execution unit 2, the order of Elements D → E matches the relative display position relationship (D is to the left of E). If the order of the elements within the same minimum execution unit in the first element sequence completely matches the relative display position relationship, and all minimum execution units meet this condition, then determine that the first element sequence meets the target condition.

[0064] The method of the embodiment of the present application determines the clustering element grouping of the target window and detects the first element sequence based on the clustering element grouping, which can verify the rationality of the focus switching order corresponding to the first element sequence. By detecting the distribution of elements in the sequence, it is ensured that the elements within the same clustering element grouping can be switched continuously, avoiding the situation of cross-group jumps in the element sequence. At the same time, the detection of the first element sequence based on the display position relationship of the elements further ensures the consistency between the element sequence and the page layout, making the focus switching more in line with the user's visual intuition and expectations, avoiding chaos and unreasonable order when the focus switches between elements, and improving the user experience.

[0065] In some embodiments, the processing flow diagram of the control method Figure 6 , such as Figure 6 shown, the control method may further include:

[0066] Step S601: Based on the input behavior, determine the first element sequence corresponding to the elements in the target window.

[0067] Step S602a: Determine the clustering element grouping corresponding to the target window, and detect the first element sequence based on the clustering element grouping.

[0068] Step S602b: Detect the first element sequence based on the display position relationship between the elements in the target window.

[0069] The specific explanations for steps S601 - S602b are the same as those for steps S501 - S503, and will not be elaborated here.

[0070] As an example, determine the first element sequence of the elements in the target window. For example, the sequence ABCDEF formed by elements A, B, C, D, E, and F. Then, detect through step S602a or step S602b: In step S602a, it is determined that the clustering element grouping in the target window includes an input box group and a button group. The input box group includes elements A, B, and C, and the button group includes elements D, E, and F. Detect the first element sequence according to the input box group and the button group. If there is no cross-group element crossing in the first element sequence, it is determined that the input focus switching corresponding to the first element sequence is logical;

[0071] Alternatively, in step S602b, check whether the adjacent elements in the first element sequence are consistent with the adjacent relationship on the page to ensure that the sequence conforms to the actual layout of the elements on the page. For example, element A and element B form a minimum execution unit. The display positions of element A and element B on the target window are adjacent. If element A and element B are adjacent in the first element sequence, it is determined that the input focus switching corresponding to element A and element B in the first element sequence is logical. Traverse all the minimum execution units. If the input focus switching corresponding to all the minimum execution units is logical, it is determined that the input focus switching corresponding to the first element sequence is logical.

[0072] The method of the embodiment of the present application determines the clustering element grouping of the target window and detects the first element sequence based on the clustering element grouping, which can verify the rationality of the focus switching order corresponding to the first element sequence. By detecting the distribution of elements in the sequence, it is ensured that the elements within the same clustering element grouping can be switched continuously, avoiding the situation of cross-group jumps in the element sequence. Alternatively, the detection of the first element sequence based on the display position relationship of the elements further ensures the consistency between the element sequence and the page layout, making the focus switching more in line with the user's visual intuition and expectations, avoiding the chaos and unreasonable order of the elements during focus switching, and improving the user experience.

[0073] In some embodiments, the processing flow of the control method is schematically shown Figure 7 , such as Figure 7 shown, the control method may further include:

[0074] Step S701, in response to the second element having an input focus and receiving a return instruction for the target window, adjust the first element sequence to obtain a second element sequence.

[0075] In some embodiments, the second element sequence includes at least a first element and a third element; the third element is located after the first element; the third element is different from the second element. The return instruction can be used to return the input focus to the previous element. The return instruction may include: an instruction generated by pressing the "return" key or the "Esc" key on the keyboard. The return instruction may also include other types of return instructions, which are not limited in the embodiments of the present application.

[0076] As an example, the first element sequence corresponding to the target window of the web page is generated based on the tabindex value of the elements. The tabindex of element A is 0, the tabindex of element B is 1, the tabindex of element C is 2, the tabindex of element D is 3, and the tabindex of element E is 4. The input focus switching order corresponding to the first element sequence is: element B → element C → element D → element E → element A. When the input focus is on element C and a return instruction triggered by pressing the "return" key is received. Adjust the tabindex value according to the return instruction: set the tabindex value of element C to tabindex = 5, and reassign the tabindex values of other elements as: element A(0), element B(1), element D(2), element E(3), element C(4). The second element sequence obtained is: element B → element D → element E → element C → element A.

[0077] Step S702, output the second element sequence.

[0078] Step S703, in response to receiving an adjustment instruction for the second element sequence, adjust the second element sequence based on the adjustment instruction.

[0079] In this embodiment, the adjustment instruction can be used to adjust the current second element sequence. For example, by means of dragging and keyboard operations, etc., change the order of elements in the second element sequence in the adjustment interface. The currently output second element sequence is: element B → element D → element E → element C → element A. Display the icons corresponding to each element in the second element sequence in the adjustment interface. In response to the adjustment interface receiving an adjustment instruction for the icon, for example, the user moves the icon corresponding to element B after the icon corresponding to element E, adjust the tabindex values of the elements in the second element sequence: element A(0), element C(4), element D(1), element E(2), element B(3), so as to obtain a new second element sequence: element D → element E → element B → element C → element A.

[0080] In the method of the embodiment of the present application, when the user is not satisfied with the input focus switching order of certain elements, the sequence can be rearranged through a control instruction, and a new element sequence is generated according to the control instruction, ensuring that the input focus switching is more in line with the user's operation habits and expectations, avoiding chaos and unreasonable order when the focus switches between elements, and improving the user experience.

[0081] Figure 8 Shows an application scenario diagram of the control method provided by the embodiment of the present application.

[0082] Reference Figure 8, An application scenario of the control method provided by the embodiments of the present application, which is applied to the determination process of the input focus switching order of the elements of the target window. The determination process may include:

[0083] Step S1, based on the input behavior, determine the first element sequence corresponding to the elements in the target window. Specifically, the determined input behavior is to browse from top to bottom through the keyboard. For example, in the target window, when it is determined that the input behavior is to browse content in the target window, it is usually browsed in sequence from top to bottom. Based on the determined input behavior, determine the first element sequence corresponding to the elements in the target window. Specifically, determining the first element sequence corresponding to the elements in the target window may be to traverse the DOM structure of the target window to obtain the elements that can be focused by the keyboard and their corresponding parent elements, and obtain the corresponding focus tree structure. Traverse the focus tree structure to obtain the tabindex values corresponding to the elements. Sort based on the tabindex values to obtain the first element sequence.

[0084] Step S2, analyze the element correlation degree between the elements in the target window based on the AI (Artificial Intelligence) model. Specifically, input the source code of the target window into the AI model, and determine the logical distance between each pair of elements in the target window through the AI model. Cluster and group the elements in the target window according to the logical distance to obtain the clustered element groups representing the element correlation degree.

[0085] Step S3, detect the first element sequence based on the element correlation degree. Specifically, if the storage method of the elements within the same clustered group in the first element sequence is such that there are no elements from other groups between any two elements, determine that the detection result is that the first element sequence meets the switching logic of the element correlation degree.

[0086] Step S4, detect the first element sequence based on the display position relationship between the elements in the target window. Specifically, determine multiple minimum execution units in the first element sequence; each minimum execution unit includes at least two elements; based on the page layout information of the target window, determine the relative display position relationship between the elements in each minimum execution unit; in response to the order of the elements within the same minimum execution unit in the first element sequence matching the relative display position relationship, determine that the input focus switching of the elements corresponding to this minimum execution unit conforms to the logic. Traverse all minimum execution units. If the input focus switching corresponding to all minimum execution units conforms to the logic, then determine that the detection result is that the first element sequence meets the switching logic of the display position relationship.

[0087] Step S5: Adjust the first element sequence according to the detection result to obtain a second element sequence. Specifically, if the detection result is that the first element sequence does not meet the switching logic of the element association degree or the display position relationship, then adjust the first element sequence. The specific adjustment method is to modify the tabindex value of the elements in the first element sequence to obtain the second element sequence, and the focus switching order of the elements in the second element sequence conforms to logic and visual intuition. If the detection result is that the first element sequence meets the switching logic of the element association degree and the display position relationship, there is no need to adjust, and the first element sequence is directly output as the second element sequence.

[0088] It can be understood that Figure 8 the application scenarios of the control method are only some exemplary implementation manners in the embodiments of the present application. The application scenarios of the control method in the embodiments of the present application include but are not limited to Figure 8 the application scenarios of the control method shown.

[0089] According to an embodiment of the present application, the present application also provides an electronic device and a non-transitory computer-readable storage medium.

[0090] Figure 9 FIG. shows a schematic block diagram of an exemplary electronic device 800 that can be used to implement the embodiments of the present application. The electronic device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present application described and / or claimed herein.

[0091] As Figure 9 shown, the electronic device 800 includes a computing unit 801, which can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 802 or the computer program loaded from the storage unit 808 into the random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. The input / output (I / O) interface 805 is also connected to the bus 804.

[0092] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disc, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0093] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 executes the various methods and processes described above, such as the control method. For example, in some embodiments, the control method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the control method described above can be executed. Alternatively, in other embodiments, the computing unit 801 can be configured to execute the control method in any other suitable manner (e.g., by means of firmware).

[0094] The various embodiments of the systems and technologies described above in this document can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor, receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0095] The program code for implementing the methods of this application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable control devices, such that when the program codes are executed by the processor or controller, the functions / operations specified in the flowchart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, executed partially on the machine and partially on a remote machine as an independent software package, or executed entirely on a remote machine or server.

[0096] In the context of this application, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0097] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0098] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.

[0099] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, can also be a server of a distributed system, or a server incorporating a blockchain.

[0100] It should be understood that various forms of the processes shown above can be used, steps can be reordered, added, or deleted. For example, the steps recited in this application can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this application can be achieved, and no limitations are imposed herein.

[0101] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" can explicitly or implicitly include at least one of such features. In the description of this application, "a plurality" means two or more unless otherwise specifically defined.

[0102] As described above, the above is only the specific embodiment of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. A control method, comprising: In response to a target key being triggered, for a first element displayed with a first effect in a target window, display it with a second effect to indicate that the first element has the input focus of an input component; The first element is an element in the target window that can be clicked for interaction, and the determination of the first element is at least related to the input behavior before the target operation triggers the target key; The target key is a function key in the input component, and the target window is a display window in the display component.

2. The method according to claim 1, the method further comprising: Based on the input behavior, determine a first element sequence corresponding to the elements in the target window; The first element sequence represents the switching order in which the input focus switches between the elements; the first element sequence at least includes the first element and the second element; when the first element has the input focus, trigger the target key again, and the second element has the input focus.

3. The method according to claim 2, the determining the first element sequence corresponding to the elements in the target window includes: Determine the first attribute information of the elements in the target window, and sort the elements in the target window based on the first attribute information to obtain the first element sequence; the first attribute information of the elements at least includes a first type of attribute value and a second type of attribute value. In the first element sequence, the elements corresponding to the first type of attribute value are arranged after the elements corresponding to the second type of attribute value, and the elements corresponding to the second type of attribute value are sorted in ascending order of numerical value.

4. The method according to claim 3, the method further comprising: Input the input behavior into a first model, and input the first attribute information into the first model; Obtain the first element sequence output by the first model.

5. The method according to claim 2, the method further comprising: Determine the clustering element grouping corresponding to the target window, and detect the first element sequence based on the clustering element grouping; And / or, detect the first element sequence based on the display position relationship between the elements in the target window.

6. The method according to claim 5, the determining the clustering element grouping corresponding to the target window includes: Obtain the source code of the target window; Input the source code into a second model, so that the second model outputs the logical distance between the elements in the target window; Based on the logical distance, perform a clustering analysis on the elements in the target window to determine the clustering element grouping corresponding to the target window.

7. The method according to claim 5, the detecting the first element sequence based on the clustering element grouping includes: If the storage method of the elements in the same clustering group in the first element sequence is that no element of other groups is included between any two elements, determine that the first element sequence meets the target condition.

8. The method according to claim 5, the detecting the first element sequence based on the display position relationship between the elements in the target window includes: Determine a plurality of minimum execution units in the first element sequence; Each of the minimum execution units includes at least two elements; Based on the page layout information of the target window, determine the relative display position relationship between the elements in the minimum execution unit; In response to the order of the elements within the same minimum execution unit in the first element sequence matching the relative display position relationship, determine that the first element sequence meets the target condition.

9. The method according to claim 2, wherein the method further comprises: In response to the second element having the input focus and receiving a return instruction for the target window, adjust the first element sequence to obtain a second element sequence; the second element sequence includes at least the first element and a third element; the third element is located after the first element; the third element is different from the second element.

10. The method according to claim 9, wherein the method further comprises: Output the second element sequence; In response to receiving an adjustment instruction for the second element sequence, adjust the second element sequence based on the adjustment instruction.

Images