Virtual keyboard processing method and device, electronic equipment, storage medium and product

Abstract

The present disclosure provides a virtual keyboard processing method and device, electronic equipment, storage medium and product, relates to the technical field of interface traversal test, and particularly relates to the field of keyboard scene processing in the traversal test process. The specific implementation scheme is: in the traversal test process of the function of the application interface control, the state of the virtual keyboard in the application interface control is detected; in response to detecting that the virtual keyboard is in an open state, the number of continuous times that the virtual keyboard continuously stays in the open state is determined; and the virtual keyboard is processed according to the number of continuous times. Through the present disclosure, it can be ensured that the keyboard control depending on the virtual keyboard and the event associated with the keyboard control can be tested, and at the same time, the influence of the keyboard covering for a long time on the traversal efficiency is avoided.

Description

Technical Field

[0001] This disclosure relates to the field of interface traversal testing technology, and more particularly to the field of keyboard scene processing during traversal testing, specifically to a virtual keyboard processing method, device, electronic device, storage medium, and product. Background Technology

[0002] During the application interface (UI) traversal test on a mobile terminal, operations related to the virtual keyboard control may occur. These operations actually cover roughly the same functions and may also cover other controls for a long time, affecting the traversal effect.

[0003] Forcibly removing the keyboard overlay will prevent some controls and events that can only be displayed when the keyboard is invoked from being tested. Summary of the Invention

[0004] This disclosure provides a virtual keyboard processing method, apparatus, electronic device, storage medium, and product.

[0005] According to a first aspect of this disclosure, a virtual keyboard processing method is provided, the method comprising:

[0006] During the traversal testing of the application interface control functions, the virtual keyboard in the application interface control is state-detected; in response to the detection that the virtual keyboard is in the on state, the number of consecutive times the virtual keyboard is in the keyboard scene is determined; and the virtual keyboard is processed according to the number of consecutive times.

[0007] According to a second aspect of this disclosure, a virtual keyboard processing apparatus is provided, the apparatus comprising:

[0008] The detection module is used to perform scene detection on the virtual keyboard in the application interface controls during the traversal test of the application interface control functions; the determination module is used to determine the number of consecutive times the virtual keyboard is continuously turned on in response to the detection that the virtual keyboard is in the on state; the processing module is used to process the virtual keyboard according to the number of consecutive times.

[0009] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0010] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of the first aspect.

[0011] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method according to the first aspect.

[0012] According to a fifth aspect of this disclosure, a computer product is provided, including a computer program that, when executed by a processor, implements the method according to the first aspect.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0015] Figure 1 This is a schematic diagram of the application environment according to an embodiment of the present disclosure;

[0016] Figure 2 A flowchart illustrating a virtual keyboard processing method provided in an embodiment of this disclosure is shown;

[0017] Figure 3 A schematic flowchart of a keyboard processing method based on click rate provided in an embodiment of this disclosure is shown;

[0018] Figure 4 A flowchart illustrating a method for adjusting the click rate of a keyboard control according to an embodiment of this disclosure is shown.

[0019] Figure 5 A flowchart illustrating a method for canceling a virtual keyboard based on a keyboard control, provided in an embodiment of this disclosure, is shown.

[0020] Figure 6 A flowchart illustrating a keyboard control acquisition method provided in an embodiment of this disclosure is shown.

[0021] Figure 7 This diagram illustrates an overall schematic of a method for canceling a virtual keyboard according to an embodiment of the present disclosure.

[0022] Figure 8 A schematic diagram of the structure of a virtual keyboard processing device provided in an embodiment of this disclosure is shown;

[0023] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0024] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] During the traversal test of the application interface on the mobile terminal, operations related to the virtual keyboard control may occur. These operations actually cover roughly the same functions and may also cover other controls for a long time, affecting the traversal effect.

[0026] In related technologies, there are two main approaches to handling virtual keyboard scenarios during mobile UI traversal testing. One approach is to forcibly eliminate the influence of the virtual keyboard. Specifically, this involves matching keywords (e.g., `XCUIElementTypeKeyboard`) of the virtual keyboard in the control tree to identify whether the traversal is in a keyboard scenario. When a keyboard scenario is identified, the virtual keyboard is removed by using a swipe-down operation in the wda interface. If hiding the virtual keyboard fails, the virtual keyboard controls are filtered to avoid clicking them, and input is completed directly through the wda input operation. The second approach is to reduce virtual keyboard operations. This involves matching keywords (e.g., `XCUIElementTypeKeyboard`) of the virtual keyboard controls in the control tree to identify keyboard controls, aggregating them into a single control for operation, reducing the probability of keyboard operations, retaining keyboard input functionality, and limiting the maximum keyboard usage time to avoid prolonged keyboard obscuring.

[0027] However, the above method has the following technical problems:

[0028] 1. Directly removing the virtual keyboard overlay will prevent some controls and events that only become visible when the virtual keyboard is activated from being tested. For example, controls that do not support input when the virtual keyboard is not activated may become accessible after the virtual keyboard is activated. Also, controls associated with input events, such as "Send" and "Publish," may not be visible or operable when the virtual keyboard is not activated, leading to missed tests of controls and events.

[0029] 2. Directly filtering out virtual keyboard controls will also filter out functional controls within the virtual keyboard that can trigger the events under test. For example, controls such as "Done," "Send," and "Confirm," which are usually located in the lower right corner of a custom keyboard and can trigger events under test, will also be filtered out, causing the virtual keyboard to fail to collapse and events to be missed. For instance, text content can be entered using the wda input method, but custom emoticon controls still require clicking the keyboard to complete the input; filtering all of them will prevent custom emoticon controls from being tested.

[0030] 3. The virtual keyboard controls are aggregated into a single control without considering the diversity of keyboard control types. For example, functional keyboard controls such as "Done," "Send," and "Confirm," which can trigger the event to be tested, located in the lower right corner of a custom keyboard, have different testing value than ordinary spelling keyboard controls such as "A," "1," and "Space." The approach of directly aggregating them into a single control has flaws.

[0031] 4. There is a possibility that eliminating virtual keyboard obscuring may fail. The wda's keyboard collapse or swipe-down methods may fail to collapse the virtual keyboard, leading to prolonged keyboard obscuring. Simply using a cold start to force exit the keyboard scene will impact traversal efficiency. For example, in some custom virtual keyboard usage scenarios, specific controls must be clicked to exit the keyboard scene.

[0032] 5. There is an issue with virtual keyboard controls failing to be recognized. Keyboard controls are diverse, including letters, emojis, and text that automatically appears on the keyboard based on the input content. Not all of these keyboard controls can be recognized using a few unified, specific keywords.

[0033] To address the aforementioned technical deficiencies, this disclosure provides a virtual keyboard processing method and apparatus, and further provides multiple keyboard processing methods. These multiple methods handle the virtual keyboard under different circumstances, thereby improving the flexibility of virtual keyboard processing. Determining the appropriate keyboard processing method based on the number of consecutive times the virtual keyboard appears in the keyboard scene ensures that keyboard controls dependent on the virtual keyboard and events associated with those controls can be tested, while also preventing prolonged keyboard obscuring from affecting traversal efficiency.

[0034] The keyboard processing method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 101 communicates with server 102 via a network. Terminal 101 can display the traversal test process of the application interface controls, and server 102 can operate the test process and process the virtual keyboard. Terminal 101 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, and server 102 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0035] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0036] The following embodiments will describe the keyboard processing method and apparatus provided in this disclosure in conjunction with the accompanying drawings.

[0037] Figure 2 A flowchart illustrating a virtual keyboard processing method provided in an embodiment of this disclosure is shown, as follows: Figure 2 As shown, the method may include:

[0038] In step S210, during the traversal test of the application interface control functions, the virtual keyboard in the application interface control is state-detected.

[0039] In this embodiment of the disclosure, during the traversal testing of the application interface control functions based on the terminal, the page source code (pagesource) of the page is obtained. The pagesource is parsed to obtain the control tree, which is a hierarchical structure composed of controls. By detecting the page source, the keyboard control keywords in the page are determined, thereby performing scene detection on the virtual keyboard.

[0040] For example, the keyword for the keyboard control could be "Keyboard". Detecting the virtual keyboard using this keyword can improve the accuracy of keyboard detection.

[0041] In this disclosure, the state detection of the virtual keyboard in the application interface control can be understood as the scene detection of the virtual keyboard, that is, to confirm whether the virtual keyboard is displayed in the current application interface.

[0042] It should be noted that the application scenario of the virtual keyboard in this disclosure is the virtual keyboard being turned on.

[0043] In step S220, in response to detecting that the virtual keyboard is in the on state, the number of consecutive times the virtual keyboard is in the on state is determined.

[0044] In this embodiment of the disclosure, the automated UI traversal test consists of a series of consecutive test operations (such as clicking controls, swiping pages, etc.). Before each operation, it is checked whether the current page belongs to a keyboard scenario (i.e., the active state). For example, if the keyboard is detected in the first operation, the consecutive count is 1.

[0045] The keyboard was detected on the second attempt, making a total of 2 consecutive attempts.

[0046] The keyboard was detected on the 3rd attempt, making a total of 3 consecutive attempts.

[0047] The fourth operation did not detect the keyboard, so the sequence was no longer continuous, and the number of consecutive attempts was 0.

[0048] The keyboard was not detected on the 5th attempt, and the consecutive count was 0.

[0049] The keyboard was detected on the 6th attempt, with a consecutive count of 1.

[0050] The keyboard was detected on the 7th attempt, making the consecutive count 2.

[0051] The keyboard was not detected on the 8th attempt, so the sequence was no longer continuous, and the number of consecutive attempts was 0.

[0052] If the virtual keyboard is detected to be in a keyboard scene, record that the virtual keyboard is in a keyboard scene and determine the number of consecutive times it is in a keyboard scene.

[0053] In step S230, the virtual keyboard is processed according to the number of consecutive times.

[0054] In this embodiment of the disclosure, the processing method for the virtual keyboard is determined based on the number of consecutive times the virtual keyboard is continuously in the keyboard scene, and the virtual keyboard is processed according to the determined processing method.

[0055] The keyboard processing method proposed in this disclosure determines the method for processing the virtual keyboard by determining the number of consecutive times the virtual keyboard is continuously in the keyboard scene. This ensures that keyboard controls that depend on the virtual keyboard and events associated with the keyboard controls can be tested, while also avoiding the impact of prolonged keyboard obscuring on traversal efficiency.

[0056] In this embodiment, multiple different thresholds can be preset for the number of consecutive times the virtual keyboard is continuously in the keyboard scene. The number of consecutive times the virtual keyboard is continuously in the keyboard scene is then compared with the preset thresholds to obtain the relationship between the number of consecutive times and the thresholds. Based on this determined relationship, a corresponding virtual keyboard processing method is determined, and the virtual keyboard is processed according to the determined virtual keyboard processing method.

[0057] In this disclosure, the following embodiments will describe an implementation method for processing a virtual keyboard based on a virtual keyboard processing method corresponding to size relationships.

[0058] Figure 3 A schematic flowchart of a keyboard processing method based on click rate provided in an embodiment of this disclosure is shown, as follows: Figure 3 As shown, the method may include:

[0059] In step S310, in response to determining that the number of consecutive clicks is less than or equal to a first threshold, the keyboard controls associated with the virtual keyboard are acquired, and the click rate of the keyboard controls is adjusted based on their weights.

[0060] In step S320, the virtual keyboard is processed by operating the keyboard control based on the click rate.

[0061] In this embodiment of the disclosure, multiple thresholds are preset. After determining the number of consecutive times the virtual keyboard is continuously in the on state (i.e., keyboard scene), this number of consecutive times can be first compared with the minimum value among the multiple thresholds, and then compared with the thresholds one by one from smallest to largest. Wherein, the thresholds are integer values.

[0062] If it is determined that the number of consecutive times the virtual keyboard is continuously in the keyboard scene is less than or equal to a first threshold, the keyboard controls associated with the virtual keyboard are obtained, and the weights of the obtained keyboard controls are further determined. Based on the weights of the keyboard controls, the click rate of the keyboard controls is adjusted. It should be noted that the first threshold in this disclosure can be the minimum value among preset thresholds.

[0063] In this disclosure, keyboard controls are sorted according to their click rate, and those with higher click rates are prioritized for operation, thus processing the virtual keyboard. This avoids repetitive and meaningless input while ensuring that events dependent on keyboard controls can be tested.

[0064] In this disclosure, the implementation method for adjusting the click rate of the keyboard control is as follows.

[0065] Figure 4 A flowchart illustrating a method for adjusting the click rate of a keyboard control according to an embodiment of this disclosure is shown, as follows: Figure 4 As shown, the method may include:

[0066] In step S410, the acquired keyboard controls are classified based on preset keyboard control types.

[0067] In this embodiment of the disclosure, the keyboard control type can be predetermined. For example, the keyboard control type can be a functional type, a spelling type, an emoji type, or other types of keyboard controls.

[0068] Among them, functional keyboard controls can be "Send", "Publish", "OK", etc.; spelling keyboard controls can be letters, operators, numbers, etc., such as "AZ", "space", "+", "1", etc.

[0069] In this disclosure, the keyboard control type can be determined based on events associated with the keyboard control.

[0070] The obtained keyboard controls are further categorized according to the preset keyboard control types.

[0071] In step S420, a preset weight is determined for each keyboard control type.

[0072] In this embodiment of the disclosure, a corresponding weight can be determined for each keyboard control type. For example, functional types correspond to a first weight, spelling types correspond to a second weight, and emoji types correspond to a third weight, etc.

[0073] In step S430, the click-through rate of the categorized keyboard controls is adjusted based on their weights.

[0074] In this embodiment of the disclosure, after determining the corresponding weight for each keyboard control type, the weight corresponding to the categorized keyboard controls is determined based on the weight determined for each keyboard control type, thereby adjusting the click-through rate of the categorized keyboard controls based on the weight of each keyboard control.

[0075] For example, a weight of zero can be set for a keyboard control of type spelling. This can filter out keyboard controls of type spelling when they are clicked, avoiding repetitive and meaningless input, and also reducing the risk of test accounts being banned.

[0076] When there are many emoji keyboard controls and repeated testing is not very valuable, emoji controls can be aggregated into one control and share a single weight.

[0077] Set a relatively high weight for functional keyboard controls so that events that depend on keyboard controls can be tested.

[0078] In this embodiment of the disclosure, the keyboard control corresponding to the highest click rate can also be manipulated to trigger an event associated with the keyboard control. The triggered event is then executed, and the virtual keyboard is preserved.

[0079] Furthermore, by iterating through the UI based on the operation weights of the controls, the control with the highest weight is selected for operation. The operation weight of a control decreases as the number of operations increases. Sharing a single operation weight can reduce the click probability of emoji keyboard controls.

[0080] This disclosure enables the triggering of more types of events and the testing of more types of controls by taking different measures for different keyboard controls.

[0081] The following embodiments of this disclosure will illustrate the method for canceling the keyboard.

[0082] In some embodiments of this disclosure, in response to determining that the number of consecutive times the virtual keyboard is in the keyboard scene is greater than a first threshold and less than or equal to a second threshold, the virtual keyboard is canceled based on a gesture operation.

[0083] In other embodiments of this disclosure, in response to determining that the number of consecutive times the virtual keyboard is in the keyboard scene is greater than a second threshold and less than or equal to a third threshold, the keyboard control of the virtual keyboard is obtained, and the keyboard control is operated to cancel the virtual keyboard.

[0084] In some embodiments of this disclosure, in response to determining that the number of consecutive times the virtual keyboard is continuously in the keyboard scene is greater than a third threshold, the virtual keyboard is canceled based on a cold start operation.

[0085] In this embodiment of the disclosure, the virtual keyboard can be canceled based on the keyboard control.

[0086] Figure 5 The diagram illustrates a flowchart of a method for canceling a virtual keyboard based on a keyboard control, as provided in an embodiment of this disclosure. Figure 5 As shown, the method may include:

[0087] In step S510, a keyboard control of a function type is obtained and operated.

[0088] In step S520, the virtual keyboard is canceled in response to triggering an event associated with the keyboard control of the function type.

[0089] In this embodiment of the disclosure, the virtual keyboard can be canceled using a function-type keyboard control. For example, when a user enters content in the comment section of an application, the send control in the keyboard control is triggered to send the entered content, thereby causing the virtual keyboard to collapse, thus canceling the virtual keyboard in a keyboard scenario.

[0090] There are several different ways to cancel the virtual keyboard, which can reduce the keyboard cover time, increase the success rate of retracting the keyboard, and effectively control the keyboard cover time.

[0091] In this embodiment of the disclosure, the keyboard control can be obtained in different ways, as follows.

[0092] Figure 6 The diagram illustrates a flowchart of a keyboard control acquisition method provided in an embodiment of this disclosure, as shown below. Figure 6 As shown, the method may include:

[0093] In step S610, based on a preset keyboard control keyword, a keyboard control corresponding to the keyboard control keyword is obtained.

[0094] In step S620, the coordinate range of the keyboard control is determined, and other keyboard controls within the coordinate range are obtained.

[0095] In an embodiment of the present disclosure, based on a preset keyboard control keyword, in the parsed control tree, a keyboard control corresponding to the keyboard control keyword can be obtained.

[0096] Furthermore, based on the determined keyboard control, the coordinate range of the keyboard control can be further determined, and other keyboard controls within the coordinate range are obtained.

[0097] Exemplarily, the keyboard control keyword can be a predefined naming rule, whether the xpath contains the XCUIElementTypeKeyboard control type, etc. The coordinate range of the keyboard control can be identified based on the coordinate range of the XCUIElementTypeKeyboard class control. For example, based on the coordinate range of the XCUIElementTypeKeyboard control, the text that appears automatically associated with the content input by the keyboard within 50 pixels above the keyboard can also be regarded as a keyboard control. For example, when the user inputs the text "我", words related to "我" will appear above the keyboard, such as "的", "们", etc. This part of the text automatically associated with "我" for input is a keyboard control, that is, words such as "的", "们" also belong to the keyboard control, and different operation weights are configured based on the keyboard control category to which the control belongs later.

[0098] By combining the keyboard coordinate range and the keyboard control keyword in the present disclosure, the accuracy rate of identifying keyboard controls is improved.

[0099] In an embodiment of the present disclosure, after processing the virtual keyboard, the virtual keyboard is re-detected for the scene to determine whether the virtual keyboard has been successfully cancelled, or further determine the consecutive number of times the virtual keyboard continuously remains in the keyboard scene.

[0100] Figure 7 The overall schematic diagram of a method for cancelling a virtual keyboard provided by an embodiment of the present disclosure is shown, as Figure 7 shown in, includes the following steps:

[0101] Step 1: Obtain the pagesource of the current page based on wda.

[0102] Step 2: Determine if the pagesource contains the keyword "Keyboard" to determine if it is in a keyboard scene. If it is, proceed to Step 3 and record the number of consecutive keyboard overlays. Otherwise, proceed to Step 13, perform no keyboard-related processing, and execute normal traversal.

[0103] Step 3: Record the number of consecutive times the keyboard scene is continuously displayed, and then take different keyboard processing methods based on the number of consecutive times.

[0104] Step 4: Determine if the number of consecutive times the keyboard scene is continuously in the loop exceeds 'a' times (the value of 'a' can be adjusted according to the actual traversal effect, aiming to trigger more events and test more controls). If it exceeds 'a' times, proceed to Step 5, which handles the keyboard. Otherwise, proceed to Step 11, which does not handle the keyboard and directly traverses the keyboard scene.

[0105] Step 5: Determine if the number of consecutive keyboard scenes exceeds a+1. If it exceeds a+1, proceed to Step 6 for the next step. Otherwise, proceed to Step 9 to try a simpler keyboard removal method.

[0106] Of course, in this embodiment, the threshold can also be determined as a+2, but no specific limitation is made here.

[0107] Step 6: Determine if the number of consecutive times the user remains in the keyboard scene exceeds a+2. If it exceeds a+2, it means the user has been in the keyboard scene for too long; proceed to Step 7 to force exit the keyboard scene. Otherwise, proceed to Step 8 to attempt a more complex keyboard removal method.

[0108] Step 7: Cold start, forcibly end the traversal in the keyboard scene to avoid staying in the keyboard scene for too long, and proceed to step 10.

[0109] Step 8: Match control properties based on keywords or perform image recognition based on screenshots to identify controls on the page that can be used to eliminate the keyboard (including closing the keyboard, triggering keyboard-related events, etc.) (such as "<", "Done", "Send", "Publish", "Search", etc.). Operate the control and proceed to Step 10.

[0110] Step 9: Use wda methods (such as swiping down or collapsing the keyboard) to remove the keyboard.

[0111] Step 10: Obtain pagesource again and determine whether the keyboard was successfully cleared based on the keyboard keyword Keyboard.

[0112] Step 11: If a keyboard is detected, obtain the coordinate range of the keyboard. Based on the keyboard control keyword XCUIElementTypeKeyboard and the keyboard coordinate range, identify the keyboard control, controls covered by the keyboard, and text that is automatically associated with the keyboard input content and displayed...

[0113] Step 12: Classify keyboard controls based on the types of events associated with them, such as spelling controls (e.g., "A", "space", "+", "1", automatically associated text, etc.), emoticons, function controls (e.g., "send", "post", "confirm", etc.), keyboard overlay controls, etc., to configure different operation weights.

[0114] Step 13: Configure different operation weights based on control properties, type, number of operations, etc.

[0115] Step 14: Obtain the control with the highest weight and operate on it.

[0116] Step 15: Record the number of times the control is operated.

[0117] The keyboard handling method disclosed herein takes into account the diversity of keyboard control types, classifies keyboard controls according to the types of events associated with them, and adopts different handling methods for different categories of keyboard controls, configuring different operation weights according to test value. Furthermore, this keyboard handling method is applicable to system keyboards and various custom keyboards, making it more general and applicable, and can greatly improve the success rate of keyboard removal and keyboard control recognition.

[0118] Based on and Figure 2 The method shown follows the same principle. Figure 8 A schematic diagram of the structure of a virtual keyboard processing device provided in an embodiment of this disclosure is shown, such as... Figure 8 As shown, the virtual keyboard processing device 800 may include:

[0119] The detection module 801 is used to detect the status of the virtual keyboard in the application interface control during the traversal test of the application interface control function; the determination module 802 is used to determine the number of consecutive times the virtual keyboard is continuously in the on state in response to the detection that the virtual keyboard is in the on state; the processing module 803 is used to process the virtual keyboard according to the number of consecutive times.

[0120] In this embodiment of the disclosure, the processing module 803 is configured to, in response to determining that the consecutive number of times is less than or equal to a first threshold, acquire a keyboard control associated with the virtual keyboard, and adjust the click rate of the keyboard control based on the weight of the keyboard control; and operate the keyboard control based on the click rate to process the virtual keyboard.

[0121] In this embodiment of the disclosure, the processing module 803 is further configured to classify the acquired keyboard controls based on preset keyboard control types; determine a preset weight for each keyboard control type; and adjust the click rate of the classified keyboard controls based on the weights.

[0122] In this embodiment of the disclosure, the processing module 803 is used to operate on the keyboard control corresponding to the maximum click rate, trigger an event associated with the keyboard control, execute the event, and retain the virtual keyboard.

[0123] In this embodiment of the disclosure, the processing module 803 is configured to cancel the virtual keyboard based on a gesture operation in response to determining that the number of consecutive times is greater than a first threshold and less than or equal to a second threshold.

[0124] or

[0125] In response to determining that the consecutive number of times is greater than a second threshold and less than or equal to a third threshold, the keyboard control of the virtual keyboard is obtained, and the keyboard control is operated to cancel the virtual keyboard;

[0126] or

[0127] In response to determining that the number of consecutive occurrences exceeds a third threshold, the virtual keyboard is canceled based on a cold start operation.

[0128] In this embodiment of the disclosure, the processing module 803 is configured to acquire a keyboard control of a function type and operate the keyboard control of the function type; and cancel the virtual keyboard in response to triggering an event associated with the keyboard control of the function type.

[0129] In this embodiment of the present disclosure, the device further includes: an acquisition module 804;

[0130] The acquisition module 804 is used to acquire keyboard controls corresponding to preset keyboard control keywords; determine the coordinate range of the keyboard controls; and acquire other keyboard controls within the coordinate range.

[0131] In this embodiment of the disclosure, the detection module 801 is further configured to re-detect the virtual keyboard scene.

[0132] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0133] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0134] Figure 9A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0135] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0136] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0137] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as keyboard processing methods. For example, in some embodiments, the keyboard processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the keyboard processing method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform keyboard processing methods by any other suitable means (e.g., by means of firmware).

[0138] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0139] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0140] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0141] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0142] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0143] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0144] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0145] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A virtual keyboard processing method, the method comprising: During the traversal testing of the application interface control functions, the virtual keyboard in the application interface control is subjected to state detection. In response to detecting that the virtual keyboard is in an on state, determine the number of consecutive times the virtual keyboard is in an on state; The virtual keyboard is processed based on the number of consecutive occurrences; The process of processing the virtual keyboard based on the number of consecutive occurrences specifically includes: In response to determining that the number of consecutive clicks is less than or equal to a first threshold, the keyboard controls associated with the virtual keyboard are obtained, and the click rate of the keyboard controls is adjusted based on the weight of the keyboard controls. The keyboard controls are then operated based on the click rate, and the keyboard controls are sorted, with priority given to operating the keyboard controls with high click rates, so as to process the virtual keyboard. In response to determining that the number of consecutive times the virtual keyboard is continuously in the keyboard scene is greater than a first threshold and less than or equal to a second threshold, the virtual keyboard is canceled based on a gesture operation; In response to determining that the consecutive number of times is greater than a second threshold and less than or equal to a third threshold, the keyboard control of the virtual keyboard is obtained, and the keyboard control is operated to cancel the virtual keyboard; In response to determining that the number of consecutive occurrences exceeds a third threshold, the virtual keyboard is canceled based on a cold start operation.

2. The method according to claim 1, wherein, Adjusting the click rate of the keyboard control based on its weight includes: The acquired keyboard controls are categorized based on preset keyboard control types; Determine the preset weight for each of the keyboard control types; The click-through rate of the keyboard control after being categorized based on the weights is adjusted.

3. The method according to claim 1, wherein, The method includes: The keyboard control corresponding to the highest click rate is operated to trigger the event associated with the keyboard control; Execute the event to retain the virtual keyboard.

4. The method according to claim 1, wherein, The step of obtaining the keyboard control of the virtual keyboard and operating the keyboard control to cancel the virtual keyboard includes: Obtain the keyboard control of the specified function type, and manipulate the keyboard control of the specified function type; In response to triggering an event associated with the keyboard control of the function type, the virtual keyboard is canceled.

5. The method according to any one of claims 1-4, wherein, The method further includes: Based on preset keyboard control keywords, obtain the keyboard control corresponding to the keyboard control keywords; Determine the coordinate range of the keyboard control, and obtain other keyboard controls within the coordinate range.

6. The method according to claim 1, wherein, After processing the virtual keyboard according to the number of consecutive iterations, the method further includes: Re-detect the virtual keyboard in the scene.

7. A virtual keyboard processing device, the device comprising: The detection module is used to detect the status of the virtual keyboard in the application interface controls during the traversal test of the application interface control functions. A determining module is configured to determine the number of consecutive times the virtual keyboard is continuously turned on in response to detecting that the virtual keyboard is in an on state; The processing module is used to process the virtual keyboard according to the number of consecutive times. Specifically, the processing module is used to: in response to determining that the number of consecutive times is less than or equal to a first threshold, obtain the keyboard controls associated with the virtual keyboard, adjust the click rate of the keyboard controls based on the weight of the keyboard controls, operate the keyboard controls based on the click rate, sort the keyboard controls, and prioritize the operation of keyboard controls with high click rates, so as to realize the processing of the virtual keyboard. In response to determining that the number of consecutive times the virtual keyboard is continuously in the keyboard scene is greater than a first threshold and less than or equal to a second threshold, the virtual keyboard is canceled based on a gesture operation; In response to determining that the consecutive number of times is greater than a second threshold and less than or equal to a third threshold, the keyboard control of the virtual keyboard is obtained, and the keyboard control is operated to cancel the virtual keyboard; In response to determining that the number of consecutive occurrences exceeds a third threshold, the virtual keyboard is canceled based on a cold start operation.

8. The apparatus according to claim 7, wherein, The processing module is further configured to: The acquired keyboard controls are categorized based on preset keyboard control types; Determine the preset weight for each of the keyboard control types; The click-through rate of the keyboard control after being categorized based on the weights is adjusted.

9. The apparatus according to claim 8, wherein, The processing module is used for: The keyboard control corresponding to the highest click rate is operated to trigger the event associated with the keyboard control; Execute the event to retain the virtual keyboard.

10. The apparatus according to claim 7, wherein, The processing module is used for: Obtain the keyboard control of the specified function type, and manipulate the keyboard control of the specified function type; In response to triggering an event associated with the keyboard control of the function type, the virtual keyboard is canceled.

11. The apparatus according to any one of claims 7-10, wherein, The device further includes: an acquisition module; The acquisition module is used to acquire the keyboard control corresponding to the preset keyboard control keyword; Determine the coordinate range of the keyboard control, and obtain other keyboard controls within the coordinate range.

12. The apparatus according to claim 7, wherein, The detection module is also used for: Re-detect the virtual keyboard in the scene.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images