An automated recognition method, device, medium and electronic device for a secure keyboard

Abstract

An automated recognition method, device, medium and electronic device for a secure keyboard are disclosed in an embodiment of the present application. This embodiment relates to the field of automation technology. Among them, the method includes: if a secure keyboard pop-up event is detected, determine the elements within the intercepted range according to the UI hierarchy tree of the secure keyboard; establish a position comparison and calibration table for each element based on the reference positions within the intercepted range; cut the intercepted range according to the position comparison and calibration table to obtain character key pictures, and generate names for each character key picture; if an automated recognition event of the secure keyboard is detected, determine the input characters of the target input password, and determine the target key positions corresponding to the input characters according to the correspondence between the input characters and the names of the character key pictures, and simulate to complete the automated test. By adopting the technical solution of the present application, the accuracy of image recognition is improved to ensure that the automated test process can proceed smoothly, and the problem that the automated test cannot be carried out due to different layouts of the secure keyboard is solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of automation technology, and in particular, to an automated recognition method, device, medium, and electronic device for a secure keyboard. Background Art

[0002] With the development of enterprise digital informatization, automation technology is increasingly widely used in various industries. Some current automation products (such as RPA and automated testing) play a crucial role in reducing costs and increasing efficiency for enterprises. At the same time, enterprises are also paying more and more attention to information security. In particular, the financial industry is increasing its investment in information security, posing new challenges and requirements for automation technology.

[0003] Currently, traditional automated testing technologies generally perform automated operations by identifying element controls, but some automated requirements cannot be met. For example, secure password controls for enterprise online banking, secure password controls for mobile banking, etc., often have problems such as inability to identify controls. Some automation products achieve automated operations by simulating positions. For example, the positions of keys in a virtual keyboard are simulated based on the positions of keys in a physical keyboard.

[0004] This solution can only solve some scenarios where the positions of the keys on a secure keyboard remain unchanged, and cannot handle the situation where the positions of the keys are disrupted each time a secure keyboard is opened. Summary of the Invention

[0005] The embodiments of the present application provide an automated recognition method, device, medium, and electronic device for a secure keyboard, which can solve the problem that automated testing cannot be performed due to different layouts each time a secure keyboard is opened through secure keyboard cutting and image recognition technologies, thereby achieving the purpose of performing automated testing operations on the secure keyboard.

[0006] In a first aspect, the embodiments of the present application provide an automated recognition method for a secure keyboard, the method comprising:

[0007] If a secure keyboard pop-up event is detected, then in response to a secure keyboard capture instruction, determine the elements within the capture range according to the UI hierarchy tree of the secure keyboard;

[0008] Establish a position comparison and calibration table for each element according to the reference positions within the capture range;

[0009] In response to the confirmation operation of the position comparison and calibration table, cut the capture range of the secure keyboard according to the position comparison and calibration table to obtain character key pictures, and generate names for each character key picture;

[0010] If an automated recognition event of the secure keyboard is detected, then determine the input characters of the target input password;

[0011] According to the correspondence between the input characters and the names of the character button pictures, determine the positions of the target buttons corresponding to the input characters, and perform simulated clicks to complete the automated test.

[0012] In a second aspect, an embodiment of the present application provides an automated recognition device for a secure keyboard, the device comprising:

[0013] An element determination module, configured to, if a secure keyboard pop-up event is detected, in response to a secure keyboard capture instruction, determine elements within the capture range according to the UI hierarchy tree of the secure keyboard;

[0014] A calibration table establishment module, configured to establish a position comparison and calibration table for each element according to the reference positions within the capture range;

[0015] A secure keyboard processing module, configured to, in response to a confirmation operation of the position comparison and calibration table, cut the capture range of the secure keyboard according to the position comparison and calibration table to obtain character button pictures, and generate names for each character button picture;

[0016] An input character determination module, configured to determine the input characters of the target input password if an automated recognition event of the secure keyboard is detected;

[0017] An automated test module, configured to determine the positions of the target buttons corresponding to the input characters according to the correspondence between the input characters and the names of the character button pictures, and perform simulated clicks to complete the automated test.

[0018] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the automated recognition method for a secure keyboard as described in the embodiments of the present application.

[0019] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, it implements the automated recognition method for a secure keyboard as described in the embodiments of the present application.

[0020] In the technical solution provided by the embodiment of the present application, when a secure keyboard pop-up event is detected, elements within the capture range are determined according to the UI hierarchy tree of the secure keyboard; then, based on the reference positions within the capture range, a position comparison and calibration table for each element is established; furthermore, the capture range of the secure keyboard is cut according to the position comparison and calibration table to obtain character key pictures, and names for each character key picture are generated; when an automated recognition event of the secure keyboard is detected, the input characters of the target input password are determined, and according to the correspondence between the input characters and the names of each character key picture, the positions of the target keys corresponding to the input characters are determined, and simulated clicks are performed to complete the automated test. By adopting the technical solution of the present application, the key buttons of the password secure keyboard are cut based on the UI element hierarchy tree, and a position comparison and calibration table is provided for manual correction, thereby improving the accuracy of key button cutting and image recognition to ensure that the automated test process can proceed smoothly. By naming each character key picture obtained by cutting, it is ensured that during automated execution, the target character pictures can be quickly searched and compared with the secure keyboard, effectively solving the problem that automated testing cannot be performed due to different keyboard layouts each time the secure keyboard is opened. Description of the Drawings

[0021] Figure 1 is a flowchart of an automated recognition method for a secure keyboard provided by Embodiment 1 of the present application;

[0022] Figure 2 is an effect diagram of letter key recognition and positioning provided by Embodiment 1 of the present application;

[0023] Figure 3 is a flowchart of an automated recognition method for a secure keyboard provided by Embodiment 2 of the present application;

[0024] Figure 4 is a flowchart of an automated recognition method for a secure keyboard provided by Embodiment 3 of the present application;

[0025] Figure 5 is a structural block diagram of an automated recognition device for a secure keyboard provided by Embodiment 4 of the present application;

[0026] Figure 6 is a schematic structural diagram of an electronic device provided by Embodiment 6 of the present application. Detailed Embodiments

[0027] The present application will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, rather than limiting the present application. Additionally, it should be noted that for the sake of description, only parts related to the present application are shown in the drawings, rather than all the structures.

[0028] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts depict the steps as sequential processes, many of the steps can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operations are completed, but it can also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0029] Embodiment 1

[0030] Figure 1 It is a flowchart of an automatic recognition method for a security keyboard provided in Embodiment 1 of the present application. This embodiment is applicable to the scenario where the automatic test cannot be performed due to inconsistent keyboard layouts each time the security keyboard is opened. This method can be executed by the automatic recognition device of the security keyboard provided in the embodiments of the present application. The device can be implemented in software and / or hardware and can be integrated into an electronic device.

[0031] As Figure 1 shown, the automatic recognition method for the security keyboard includes:

[0032] S110, if a security keyboard pop-up event is detected, then in response to the security keyboard capture instruction, determine the elements within the capture range according to the UI hierarchy tree of the security keyboard.

[0033] In this embodiment, an intelligent device connected to the device under test can be used as the execution entity for automatic recognition, and the intelligent device performs the automatic recognition process of the security keyboard on the device under test. Among them, the connection relationship between the device under test and the intelligent device can be a wired connection or a wireless connection. This solution can be in the process of system testing such as in a bank, where passwords or verification information are input through a security keyboard. Since the keyboard layout of the security keyboard may be inconsistent each time it is opened, it may cause the automatic test to not proceed smoothly. Therefore, the purpose of this solution is to still achieve automatic recognition of the security keyboard when the positions of the keys in the security keyboard change.

[0034] In this embodiment, the security keyboard pop-up event can refer to the operation event of popping up the security keyboard when the device under test receives a security keyboard pop-up instruction sent by a tester through an intelligent device. Among them, the security keyboard can be a virtual keyboard with the same key composition as a physical keyboard. Specifically, the key layout in the security keyboard can be the same as that of the physical keyboard or different from that of the physical keyboard. Exemplarily, the key layout in the security keyboard can be fixed and consistent with the physical keyboard each time the security keyboard pops up, or it can change randomly each time the security keyboard pops up, forming a key layout different from that of the physical keyboard.

[0035] In this embodiment, the security keyboard capture instruction may be an operation instruction issued by a tester for capturing the security keyboard interface popped up by the device under test. Specifically, the security keyboard capture instruction may be used to indicate a complete capture of the security keyboard interface or a partial capture of the security keyboard interface. The UI hierarchical tree may refer to the XML document of the tree-like hierarchical structure of the current interface elements of the device under test, which can be directly obtained through the system of the intelligent device. Among them, the XML document of the tree-like hierarchical structure marks the hierarchical structure of the elements in the interface and the area occupied by each element. The element may refer to each key in the security keyboard. For example, the element may be a button, text, input box, etc.

[0036] When the device under test receives the security keyboard pop-up instruction from the intelligent device, the security keyboard will pop up. At this time, when the intelligent device detects the security keyboard pop-up event sent by the device under test, it will respond to the security keyboard capture instruction sent by the tester. By capturing the security keyboard interface popped up by the device under test, a security keyboard capture range containing many elements can be obtained. Among them, the capture range can be represented in the form of a picture.

[0037] The UI hierarchical tree of the security keyboard can be directly obtained through the system of the intelligent device. Furthermore, based on the obtained UI hierarchical tree, each element in the security keyboard capture range can be determined. Specifically, if the capture range is the complete security keyboard, that is, it contains all the elements in the security keyboard, each element can be directly determined according to the corresponding UI hierarchical tree; if the capture range is only a part of the security keyboard, that is, it only contains a part of the elements in the security keyboard, then according to the mapping relationship between the security keyboard and the capture range, the hierarchical structure and element area corresponding to the capture range can be searched from the UI hierarchical tree, so as to determine each element in the capture range.

[0038] S120. Establish a position comparison and calibration table for each element according to the reference positions within the capture range.

[0039] Among them, the reference position may be the currently selected position of the mouse determined by the tester by clicking the mouse, or the system default reference position. The position comparison and calibration table may refer to selecting a reference position point in the security keyboard, and the coordinate table formed by other keys relative to the reference position point, which can be used to represent the position information of each key in the security keyboard based on the reference position point. Specifically, the position comparison and calibration table may include the key name and its corresponding relative coordinate value. Among them, the coordinate value may be the pixel distance relative to the reference position point.

[0040] In this embodiment, according to the reference position determined in the intercepted range of the security keyboard, the position coordinates of each element in the intercepted range relative to the reference position can be obtained, so as to establish a position comparison and calibration table for each element. Figure 2 is the effect diagram of letter key recognition and positioning provided by the first embodiment of this application. As Figure 2 shown, it is the result of recognizing and positioning the letter keys in the security keyboard based on the UI hierarchy tree. Exemplarily, the letter k key can be selected as the reference position, and then the position coordinates of other letter keys in the intercepted range of the security keyboard relative to the letter k key can be obtained, so as to establish a position comparison and calibration table for each element with the letter k key as the reference position.

[0041] S130. In response to the confirmation operation of the position comparison and calibration table, cut the intercepted range of the security keyboard according to the position comparison and calibration table to obtain character key pictures, and generate the names of each character key picture.

[0042] Among them, the confirmation operation can be an operation instruction issued by a tester on the electronic device used for testing to confirm the position comparison and calibration table. The character key picture can be a square picture intercepted with the position in the position comparison and calibration table as the center position and a set side length. In this solution, the character key pictures can be intercepted one by one according to the number of keys in the picture, the center position of the keys, and the preset side length. Exemplarily, for the center position of each key, the four corner positions of the key can be determined according to the preset side length. By connecting the four corner points in sequence with straight lines, several square areas containing only one key element can be divided. Cutting the intercepted range of the security keyboard according to the divided square areas can correspondingly obtain several character key pictures.

[0043] In this embodiment, the name of the character key picture can be the picture name obtained by the tester naming each cut character key picture in sequence. Exemplarily, the key name can be used as the name of the character key picture, so that the content of the key in the picture can be directly known according to the picture name.

[0044] By performing the operations in S110 - S130 above, the automated operation steps of the security keyboard can be generated. That is to say, the tester has completed the input process of the automated recognition of the security keyboard.

[0045] S140. If an automated recognition event of the security keyboard is detected, determine the input characters of the target input password.

[0046] Among them, the automated recognition event can be an instruction sent by a tester through the device under test for automatically recognizing a secure keyboard according to the automated operation steps that have been entered. The target input password can refer to the test password entered by the tester in the secure keyboard of the device under test. The input characters can refer to all the characters included in the target input password. When the device under test detects the automated recognition event of the secure keyboard, it will automatically trigger the appearance of the secure keyboard and perform character separation on the target input password filled in by the tester, so as to determine the input characters of the target input password.

[0047] S150. According to the correspondence between the input characters and the names of the character key pictures, determine the positions of the target keys corresponding to the input characters, and perform simulated clicks to complete the automated test.

[0048] Among them, the correspondence can mean that the input characters are the same as the names of the character key pictures. Specifically, the correspondence can include that the input characters and the names of the character key pictures are exactly the same letters or numbers. Exemplarily, when the input character is the letter a and the name of the character key picture is also the letter a, it can be considered that there is a correspondence between the two. The target key can refer to the character key picture corresponding to the input character. The position of the target key can refer to the position of the target key in the secure keyboard. Specifically, image recognition technology can be used to locate the target key. The simulated click can refer to the virtual operation of the device under test to click on the position of the target key corresponding to the input character. The automated test can refer to the automatic test process implemented according to the entered automated operation steps.

[0049] The technical solution provided by the embodiments of the present application cuts the keys of the password secure keyboard based on the UI element hierarchy tree and provides a position comparison calibration table for manual correction, thereby improving the accuracy of key cutting and image recognition to ensure that the automated test process can proceed smoothly. By naming each character key picture obtained by cutting, it is ensured that during automated execution, the target character picture can be quickly found and compared with the secure keyboard, thus effectively solving the problem that the automated test cannot be performed due to inconsistent keyboard layouts each time the secure keyboard is opened.

[0050] In this embodiment, optionally, after establishing the position comparison calibration table for each element according to the reference positions within the intercepted range and before responding to the confirmation operation of the position comparison calibration table, the method may include:

[0051] Display the position comparison calibration table for each element;

[0052] Respond to the calibration operation of the position comparison calibration table for each element to obtain the calibration result of the position comparison calibration table for each element.

[0053] Among them, the calibration operation can be an operation instruction issued by a tester through an intelligent device for performing position correction on each element in the position comparison calibration table. Through the calibration operation, the coordinate positions of each element can be moved to the center of the corresponding key. Specifically, the calibration operation can include operations such as calibrating and confirming the position of keyboard characters and modifying positions. The calibration result can refer to the coordinates of each element in the new position comparison calibration table obtained by sequentially calibrating the coordinate positions of each element in the position comparison calibration table.

[0054] It can be understood that when performing the calibration operation, only the elements whose coordinate positions are not at the center of the key need to be position-calibrated, while the elements whose coordinate positions are already at the center of the key do not need to be calibrated. Exemplarily, when performing the calibration operation on the letter 'a' key, first find the letter 'a' in the key name column of the position comparison calibration table. By clicking on the letter 'a' with the mouse, the relative coordinate position of the letter 'a' can be displayed within the intercepted range of the secure keyboard. If the relative coordinate position of the letter 'a' is at the center position of the letter 'a' key within the intercepted range, then there is no need to perform position correction on the letter 'a'; if the relative coordinate position of the letter 'a' is not at the center position of the letter 'a' key within the intercepted range, then position correction needs to be performed on the letter 'a' to move its relative coordinate position to the center position of the letter 'a' key within the intercepted range.

[0055] Through such a setting in this solution, the position calibration of each element in the position comparison calibration table can be achieved, thereby ensuring the accuracy of the positions of each character in the position comparison calibration table.

[0056] In this embodiment, optionally, in response to the calibration operation of the position comparison calibration table of each element, obtaining the calibration result of the position comparison calibration table of each element may include the following operations: in response to the calibration operation of the relative coordinates of the current element, determining the calibration result of the current element; differentially displaying the calibration result position of the current element within the intercepted range of the secure keyboard; if the calibration operation of the relative coordinates of each element is completed, then obtaining the calibration result of the position comparison calibration table of each element.

[0057] Among them, differential display can be used to characterize the difference in the deviation direction and deviation degree between the relative coordinate positions of each element displayed within the intercepted range of the secure keyboard and the center position of its corresponding key. Exemplarily, the relative coordinate position of the letter 'b' in the position comparison calibration table is in the upper left of the center position of the letter 'b' key within the intercepted range and is 2 pixel units away from the center position; while the relative coordinate position of the letter 'd' is in the lower right of the center position of the letter 'd' key and is 1 pixel unit away from the center position.

[0058] By sequentially calibrating the relative coordinates of each element within the intercepted range of the secure keyboard in this solution, the recognition accuracy of the relative positions of each element is improved, thereby enabling a more accurate calibration result to be obtained.

[0059] Embodiment 2

[0060] Figure 3 is a flowchart of an automated recognition method for a secure keyboard provided in Embodiment 2 of the present application. This embodiment is optimized based on the above embodiment. Specifically, the optimization is as follows: Before responding to the confirmation operation of the position comparison calibration table, the method further includes: responding to the configuration operation of the similarity to determine the similarity range for keyboard recognition; correspondingly, according to the correspondence between the input character and the names of the pictures of each character key, determining the position of the target key corresponding to the input character, including: determining the position of the target key corresponding to the input character according to the correspondence between the image of the input character and the names of the pictures of each character key recognized based on the similarity range.

[0061] As Figure 3 shown, the method of this embodiment specifically includes the following steps:

[0062] S310, if a secure keyboard pop-up event is detected, then in response to the secure keyboard capture instruction, determine the elements within the capture range according to the UI hierarchy tree of the secure keyboard.

[0063] S320, establish a position comparison calibration table for each element according to the reference positions within the capture range.

[0064] Among them, the specific implementation processes of S310 and S320 can refer to the detailed descriptions in S110 and S120.

[0065] S330, respond to the configuration operation of the similarity to determine the similarity range for keyboard recognition.

[0066] Among them, the similarity can be used to represent the similarity degree between two pictures, and can be specifically reflected in the form of a numerical value. If the similarity is higher, it means that the two pictures are more similar; on the contrary, if the similarity is lower, it means that the two pictures are less similar, that is, the similarity degree between the two pictures is smaller. The similarity range can refer to the similarity interval set by the tester according to the requirements of the actual application for the similarity size, and no specific limitation is made here.

[0067] It can be understood that the similarity size is a key factor determining the recognition accuracy of pictures. Specifically, if the similarity is larger, the recognition accuracy of the pictures is higher, and at the same time, the increase in recognition accuracy causes a decrease in the recognition success rate; on the contrary, if the similarity is smaller, the recognition accuracy of the pictures is lower, and at the same time, the decrease in recognition accuracy will also cause a decrease in the recognition success rate. Therefore, setting a reasonable similarity range is very necessary for picture recognition.

[0068] S340, in response to the confirmation operation of the position comparison calibration table, cut the capture range of the security keyboard according to the position comparison calibration table to obtain character key pictures, and generate the names of each character key picture.

[0069] S350, if an automated recognition event of the security keyboard is detected, determine the input characters of the target input password.

[0070] Among them, the specific implementation processes of S340 and S350 can refer to the detailed descriptions in S130 and S140.

[0071] S360, according to the correspondence relationship obtained by recognizing the image of the input character and the names of each character key picture based on the similarity range, determine the position of the target key corresponding to the input character, and perform a simulated click to complete the automated test.

[0072] Among them, recognition based on the similarity range can refer to recognizing the image of the input character within a set similarity range to find the corresponding character key picture. It can be understood that if the set similarity range is too large, the recognition accuracy of the input character will be greatly reduced.

[0073] The technical solution provided by the embodiment of the present application recognizes the input character based on the set similarity range, and determines the position of the target key corresponding to the input character according to the correspondence relationship between the input character and the names of each character key picture, improving the recognition accuracy of the input character, and further improving the accuracy of determining the position of the target key of the input character.

[0074] Embodiment Three

[0075] This embodiment is a preferred embodiment provided on the basis of Embodiment One. Figure 4 It is a flowchart of an automated recognition method for a security keyboard provided by Embodiment Three of the present application. The method of this embodiment specifically includes the following steps:

[0076] S410, if a security keyboard pop-up event is detected, in response to the security keyboard capture instruction, determine the elements within the capture range according to the UI hierarchy tree of the security keyboard.

[0077] S420, establish a position comparison calibration table for each element according to the reference positions within the capture range.

[0078] Among them, the specific implementation processes of S410 and S420 can refer to the detailed descriptions in S110 and S120.

[0079] S430. In response to the confirmation operation of the position comparison calibration table, the interception range of the security keyboard is cut according to the position comparison calibration table to obtain character key pictures, and each character key picture is named according to the letter and / or number content of the character key picture and the function information of the function keys, so as to obtain the names of each character key picture.

[0080] Among them, the function keys can refer to the keys used to implement a certain function. Specifically, the function keys can include keys such as case conversion, alphanumeric conversion, completion, deletion, and return. When naming each character key picture, naming rules related to the letter and / or number content of the character key picture and the function information of the function keys can be set. For example, the letter keys or number keys can be named according to the key surface characters, and the special function keys can be named by annotation.

[0081] In this embodiment, optionally, the method may further include: if there is a keyboard switching key in the security keyboard, in response to the touch operation of the switching key, the keyboard is switched to obtain each character key picture and the name of each character key picture of the switched security keyboard.

[0082] Among them, the keyboard switching key can refer to the key used to switch between two characters. Specifically, the keyboard switching key can include keys such as case conversion, alphanumeric conversion, and Chinese-English conversion. The touch operation can be a control instruction issued by the tester to click the switching key. The switching process can be a character switching operation implemented according to the touch operation of the switching key. Exemplarily, the switching process can be switching Chinese to English, or switching letters to numbers, etc.

[0083] In this solution, by setting naming rules, each character key picture is named according to the letter and / or number content of the character key picture and the function information of the function keys, making the naming of each character key picture clearer and more intuitive, and facilitating quickly finding the position of each character key picture on the security keyboard.

[0084] S440. If an automated recognition event of the security keyboard is detected, determine the input characters of the target input password.

[0085] S450. According to the correspondence between the input characters and the names of each character key picture, determine the position of the target key corresponding to the input characters, and perform a simulated click to complete the automated test.

[0086] Among them, for the specific implementation processes of S440 and S450, reference can be made to the detailed descriptions in S140 and S150.

[0087] The technical solution provided by the embodiments of the present application names each character key image obtained by cutting according to the letter and / or number content of the character key image and the function information of the function key, ensuring that when automated execution is performed, the target character image can be quickly searched for comparison with the security keyboard, effectively solving the problem that the keyboard layout is different each time the security keyboard is opened.

[0088] Embodiment 4

[0089] Figure 5 FIG. 7 is a structural block diagram of an automatic recognition device for a security keyboard provided by Embodiment 4 of the present application. This device can execute the automatic recognition method for the security keyboard provided by any embodiment of the present application, and has corresponding functional modules and beneficial effects for executing the method. As Figure 5 shown, the device may include:

[0090] An element determination module 510, configured to, if a security keyboard pop-up event is detected, in response to a security keyboard capture instruction, determine elements within the capture range according to the UI hierarchy tree of the security keyboard;

[0091] A calibration table establishment module 520, configured to establish a position comparison calibration table for each element according to the reference positions within the capture range;

[0092] A security keyboard processing module 530, configured to, in response to a confirmation operation of the position comparison calibration table, cut the capture range of the security keyboard according to the position comparison calibration table to obtain character key images, and generate names for each character key image;

[0093] An input character determination module 540, configured to determine input characters of a target input password if an automatic recognition event of the security keyboard is detected;

[0094] An automatic test module 550, configured to determine the position of a target key corresponding to the input character according to the correspondence between the input character and the names of the character key images, and perform a simulated click to complete an automatic test.

[0095] Based on the above embodiments, optionally, the device further includes:

[0096] A similarity configuration module, configured to, in response to a configuration operation of similarity, determine a similarity range for keyboard recognition;

[0097] Correspondingly, the automatic test module 550 is specifically configured to:

[0098] Determine the position of a target key corresponding to the input character according to the correspondence between the image of the input character and the names of the character key images recognized based on the similarity range.

[0099] Based on the above embodiments, optionally, the secure keyboard processing module 530 is specifically configured to:

[0100] Name each character key image according to the letter and / or number content of the character key image and the function information of the function keys, so as to obtain the names of each character key image.

[0101] Based on the above embodiments, optionally, the device further includes:

[0102] A key switching module, configured to, if there is a keyboard switching key in the secure keyboard, in response to a touch operation on the switching key, perform a switching process on the keyboard, and obtain each character key image and the name of each character key image of the secure keyboard after the switching process.

[0103] Based on the above embodiments, optionally, the device further includes:

[0104] A calibration table display module, configured to display the position comparison calibration table of each element;

[0105] A calibration table calibration module, configured to, in response to a calibration operation on the position comparison calibration table of each element, obtain a calibration result of the position comparison calibration table of each element.

[0106] Based on the above embodiments, optionally, the calibration table calibration module is specifically configured to:

[0107] In response to a calibration operation on the relative coordinates of the current element, determine the calibration result of the current element;

[0108] Differentially display the calibration result position of the current element within the intercepted range of the secure keyboard;

[0109] If the calibration operation on the relative coordinates of each element is completed, obtain the calibration result of the position comparison calibration table of each element.

[0110] The above product can execute the automated recognition method of the secure keyboard provided by the embodiments of the present application, and has the corresponding functional modules and beneficial effects for executing the method.

[0111] Embodiment Five

[0112] Embodiment Five of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the automated recognition method of the secure keyboard provided by all inventive embodiments of the present application:

[0113] If a secure keyboard pop-up event is detected, then in response to a secure keyboard interception instruction, determine the elements within the intercepted range according to the UI hierarchy tree of the secure keyboard;

[0114] Establish a position comparison and calibration table for each element according to the reference positions within the intercepted range;

[0115] In response to the confirmation operation of the position comparison and calibration table, cut the intercepted range of the security keyboard according to the position comparison and calibration table to obtain character key pictures, and generate names for each character key picture;

[0116] If an automated recognition event of the security keyboard is detected, determine the input characters of the target input password;

[0117] According to the correspondence between the input characters and the names of the character key pictures, determine the positions of the target keys corresponding to the input characters, and perform simulated clicks to complete the automated test.

[0118] Any combination of one or more computer-readable media may be employed. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, the computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0119] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which computer-readable program code is carried. Such a propagated data signal may take many forms, including - but not limited to - an electromagnetic signal, an optical signal, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0120] The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including - but not limited to - wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0121] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., connected through the Internet using an Internet service provider).

[0122] Embodiment Six

[0123] Embodiment Six of this application provides an electronic device. Figure 6 It is a schematic structural diagram of an electronic device provided by Embodiment Six of this application. As Figure 6 shown, this embodiment provides an electronic device 600, which includes: one or more processors 620; a storage device 610 for storing one or more programs. When the one or more programs are executed by the one or more processors 620, the one or more processors 620 implement the automated recognition method of the secure keyboard provided by the embodiments of this application. The method includes:

[0124] If a secure keyboard pop-up event is detected, then in response to a secure keyboard capture instruction, determine the elements within the capture range according to the UI hierarchy tree of the secure keyboard;

[0125] Establish a position comparison and calibration table for each element according to the reference positions within the capture range;

[0126] In response to the confirmation operation of the position comparison and calibration table, cut the capture range of the secure keyboard according to the position comparison and calibration table to obtain character key pictures, and generate names for each character key picture;

[0127] If an automated recognition event of the secure keyboard is detected, determine the input characters of the target input password;

[0128] According to the correspondence between the input characters and the names of the respective character key pictures, determine the positions of the target keys corresponding to the input characters, and perform simulated clicks to complete the automated test.

[0129] Of course, those skilled in the art can understand that the processor 620 also implements the technical solutions of the automated recognition method of the secure keyboard provided by any embodiment of this application.

[0130] Figure 6 The displayed electronic device 600 is merely an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0131] As Figure 6 shown, the electronic device 600 includes a processor 620, a storage device 610, an input device 630, and an output device 640; the number of processors 620 in the electronic device can be one or more, Figure 6 and one processor 620 is taken as an example herein; the processor 620, the storage device 610, the input device 630, and the output device 640 in the electronic device can be connected through a bus or other means, Figure 6 and connection through the bus 650 is taken as an example herein.

[0132] The storage device 610, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and module units, such as the program instructions corresponding to the automated recognition method of the secure keyboard in the embodiments of the present application.

[0133] The storage device 610 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the terminal, etc. In addition, the storage device 610 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some instances, the storage device 610 can further include a memory remotely set relative to the processor 620, and these remote memories can be connected through a network. Examples of the above network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0134] The input device 630 can be used to receive input digital, character information, or voice information, and generate key signal inputs related to the user settings and function controls of the electronic device. The output device 640 can include electronic devices such as a display screen and a speaker.

[0135] The automated recognition device, medium, and electronic device of the secure keyboard provided in the above embodiments can execute the automated recognition method of the secure keyboard provided in any embodiment of the present application, and have corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in the above embodiments can be referred to the automated recognition method of the secure keyboard provided in any embodiment of the present application.

[0136] Note that the above is only a preferred embodiment of the present application and the technical principles applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and various obvious changes, re-adjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments. Without departing from the concept of the present application, more other equivalent embodiments can be included, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. An automated recognition method for a secure keyboard, characterized in that, The method includes: If a secure keyboard pop-up event is detected, in response to a secure keyboard capture instruction, determine the elements within the capture range according to the UI hierarchy tree of the secure keyboard; the secure keyboard capture instruction is used to indicate a full capture or partial capture of the secure keyboard interface; when the keyboard layout of the secure keyboard pops up, it will randomly change to form a keyboard layout different from the physical keyboard; Establish a position comparison and calibration table for each element according to the reference positions within the capture range; the position comparison and calibration table is used to represent the position information of each key in the secure keyboard based on the reference position points; In response to the confirmation operation of the position comparison and calibration table, cut the capture range of the secure keyboard according to the position comparison and calibration table to obtain character key pictures and generate names for each character key picture; If an automated recognition event of the secure keyboard is detected, determine the input characters of the target input password; According to the correspondence between the input characters and the names of the character key pictures, determine the positions of the target keys corresponding to the input characters and perform simulated clicks to complete the automated test; Among them, if there is a keyboard switch key in the secure keyboard, in response to the touch operation of the switch key, perform a switching process on the keyboard to obtain the character key pictures and the names of the character key pictures of the secure keyboard after the switching process; the keyboard switch key refers to a key used to switch between two types of characters, including uppercase and lowercase switching, letter and number switching, and Chinese and English switching keys; the touch operation is a control instruction issued by a tester to click on the keyboard switch key; the switching process is a character switching operation implemented according to the touch operation of the switch key.

2. The method according to claim 1, wherein Before responding to the confirmation operation of the position comparison and calibration table, the method further includes: In response to the configuration operation of the similarity, determine the similarity range for keyboard recognition; Correspondingly, according to the correspondence between the input characters and the names of the character key pictures, determining the positions of the target keys corresponding to the input characters includes: Determine the positions of the target keys corresponding to the input characters according to the correspondence between the image of the input characters and the names of the character key pictures recognized based on the similarity range.

3. The method according to claim 1, characterized in that, Generating names for each character key picture includes: Name each character key picture according to the letter and / or number content of the character key picture and the function information of the function keys to obtain the names of each character key picture.

4. The method according to claim 1, characterized in that After establishing the position comparison and calibration table for each element according to the reference positions within the capture range and before responding to the confirmation operation of the position comparison and calibration table, the method further includes: Display the position comparison and calibration table of each element; In response to the calibration operation of the position comparison and calibration table of each element, obtain the calibration result of the position comparison and calibration table of each element.

5. The method according to claim 4, wherein Responding to the calibration operation of the position comparison and calibration table of each element to obtain the calibration result of the position comparison and calibration table of each element includes: In response to the calibration operation of the relative coordinates of the current element, determine the calibration result of the current element; Differentially display the calibration result position of the current element within the capture range of the secure keyboard; If the calibration operation of the relative coordinates of each element is completed, the calibration result of the position comparison calibration table for each element is obtained.

6. An automatic recognition device for a security keyboard, characterized in that, The device includes: An element determination module, configured to, if a secure keyboard pop-up event is detected, in response to a secure keyboard capture instruction, determine elements within the capture range according to the UI hierarchy tree of the secure keyboard; the secure keyboard capture instruction is used to indicate a complete capture or a partial capture of the secure keyboard interface; when the keyboard layout of the secure keyboard pops up, it will randomly change to form a keyboard layout different from the physical keyboard; A calibration table establishment module, configured to establish a position comparison calibration table for each element according to the reference positions within the capture range; the position comparison calibration table is used to represent the position information of each key in the secure keyboard based on the reference position points; A secure keyboard processing module, configured to, in response to the confirmation operation of the position comparison calibration table, cut the capture range of the secure keyboard according to the position comparison calibration table to obtain character key pictures and generate names for each character key picture; An input character determination module, configured to, if an automated recognition event of the secure keyboard is detected, determine the input characters of the target input password; An automated test module, configured to determine the position of the target key corresponding to the input character according to the correspondence between the input character and the names of the respective character key pictures, and perform a simulated click to complete the automated test; A key switch module, configured to, if there is a keyboard switch key in the secure keyboard, in response to a touch operation of the switch key, perform a switching process on the keyboard to obtain the character key pictures and the names of the character key pictures of the secure keyboard after the switching process; the keyboard switch key refers to a key used to switch between two types of characters, including a case switch, an alphanumeric switch, and a Chinese-English switch key; the touch operation is a control instruction issued by a tester to click on the keyboard switch key; the switching process is a character switching operation implemented according to the touch operation of the switch key.

7. The device according to claim 6, characterized in that, The device further includes: A similarity configuration module, configured to, in response to a similarity configuration operation, determine the similarity range for keyboard recognition; Correspondingly, the automated test module is specifically configured to: Determine the position of the target key corresponding to the input character according to the correspondence between the image of the input character and the names of the respective character key pictures recognized based on the similarity range.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the automated recognition method of the secure keyboard as described in any one of claims 1-5.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the automated recognition method of the secure keyboard as described in any one of claims 1-5.

Images