Method and device for automatically logging in application

Abstract

The invention provides a method and device for automatically logging in an application, computing equipment and a computer readable storage medium, and the method comprises the steps: capturing all screens connected with current equipment after a script is started, and carrying out the image matching of the contents on all screens, thereby obtaining a target screen; that is, through global multi-screen matching logic, the application can run stably when being aroused at any position; furthermore, the script simulates the user operation on the target screen to complete the login operation, and in the automatic login process, the verification code on the other device is obtained through the synchronization technology and then is automatically extracted and input, so that full-automatic login is realized. According to the method provided by the embodiment of the invention, an efficient, stable and safe automatic login solution is provided, the method is suitable for automatic login in various complex working environments, and the working efficiency of personnel is greatly improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computing device and computer-readable storage medium for automatically logging into an application. Background Art

[0002] In the prior art, automated login solutions are mostly implemented through scripts and automation tools (such as Selenium, etc.) to simulate user input, operation interface and login process. Most automated scripts rely on fixed steps and positions to operate, and are prone to failure when the interface layout changes or is delayed. In addition, existing login environments often have multiple screens, and most multi-screen support solutions require manual configuration of screen parameters, which is not intelligent enough and requires more pre-configuration work to adapt to different environments. When automated operations identify page positions and elements in a multi-screen environment, inaccurate positioning is prone to occur. Summary of the invention

[0003] In view of this, the present application example provides a method, apparatus, computing device and computer-readable storage medium for automatically logging into an application to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of the present application, a method for automatically logging into an application is provided, comprising:

[0005] Starting an automatic login script on the current device and detecting the instance running status of the automatic login script;

[0006] Capturing all screens connected to the current device through the automatic login script instance, performing image matching on the contents on all screens, and acquiring the target screen;

[0007] The automatic login script is used to simulate user operations on the target screen to complete the login operation;

[0008] Verify whether the login is successful based on the content on the target screen.

[0009] According to a second aspect of an embodiment of the present application, a device for automatically logging into an application is provided, including:

[0010] A starting unit, used to start the automatic login script on the current device and detect the instance running status of the automatic login script;

[0011] A capture unit, used to capture all screens connected to the current device through the automatic login script instance, perform image matching on the contents on all the screens, and obtain a target screen;

[0012] A simulation unit, used for simulating user operations on the target screen to complete a login operation through the automatic login script;

[0013] The verification unit verifies whether the login is successful according to the content on the target screen.

[0014] According to a third aspect of an embodiment of the present application, a computing device is provided, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the method for automatically logging into an application when executing the instructions.

[0015] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer instructions, and the instructions are executed by a processor to perform the steps of the method for automatically logging into an application.

[0016] In an embodiment of the present application, a method for automatically logging into an application is proposed. After startup, the script in the method captures all screens connected to the current device, performs image matching on the contents on all screens, and thereby obtains the target screen; that is, through the global multi-screen matching logic, the application can run stably at any location when it is invoked; further, the script simulates the user operation on the target screen to complete the login operation, and in the process of automatic login, the verification code on another device is automatically extracted and input through synchronization technology to achieve fully automatic login. The method of the embodiment of the present application provides an efficient, stable and secure automated login solution, which is suitable for automatic login in various complex working environments and greatly improves the work efficiency of personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a structural block diagram of a computing device provided in an embodiment of the present application;

[0018] Figure 2 It is a flowchart of a method for automatically logging into an application provided in an embodiment of the present application;

[0019] Figure 3 It is a structural diagram of a device for automatically logging into an application provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0021] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.

[0022] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "in response to determination".

[0023] In the present application, a method, apparatus, computing device and computer-readable storage medium for automatically logging into an application are provided, which are described in detail one by one in the following embodiments.

[0024] Figure 1 The structure block diagram of a computing device 100 according to an embodiment of the present application is shown. The components of the computing device 100 include but are not limited to a memory 110 and a processor 120. The processor 120 is connected to the memory 110 via a bus 130, and the database 150 is used to store data.

[0025] The computing device 100 also includes an access device 140 that enables the computing device 100 to communicate via one or more networks 160. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 140 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0026] In one embodiment of the present application, the above components of the computing device 100 and Figure 1 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 1The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0027] The computing device 100 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 100 may also be a mobile or stationary server.

[0028] With the development of information technology and the popularization of the Internet, users need to frequently log in to various online services and systems in their daily life and work. In order to ensure security, these platforms often require users to enter usernames, passwords, and verification codes or perform other forms of identity authentication. However, the repetitive and cumbersome login process not only wastes time, but also easily leads to poor user experience. Especially for the OA system and remote office (verification) system within the enterprise, it is necessary to frequently log in to the intranet system to access work resources during work, and each login requires entering a username, password, and verification code, which is time-consuming and prone to errors. Some existing automation tools have many limitations in practical applications, especially when facing complex verification code recognition and multi-screen environments, the success rate is low and unstable, most of them need to manually configure screen parameters, are not intelligent enough, require more pre-configuration work when adapting to different environments, and are prone to inaccurate positioning problems.

[0029] Therefore, in order to solve the above problems, this application proposes a method for automatically logging into an application. Figure 1 The processor 120 in the embodiment may execute Figure 2 A method for automatically logging into an application is shown, comprising the following steps:

[0030] Step 202: Start the automatic login script on the current device and detect the instance running status of the automatic login script.

[0031] In this step, detect and ensure that only one instance of the automatic login script is running at the same time, so as to avoid resource competition and conflict and enhance the stability and reliability of the system. In order to prevent multiple instances of the same script from running at the same time, the process detection tool provided by the system or the file-based locking mechanism can be used to implement instance management. The core of this step is to determine whether there is already a script instance running. If there is already an identical instance running, the current script instance should terminate the startup or exit the operation; if there is no identical script instance running, continue to execute the subsequent steps.

[0032] Step 204: Capture all screens connected to the current device through the automatic login script instance, perform image matching on the contents on all screens, and obtain the target screen.

[0033] In this step, the interfaces displayed on all screens connected to the current device are detected by an image template matching method to determine whether to start the login operation. Specifically, the steps include:

[0034] (1) Get all screen object instances connected to the current device, such as using a script to get all screens connected to the current device.

[0035] (2) Obtain the coordinates of the upper left corner and lower right corner of each screen. The coordinates of the upper left corner and lower right corner of each screen are related to the resolution of each screen and the position of the main and auxiliary screens (such as the main and auxiliary screens are distributed left and right, or up and down). Therefore, the coordinates of the upper left corner and lower right corner of each screen are calculated using the following steps:

[0036] screen_info=[

[0037] int(int(bounds.origin.x) / int(bounds.size.width)*max_res[0]),#upper left corner horizontal coordinate

[0038] int(int(bounds.origin.y) / int(bounds.size.height)*max_res[1]),#Upper left corner vertical coordinate

[0039] int(int(bounds.origin.x) / int(bounds.size.width)*max_res[0])+max_res[0],#horizontal coordinate of the lower right corner

[0040] int(int(bounds.origin.y) / int(bounds.size.height)*max_res[1])+max_res[1]#lower right corner ordinate ]

[0042] Among them, bounds is an object containing the screen boundaries, usually with origin (indicating the coordinates of the upper left corner of the screen) and size (indicating the width and height of the screen) attributes.

[0043] max_res is a list or tuple containing the maximum screen resolution, in the form [width, height].

[0044] Calculate the first value:

[0045] Divide the horizontal coordinate of the border (origin.x) by the width (size.width) to get a ratio, then multiply it by the width of the maximum resolution (max_res[0]), and finally round it to an integer to get a horizontal coordinate relative to the maximum resolution.

[0046] Calculate the second value:

[0047] Divide the vertical coordinate of the boundary (origin.y) by the height (size.height) to get a ratio, then multiply it by the height of the maximum resolution (max_res[1]), and finally round it to an integer to get a vertical coordinate relative to the maximum resolution.

[0048] Calculate the third value:

[0049] First, the first value (the horizontal coordinate of the left border) is repeated to get the horizontal coordinate value of the left border of the screen at the maximum resolution. Then the width of the maximum resolution, max_res[0], is added to get the horizontal coordinate of the right border. This value represents the horizontal coordinate of the right border at the relative maximum resolution.

[0050] Calculate the fourth value:

[0051] First, the second value (the vertical coordinate of the upper boundary) is repeated to obtain the vertical coordinate value of the upper boundary of the screen at the maximum resolution. Then the maximum resolution height max_res[1] is added to obtain the vertical coordinate of the lower boundary. This value represents the vertical coordinate of the lower boundary at the relative maximum resolution.

[0052] Add this information to the array screens_info.

[0053] (3) Obtaining screenshot information of each screen according to the rectangle formed by the coordinates of the upper left corner and the lower right corner of each screen;

[0054] screenshot=ImageGrab.grab(bbox=screens_info[i]).convert('L')

[0055] (4) Traverse the screenshot information of each screen, match the screenshot information with the first image template, and search whether the target login interface exists in the screenshot information of the screen, for example:

[0056] result=cv2.matchTemplate(screenshot,template,...) / / match

[0057] _,max_val,_,max_loc=cv2.minMaxLoc(result) / / Get the best matching rate and its location

[0058] Among them, screenshot is the background image, that is, the screenshot; template is the first image template you want to search in the large image, such as a target login window; max_val is the maximum value in the result matrix, indicating the percentage of the best match, and max_loc is the position of the maximum value in the result matrix, indicating the position coordinates of the best match.

[0059] Furthermore, if the matching result max_val is smaller than the first threshold, the screenshot information of other screens is continuously matched until the matching result is larger than the first threshold, thereby determining the target screen.

[0060] Step 206: The automatic login script simulates the user operation on the target screen to complete the login operation.

[0061] In this step, the automatic login is realized by simulating the user's operation by using a script on the target screen obtained in step 204, including:

[0062] (1) Positioning UI elements

[0063] For example, in step 205, a template matching method is used to find the positions of UI elements that require interaction, such as input boxes and buttons:

[0064] Username_pos=(500,400): The position of the username input box.

[0065] password_pos=(500,450): the position of the password input box.

[0066] login_button_pos=(500,500): the position of the login button.

[0067] (2) Simulate mouse clicks

[0068] Move the mouse to an input box, button, etc., and then click.

[0069] pyautogui.moveTo(username_pos)

[0070] pyautogui.click()

[0071] (3) Input text

[0072] Implement a script to simulate entering a username and password in the input box:

[0073] pyautogui.write('your_username',interval=0.1)#Enter the username at 0.1 second intervals

[0074] pyautogui.write('your_password',interval=0.1)#Enter password at 0.1 second intervals

[0075] (4) Click the Login button

[0076] Continue to use the script to simulate the user clicking the "Login" button to complete the login operation:

[0077] pyautogui.moveTo(login_button_pos)

[0078] pyautogui.click()

[0079] Preferably, if there is a verification code dialog box on the login interface, the verification code extraction and input are simulated before simulating clicking the login button. In a feasible implementation, the verification code received by the mobile terminal is synchronized with the current device used for login. For example, for Apple devices, the information (Messages) is synchronized to other Apple devices such as Mac desktops through the cloud. Those skilled in the art should know that the above synchronization method is only for example and not exhaustive.

[0080] Furthermore, the automatic login script instance on the current device used for login obtains the SMS content from the synchronization information, and extracts the verification code that conforms to the format in the synchronization information through a regular expression; continues to simulate the keyboard to enter the verification code, and uses the simulated keyboard input to enter the verification code into the specified location.

[0081] Preferably, it is detected whether the current login process needs to set a new password. Specifically, when a login failure occurs, such as after multiple login failures, it is confirmed in the target screen by template matching whether an interface for changing the password appears; if it appears, a new password that complies with the rules is generated and entered for confirmation, and then updated to the local password record file. Specifically, the screenshot information of the target screen is matched with the second picture template. If the screenshot information includes the second picture template, a new password that complies with the rules is generated and entered for confirmation, and then updated to the local password record file, wherein the second picture template includes an interface for changing the password.

[0082] Step 208: Verify whether the login is successful based on the current content on the target screen.

[0083] In this step, a screenshot of the target screen is captured again to obtain second screenshot information, and a signature element indicating whether the login is successful is detected in the second screenshot information, thereby determining whether the login is successful.

[0084] Specifically, the second screenshot information is matched with the third picture template, so as to determine whether the login is successful according to the matching result; the third picture template includes the iconic elements of successful login to the application system.

[0085] In the embodiment of the present application, in order to solve the problem that the existing automatic login scheme is not intelligent enough and does not support multi-screen automatic login, a method for automatically logging into an application is proposed. The script in the method captures all screens connected to the current device after startup, and performs image matching on the contents on all screens to obtain the target screen. Through the global multi-screen matching logic, the application can run stably at any location when it is called up; further, the script simulates the user operation on the target screen to complete the login operation. In the process of automatic login, the verification code on another device is obtained through synchronization technology, and then the verification code is automatically extracted and input to achieve fully automatic login. The method of the embodiment of the present application provides an efficient, stable and secure automatic login solution, which is suitable for automatic login in various working environments and greatly improves the work efficiency of employees.

[0086] Corresponding to the above method embodiment, the present application also provides an embodiment of a device for automatically logging into an application, such as Figure 3 As shown, the device comprises:

[0087] A starting unit, used to start the automatic login script on the current device and detect the instance running status of the automatic login script;

[0088] A capture unit, used to capture all screens connected to the current device through the automatic login script instance, perform image matching on the contents on all the screens, and obtain a target screen;

[0089] A simulation unit, used for simulating user operations on the target screen to complete a login operation through the automatic login script;

[0090] The verification unit verifies whether the login is successful according to the content on the target screen.

[0091] The above is a schematic scheme of a device or system for automatically logging into an application in this embodiment. It should be noted that the technical scheme of the device or system for automatically logging into an application and the technical scheme of the method for automatically logging into an application belong to the same concept, so the details of the technical scheme of the device or system for automatically logging into an application that are not described in detail can all be referred to the description of the technical scheme of the method for automatically logging into an application.

[0092] In one embodiment of the present application, a computing device is also provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the method for automatically logging into an application when executing the instructions.

[0093] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned method for automatically logging into an application belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above-mentioned method for automatically logging into an application.

[0094] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the method for automatically logging into an application as described above.

[0095] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned automatic login application method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned automatic login application method.

[0096] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0097] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0098] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0099] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0100] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.

Claims

1. A method for automatically logging into an application, characterized in that: The method includes: Starting an automatic login script on the current device and detecting the instance running status of the automatic login script; Capturing all screens connected to the current device through the automatic login script instance, performing image matching on the contents on all screens, and acquiring the target screen; The automatic login script is used to simulate user operations on the target screen to complete the login operation; Verify that the login was successful based on the current content on the target screen.

2. The method according to claim 1, wherein The step of starting the automatic login script on the current device and detecting the instance running status of the automatic login script includes: If there is an instance of the automatic login script running, terminate the startup of the current script instance; if there is no identical instance running, continue to execute subsequent steps.

3. The method according to claim 1, wherein The capturing of all screens connected to the current device through the automatic login script instance, performing image matching on the contents on all screens, and acquiring the target screen comprises: Get all screen object instances connected to the current device; Get the upper left corner coordinates and lower right corner coordinates of each screen according to the size and maximum resolution of each screen; Obtaining screenshot information of each screen according to a rectangle formed by the coordinates of the upper left corner and the lower right corner of each screen; The screenshot information of each screen is traversed, and the screenshot information is matched with a first picture template. If the first picture template exists in the screenshot information, the current screen is determined to be the target screen; the first picture template includes a target login interface.

4. The method according to claim 3, wherein: If the first picture template exists in the screenshot information, determining the current screen as the target screen includes: If the matching result between the screenshot information and the first image template is less than a first threshold, the screenshot information of other screens continues to be matched.

5. The method according to claim 1, wherein: The automatic login script further includes: simulating a user operation on the target screen to complete the login operation; Before simulating clicking the login button, the extraction and input of the verification code are simulated, including: synchronizing the verification code received by the mobile terminal with the current device, and the automatic login script obtains the SMS content from the synchronization information; then extracting the verification code that conforms to the format in the synchronization information through a regular expression, and using the simulated keyboard to enter the verification code into the specified location.

6. The method according to claim 5, wherein: The method also includes: when a login failure occurs, matching the screenshot information of the target screen with the second picture template; if the screenshot information includes the second picture template, generating a new password that complies with the rules and updating the new password to the local password record file after entering and confirming; the second picture template includes an interface for modifying the password.

7. The method according to claim 1, wherein: Verifying whether the login is successful based on the content on the target screen includes: Capture the second screenshot information of the target screen again, match the second screenshot information with the third image template, and determine whether the login is successful based on the matching result; the third image template includes iconic elements of a successful login.

8. A device for automatically logging into an application, characterized in that: The method includes: A starting unit, used to start the automatic login script on the current device and detect the instance running status of the automatic login script; A capture unit, used to capture all screens connected to the current device through the automatic login script instance, perform image matching on the contents on all the screens, and obtain a target screen; A simulation unit, used for simulating user operations on the target screen to complete a login operation through the automatic login script; The verification unit verifies whether the login is successful according to the content on the target screen.

9. A computing device comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the instructions, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing computer instructions, characterized in that: When the instruction is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images