Mobile application verification code detection and automatic verification method and system

Abstract

The invention provides a mobile application verification code detection and automatic verification method and system. The method comprises the following steps: acquiring GUI component information and screen information; judging whether the GUI interface contains a verification code or not; judging the type of the verification code; performing automatic verification on different types of verification codes; and feedback is extracted and improved. The technical problems that system feature analysis and summarization of identification and automatic verification problems of the mobile application verification code are lacked, the verification code area cannot be detected in the barrier-free mobile application verification code identification process, and perfect automatic verification is not achieved are solved.

Description

Technical Field

[0001] This invention relates to the field of software engineering, and more specifically to a method and system for mobile application CAPTCHA detection and automated verification. Background Technology

[0002] Low-vision users interact with the world through voice assistants, magnification features, and accessible design. However, despite advancements in accessibility design for smart devices and applications, many still neglect the needs of visually impaired users, particularly in CAPTCHA design. According to WHO statistics in 2020, at least 2.8 billion people worldwide suffer from visual impairments, and existing CAPTCHA designs pose a significant barrier for them. Furthermore, the complexity and diversity of CAPTCHAs present a major challenge for automated verification; traditional methods struggle to effectively address evolving CAPTCHA technologies, potentially leading to security vulnerabilities and user experience issues. Existing methods include randomized methods, model-based methods, and deep learning / reinforcement learning-based methods. Randomized methods (such as Monkey) test applications by generating pseudo-random UI events, but their test paths often lack rationality, resulting in low coverage. Model-based methods optimize test case generation by building models of the application, but these methods typically ignore the semantic information of the GUI, making it difficult to cover complex application logic. Deep learning and reinforcement learning-based methods generate test scripts by learning user behavior patterns, but these methods require large amounts of training data and have limited generalization ability when facing new applications. Existing research indicates that CAPTCHA modules, as a crucial component for ensuring system security and user experience, are widely present in high-risk operation nodes of mobile apps, directly impacting user account security and business resource protection. Automated testing can systematically cover the security protection capabilities, user experience compatibility, and dynamic policy adaptability of multi-module CAPTCHAs, especially when dealing with high-frequency attacks, complex business rules, and rapid iteration requirements. However, current research lacks system detection technologies for mobile CAPTCHA recognition and automated verification. The main deficiency of existing technologies in mobile application CAPTCHA recognition and automated verification lies in the lack of system characteristic analysis and summarization of such issues, thus hindering the systematic detection of these issues during automated verification of mobile applications.

[0003] The existing invention patent application document CN119597373A, entitled "Software Interaction Method, Apparatus, and Device Based on AI, RPA, and AI Agent," describes a method that includes: acquiring dialogue information from a dialog application; identifying the dialogue information to obtain the software to be interacted with and the interactive operation; calling standard interactive components adapted to the software and interactive operation from a standard interactive component library embedded in a large language model; displaying the standard interactive components within the current dialogue; and performing interactive operations on the software based on the interactive information triggered by the standard interactive components. Therefore, by calling and displaying standard interactive components within a dialog application based on dialogue information in natural language input by the user, and by quickly driving and reshaping existing software based on the user's interaction with the standard interactive components, the convenience and efficiency of software interaction can be improved. However, the aforementioned existing solution has the following drawbacks:

[0004] 1. Lack of dedicated CAPTCHA processing capabilities

[0005] This solution focuses on general software interactions, such as calling API / RPA interfaces. Although its standard interaction component library includes a CAPTCHA input box component, it does not have logic for automatically recognizing and cracking CAPTCHAs.

[0006] The fundamental flaw is that it only provides an input interface for verification codes, still requiring manual recognition and input of the verification code content, which cannot meet the needs of visually impaired users or automated scenarios.

[0007] 2. Not adapted to dynamic CAPTCHA defense mechanism

[0008] The solution does not address countermeasures against CAPTCHA anti-automation strategies (such as slider trajectory detection and image rotation angle) and cannot handle complex CAPTCHA types (such as Google reCAPTCHA).

[0009] The fundamental flaw is that it relies on static component calls and lacks dynamic interaction strategies.

[0010] The existing invention patent application document CN119088927A, entitled "Intelligent Question Answering System and Method Based on Big Language Model for Tobacco Industry Professional Knowledge," describes a method that includes: using a preset dialogue generation model to integrate employee question-and-answer text as a knowledge base into a big language model platform; a matching answer output module to acquire the input question, search for a matching answer in the knowledge base through the big language model platform, and output the matching answer; when accessing the WeChat Work server, the application key and callback address are included; after verification, a callback is performed to access the application address, with WeChat Work carrying a verification code to obtain user information, and then data is exchanged between the user information and the database. However, the aforementioned existing solution constructs a knowledge base using a dataset. This solution trains a data acquisition unit to collect relevant professional knowledge, work processes, and policy documents related to the tobacco industry, and performs deep learning and training to form the knowledge base. The shortcomings of this existing technology are: the dataset is relatively limited, the training time is long, and the accuracy is not high. It is only applicable to the tobacco industry, and the scope of answers is limited.

[0011] In summary, existing technologies suffer from a lack of systematic feature analysis and summarization of mobile application CAPTCHA recognition and automated verification issues, an inability to detect CAPTCHA areas during accessible mobile application CAPTCHA recognition, and a lack of robust automated verification technology. Summary of the Invention

[0012] The technical problem to be solved by this invention is: how to solve the problems in the prior art that lack system feature analysis and summary of the recognition and automated verification of mobile application CAPTCHAs, cannot detect the CAPTCHA area in the process of CAPTCHA recognition of accessible mobile applications, and lack perfect automatic verification technology.

[0013] This invention solves the above-mentioned technical problems by employing the following technical solution: A method for mobile application CAPTCHA detection and automated verification includes:

[0014] S1. Extract component information from the mobile device's screen interface and the application's layout file;

[0015] S2. The verification code interface is judged by the data dictionary judgment method and the large language model judgment method. Specifically, the text information of the verification code interface in the database is retrieved according to the data dictionary. When the data dictionary judgment method cannot determine the verification code interface, the GUI interface screenshot in the component information is input into the large language model to ask whether the GUI interface contains a verification code.

[0016] S3. If so, extract the component information and context information of the input verification code from the GUI interface, use the preset prompt statement, input the component information and context information of the verification code into the large language model, and determine the verification code type.

[0017] S4. Perform rule matching on the different types of CAPTCHAs to execute different verification schemes;

[0018] S5. For cases where page transition is not triggered, extract error messages, perform differential analysis to extract error feedback information, and perform system optimization operations based on the error feedback information.

[0019] This invention analyzes and summarizes the CAPTCHAs in common mobile applications that present this type of problem, obtains the characteristics of the CAPTCHAs for this type of problem, and then systematically identifies the CAPTCHAs present in mobile applications through code feature classification.

[0020] In a more specific technical solution, in S1, the layout file includes: application .xml layout code.

[0021] In a more specific technical solution, in S2, a data dictionary is built based on a pre-set number of mobile applications containing CAPTCHAs in Google Play. The CAPTCHA interface in the data dictionary includes: text information.

[0022] Based on the component information of the GUI interface and the data in the data dictionary, determine whether the GUI interface contains a verification code;

[0023] Perform a data dictionary check. If the data dictionary check fails, input a screenshot of the GUI interface into the large language model and ask if the GUI interface contains a verification code.

[0024] This invention addresses the problem of CAPTCHA recognition in mobile software by employing a static data dictionary matching and large language model-assisted verification method. Starting with the application's XML file, it uses component information acquisition technology to obtain contextual information related to the CAPTCHA function, providing a foundation for subsequent CAPTCHA type determination.

[0025] In a more specific technical solution, S3 extracts the component information of the input verification code, extracts the context information of the verification code component, uses prompt statements, inputs the component information and context information into the large language model, and traverses the developer's code line by line.

[0026] In the operation of extracting context information, the context information includes: information surrounding the CAPTCHA component; among which, the candidate information sources of information surrounding the CAPTCHA component include: parent node control, leaf node control, controls on the same horizontal axis, and fragments of the current GUI page;

[0027] For each candidate information source, extract the text field and resource-id field, and combine the delimiters of the text field and resource-id field.

[0028] In a more specific technical solution, S4 performs automated verification of text-based CAPTCHAs. Specifically, the CAPTCHA image is saved to a local folder, and the ddddOCR library is integrated to segment and capture the coordinates of the text in the CAPTCHA image.

[0029] Using a large language model, task information for selecting CAPTCHA text is obtained from CAPTCHA screenshots; the mobile device screenshot, task information, and segmented text coordinate information are input into the large language model to design questions for the large language model.

[0030] The large language model identifies and returns the coordinates of the text to be clicked based on body temperature; it then retrieves the text coordinates from the console and uses ADB commands to perform the click operations sequentially.

[0031] This invention addresses the problem of identifying CAPTCHA types in mobile software. By acquiring the context information of the CAPTCHA component and generating input information for an input large language model, it asks which of the five types of CAPTCHAs the interface belongs to, thereby adapting appropriate automated verification steps to achieve accurate detection of this type of question and providing a foundation for subsequent automated CAPTCHA verification.

[0032] In a more specific technical solution, S4 performs automated verification of the slider CAPTCHA. The slider parameters are obtained through the component information extraction module, including: initial coordinates (x0, y0), background image size (W×H), and screen positioning parameters of the CAPTCHA area (x1, y1, x2, y2).

[0033] Input the slider parameters and a screenshot of the moving interface into the large language model, and set the problem for analysis and processing;

[0034] Obtain the coordinates of the endpoint to be slid from the large language model; use ADB commands to slide according to the initial coordinates and the coordinates of the endpoint to be slid.

[0035] In a more specific technical solution, S4 performs automated verification of rotating CAPTCHAs; specifically, the slider parameters are obtained through the component information extraction module.

[0036] Using a large language model, the task information containing the rotating CAPTCHA is obtained from the screenshot of the rotating CAPTCHA, and the degree of rotation to be obtained is analyzed using the large language model.

[0037] By obtaining and utilizing the relationship between the degree of rotation and the distance the slider moves, the coordinates of the endpoint to be slid are obtained. Then, using ADB commands, the CAPTCHA rotation operation is controlled according to the coordinates of the endpoint to be slid.

[0038] In a more specific technical solution, S4 performs automated verification of image-based CAPTCHAs, which involves obtaining the mobile device size W×H and the screen positioning parameters (x1, y1, x2, y2) of the CAPTCHA area.

[0039] Using a large language model, obtain the task information of the image-based CAPTCHA in the CAPTCHA screenshot, set questions, and ask the large language model.

[0040] The coordinates of the images to be selected are obtained from the large language model, and the ADB commands are used to select the images sequentially according to the coordinates.

[0041] In a more specific technical solution, S4 performs automated verification of Google reCAPTCHA CAPTCHA. The component information extraction module obtains the initial coordinates of the slider (x0, y0), the mobile device size W×H, and the screen positioning parameters (x1, y1, x2, y2) of the CAPTCHA area.

[0042] The task information of the Google reCAPTCHA CAPTCHA contained in the screenshot of the CAPTCHA is obtained by using a large language model, and questions are set to ask the large language model.

[0043] The coordinates of the image to be reCAPTCHAed are obtained from the large language model. The reCAPTCHA operation is performed according to the coordinates of the image using ADB commands. The operation is repeated in a loop according to a preset period until the interface redirects or the preset time limit is reached.

[0044] In a more specific technical solution, a mobile application CAPTCHA detection and automated verification system includes:

[0045] The information extraction module is used to extract component information from the screen interface of the mobile device and the layout file of the application;

[0046] The CAPTCHA determination module is used to determine the CAPTCHA interface using data dictionary determination method and large language model determination method. Specifically, according to the data dictionary, the CAPTCHA interface text information in the database is retrieved for determination. When the data dictionary determination method cannot determine the CAPTCHA interface, a screenshot of the GUI interface in the component information is input into the large language model to query whether the GUI interface contains a CAPTCHA. The CAPTCHA determination module is connected to the information extraction module.

[0047] The verification code type determination module is used to extract the component information and context information of the verification code component when the GUI interface contains a verification code. It uses preset prompt statements to input the component information and context information of the verification code component into the large language model to determine the verification code type. The verification code type determination module is connected to the verification code determination module.

[0048] The difference type verification module is used to perform rule matching operations on the difference type of CAPTCHA in order to execute different verification schemes. The difference type verification module is connected to the CAPTCHA type determination module.

[0049] The error analysis and system optimization module is used to extract error messages for cases where page transitions are not triggered. Through difference analysis, it extracts error feedback information and performs system optimization operations based on the error feedback information. The error analysis and system optimization module is connected to the difference type verification module.

[0050] The present invention has the following advantages over the prior art:

[0051] This invention combines a large language model with CAPTCHA verification, pioneering a new approach to automated CAPTCHA operations. It supports CAPTCHAs for most mobile applications currently on the market, including text selection verification, slider verification, image rotation verification, image selection verification, and Google reCAPTCHA verification. Extensive evaluation and testing were conducted on popular mobile applications on Google Play, achieving satisfactory results in terms of success rate and time.

[0052] This invention addresses the problem of CAPTCHA recognition in mobile software by employing a static data dictionary matching and large language model-assisted verification method. Starting with the application's XML file, it uses component information acquisition technology to obtain contextual information related to the CAPTCHA function, providing a foundation for subsequent CAPTCHA type determination.

[0053] This invention addresses the problem of identifying CAPTCHA types in mobile software. By obtaining the context information of the CAPTCHA component and generating input information for a large input language model, it asks which of the five types of CAPTCHAs the interface belongs to, thereby adapting appropriate automated verification steps and achieving accurate detection of this type of question.

[0054] This system enables visually impaired users to efficiently complete CAPTCHA operations and experience the convenience of accessibility technology in a wide range of scenarios. Its widespread application enhances the mobile application experience for visually impaired users and provides a theoretical and practical foundation for research on accessible CAPTCHA design.

[0055] This invention addresses the shortcomings of existing technologies, such as the lack of system feature analysis and summary for mobile application CAPTCHA recognition and automated verification, the inability to detect CAPTCHA areas during accessible mobile application CAPTCHA recognition, and the lack of robust automated verification. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the basic steps of a mobile application CAPTCHA detection and automated verification method according to Embodiment 1 of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Example 1

[0059] like Figure 1 As shown, the present invention provides a mobile application CAPTCHA detection and automated verification method, which includes the following basic steps:

[0060] S1. Obtaining GUI component information and screen information;

[0061] In this embodiment, the installation file of the mobile application is used as input to scan the code in the installation file, identify the native code that is not a third-party library, and obtain GUI component information.

[0062] In this embodiment, the .xml file of a mobile application under test is used as input; the code of the components involved in the application is traversed and scanned, and the component information is stored in JSON format. Component information typically includes the component's ID, text content, description information, coordinates, etc., which can provide a basis for subsequent filtering and judgment. Common CAPTCHA components fall into the following four categories:

[0063] (1) TextView: Displayed as text information in the GUI interface, used to determine the content of the interface, and can determine the type of verification code and the specific verification task based on its content;

[0064] (2) SeekBar: It is displayed as a slider on the GUI interface. SeekBar mainly appears in rotating verification codes and sliding verification codes. Users drag the slider to complete the verification.

[0065] (3) ImageView: Displayed as an image in the GUI interface. Most of the five types of verification code interfaces we have divided contain the ImageView component, which is usually the specific image that needs to be verified.

[0066] (4) Button: Displayed as a button in the GUI interface. Users click the button to submit and complete the verification process. Generally, except for slider and rotating verification codes, the three types of verification codes all contain the Button component.

[0067] S2. Determine whether the GUI interface contains a verification code;

[0068] In this embodiment, for the function of determining whether an interface contains a CAPTCHA, the component information is first analyzed using a data dictionary. Based on 100 mobile applications containing CAPTCHAs in Google Play, a data dictionary is created to determine whether the CAPTCHA interface may contain text information. The data dictionary includes the distinctive features of each CAPTCHA type, which are usually reflected in the attributes of the interface components, such as @resource-id, @text, and @content-desc. The presence of a CAPTCHA is determined by whether the component information of the test interface matches the content of the data dictionary. For example, text selection CAPTCHAs usually contain keywords related to "click" or "select," and the resource ID of the button may be related to "select" or "confirm." Slider CAPTCHAs may contain buttons with the words "slide" or "drag," or resource IDs with identifiers related to "slider." For Google's reCAPTCHA verification, the interface may contain keywords such as "reCAPTCHA" or "I am not a robot," and the resource ID usually contains "recaptcha" or similar identifiers.

[0069] If the data dictionary fails to determine the type of GUI, a screenshot of the obtained GUI interface is input into the large language model, and the model is asked whether the interface contains a CAPTCHA. If the large language model outputs "yes," the next step is to determine the CAPTCHA type. If the output is "no," the user is informed that the interface does not contain a CAPTCHA. For the prompts input to the large language model, we designed its language pattern, input it into the LLM to generate input content, and then checked the large language model's response.

[0070] S3. Determine the type of verification code;

[0071] In this embodiment, for the identified verification code area, contextual information related to the verification code is obtained through data flow and control flow analysis, and the verification code type is determined;

[0072] We extract the following three types of information in detail: (1) Text information includes the "hint-text" field, "resource-id" field, and "text" field of text controls. We select the first non-empty field in order. (2) Context information represents the information around the verification code component, which helps to determine the type of verification code. The candidate information sources include the parent node control, leaf node control, controls on the same horizontal axis, and fragments of the current GUI page. For each information source, we extract the "text" or "resource-id" field (the first non-empty field in order) and combine this information with delimiters. (3) Global context information represents information related to the high-level semantics of the verification code, which can further help to refine the judgment of the verification code type. The candidate information sources include the current activity name, application name, and multiple relevant components on the GUI page.

[0073] Preprocess the extracted information and design language patterns to generate the input information for the LLM, and finally obtain the judgment result of the large language model on the verification code type.

[0074] S4. Automatically verify different types of verification codes;

[0075] In this embodiment, different automated verification processes are performed for the matching of five types of verification codes;

[0076] In this embodiment, for the text click verification process, first save the verification code image obtained during the component extraction process to the local folder, and integrate the ddddOCR library to implement the function of splitting the text in the verification code image and capturing the coordinates. Next, obtain the task information of the text click verification code contained in the verification code screenshot through the large language model, such as "Click the following text 'year', 'bright' in sequence". At the same time, input the mobile device screenshot, task information, and the split text coordinate information into the large language model, and design questions for the large language model, such as "The size of the mobile device (W×H), the verification code area is located at (x1, y1) to (x2, y2), please record the target coordinates containing 'year' and 'bright' in the mobile device in sequence", and wait for its response. Through this process, the large language model can more accurately identify and return the text coordinates that the user needs to click. Subsequently, the system obtains the text coordinates from the console and performs sequential click operations using ADB commands;

[0077] In this embodiment, the slider verification process first obtains the initial coordinates (x0, y0), background image size (W×H), and screen positioning parameters (x1, y1, x2, y2) of the CAPTCHA area through the component information extraction module. Next, these parameters and a screenshot of the mobile interface are input into a large language model. A problem is set and analyzed using the large language model, such as "Given the initial slider position (x0, y0), mobile device size (W×H), and the CAPTCHA area located between (x1, y1) and (x2, y2), please calculate the target coordinates of the slider's movement." This yields the required endpoint coordinates. Subsequently, ADB commands are used to slide the slider according to the initial and endpoint coordinates. However, in actual operation, many software programs implement a series of defense mechanisms to prevent non-human operation. For example, they may detect whether the slider's movement speed is uniform or whether the trajectory is a straight line. Regarding the CAPTCHA defense measures, the system employs strategies such as forcing the slider to pause midway and intentionally shaking it during the slide.

[0078] In this embodiment, the verification of the rotating CAPTCHA first uses the component information extraction module. The system has already obtained the initial coordinates (x0, y0) of the slider, the mobile device size (W×H), and the screen positioning parameters (x1, y1, x2, y2) of the CAPTCHA area. By setting a question and using a large language model to analyze the required rotation degree, such as "Given the initial position of the slider (x0, y0), the mobile device size (W×H), and the CAPTCHA area located between (x1, y1) and (x2, y2), please calculate the rotation degree to make the image clockwise positive." Then, the relationship between the rotation degree and the slider movement distance is used to calculate the required endpoint coordinates of the slide, and then ADB commands are used to perform precise operation control according to the coordinates.

[0079] In this embodiment, the image selection verification process first uses the component information extraction module to obtain the mobile device size (W×H) and the screen positioning parameters (x1, y1, x2, y2) of the verification code area. Next, the system uses a large language model to obtain the task information of the image selection verification code contained in the screenshot, such as "Select the most suitable image 'a cat'". Then, combining this known information, a question is posed to the large language model, such as "Given the initial slider position (x0, y0), mobile device size (W×H), and the verification code area located between (x1, y1) and (x2, y2), please record the coordinate information of the image containing 'a cat'". This yields the coordinates of the image to be clicked, and then ADB commands are used to perform precise click operations according to these coordinates.

[0080] Table 1. Language Pattern Design of Prompt Statements

[0081]

[0082] For the prompt statements input to the large language model, we designed its language patterns, input them into the LLM to generate input content, and checked the response of the large language model. Ultimately, we derived three expression forms: a prompt input to determine whether the interface contains a CAPTCHA, a prompt input to determine the type of CAPTCHA, and a prompt input to solve the CAPTCHA. Table 1 shows the language patterns of the prompt statements we summarized.

[0083] S5. Extract feedback and make improvements.

[0084] In this embodiment, for each issue where the page redirection fails after entering the verification code identification process, a problem report related to the issue is compiled and feedback is provided;

[0085] Considering that large language models may not always be completely accurate in determining the type of CAPTCHA generated, we require the model not only to correctly determine whether the interface contains a CAPTCHA, but also to automatically determine the type of CAPTCHA that matches the interface, in order to ensure the effectiveness and accuracy of the automated verification scheme.

[0086] During implementation, the system designs automated scripts to convert text input generated by a large language model into Android ADB command format for execution. After the operation is completed, the system checks whether the page has changed, especially whether a page transition has been triggered, to confirm whether the CAPTCHA has been correctly passed. For cases where a page transition is not triggered, the system extracts error messages and uses differential analysis technology to extract feedback information. This error message extraction helps the system further optimize its judgment. Through this feedback mechanism, the system can continuously improve the effectiveness of CAPTCHA type judgment, ensuring that the GUI automation tool can successfully pass the CAPTCHA interface.

[0087] By following the steps above, the system can perform detection of CAPTCHAs in mobile applications, and the generated detection results can further help the system perform automated verification.

[0088] The principle by which this invention addresses the deficiencies of the two prior art documents mentioned in the background section includes:

[0089] 1. Systemic characteristics analysis

[0090] This invention is the first to summarize the common features of mobile CAPTCHAs, such as the combination patterns of components like TextView / SeekBar, and to establish a data dictionary, such as a keyword library for "slide" and "click". If the data dictionary fails to detect the CAPTCHA, a large language model is used for detection and judgment.

[0091] To address the technical deficiency of CN119597373A in lacking a feature library for CAPTCHA components, this invention achieves rapid interface determination through a data dictionary, see step S2 above;

[0092] In view of the technical defect that the verification code is not analyzed in CN119088927A, the present invention takes the verification code as an independent research object and constructs a type discrimination system. Refer to step S3 in the previous text.

[0093] 2. LLM + Context Enhancement to Improve Recognition Accuracy

[0094] When the data dictionary fails, the present invention inputs the GUI screenshot + component context (parent node / same horizontal axis controls, etc.) into the LLM. Refer to step S3; extract fields such as text / resource-id, etc., and combine them to generate an LLM prompt statement, such as: "Does this interface contain a verification code?"

[0095] In view of the technical defect that the LLM in CN119597373A is only used to call components, the present invention uses the LLM to understand the semantics of the verification code, such as: "Click on the words 'year' and 'bright' in sequence."

[0096] In view of the technical defect that the LLM in CN119088927A is used for knowledge retrieval, the present invention focuses on the visual / text feature analysis of the verification code.

[0097] 3. Dynamic Interaction Strategy

[0098] The present invention designs interaction strategies for 5 types of verification codes

[0099] CAPTCHA type Key technologies Text selection ddddOCR text coordinate segmentation → LLM parsing task → ADB click (S4) Slider verification LLM calculates the endpoint coordinates → ADB simulates a jittery sliding trajectory (to prevent detection). Rotation verification LLM outputs rotation angle → converts it to slider displacement → ADB operation reCAPTCHA Repeatedly click the coordinates returned by LLM until the page redirects (S4).

[0100] In view of the technical defect that complex operations cannot be performed in CN119597373A, the present invention realizes automated physical interaction through ADB commands;

[0101] 4. Error Feedback Closed-Loop Optimization System

[0102] The present invention extracts error messages (such as the page does not jump) for the verification failure scenario, and optimizes the system through difference analysis. Specifically, refer to step S5 in the previous text.

[0103] In view of the technical defect that there is no dedicated feedback mechanism for verification codes in both CN119597373A and CN119088927A, the present invention forms a closed loop of "recognition → verification → optimization" to improve the success rate.

[0104] 5. Do Not Need to Build a Finite Knowledge Base

[0105] The present invention designs a language mode to generate the input information of the LLM, and relies on the knowledge base capacity of the large language model itself to give reasonable answers.

[0106] In response to the limitations of CN119088927A, which only addresses the construction of a knowledge base for the tobacco industry, this invention simply designs a language pattern to generate input information for the LLM and ultimately obtains the judgment result of the large language model on the CAPTCHA type, without the need to construct a knowledge base.

[0107] In summary, this invention combines a large language model with CAPTCHA verification, pioneering a new approach to automated CAPTCHA operations. This invention supports CAPTCHAs for most mobile applications currently on the market, including text selection verification, slider verification, image rotation verification, image selection verification, and Google reCAPTCHA verification. Extensive evaluation and testing were conducted on popular mobile applications on Google Play, achieving satisfactory results in terms of success rate and time.

[0108] This invention addresses the problem of CAPTCHA recognition in mobile software by employing a static data dictionary matching and large language model-assisted verification method. Starting with the application's XML file, it uses component information acquisition technology to obtain contextual information related to the CAPTCHA function, providing a foundation for subsequent CAPTCHA type determination.

[0109] This invention addresses the problem of identifying CAPTCHA types in mobile software. By obtaining the context information of the CAPTCHA component and generating input information for a large input language model, it asks which of the five types of CAPTCHAs the interface belongs to, thereby adapting appropriate automated verification steps and achieving accurate detection of this type of question.

[0110] This system enables visually impaired users to efficiently complete CAPTCHA operations and experience the convenience of accessibility technology in a wide range of scenarios. Its widespread application enhances the mobile application experience for visually impaired users and provides a theoretical and practical foundation for research on accessible CAPTCHA design.

[0111] This invention addresses the shortcomings of existing technologies, such as the lack of system feature analysis and summary for mobile application CAPTCHA recognition and automated verification, the inability to detect CAPTCHA areas during accessible mobile application CAPTCHA recognition, and the lack of robust automated verification.

[0112] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for mobile application CAPTCHA detection and automated verification, characterized in that, The method includes: S1. Extract component information from the mobile device's screen interface and the application's layout file; S2. The verification code interface is judged by the data dictionary judgment method and the large language model judgment method. Specifically, the text information of the verification code interface in the database is retrieved according to the data dictionary. When the data dictionary judgment method cannot determine the verification code interface, the GUI interface screenshot in the component information is input into the large language model to ask whether the GUI interface contains a verification code. S3. If so, for the GUI interface, extract the component information of the input verification code and the context information of the verification code component, use the preset prompt statement, input the component information and the context information of the verification code component into the large language model, and determine the verification code type. S4. Perform rule matching on the different types of CAPTCHAs to execute different verification schemes; S5. For cases where page transition is not triggered, extract error messages, perform difference analysis to extract error feedback information, and perform system optimization operations based on the error feedback information.

2. The mobile application CAPTCHA detection and automated verification method according to claim 1, characterized in that, In step S1, the layout file includes: application .xml layout code.

3. The mobile application CAPTCHA detection and automated verification method according to claim 1, characterized in that, In step S2, the data dictionary is established based on a pre-set number of mobile applications containing CAPTCHAs in Google Play, wherein the CAPTCHA interface in the data dictionary includes: text information; Based on the component information of the GUI interface and the data in the data dictionary, determine whether the GUI interface contains a verification code; A data dictionary check is performed. If the data dictionary check fails, a screenshot of the GUI interface is input into the large language model to inquire whether a verification code exists on the GUI interface.

4. The mobile application CAPTCHA detection and automated verification method according to claim 1, characterized in that, In step S3, the component information of the input verification code is extracted, the context information of the verification code component is extracted, and a prompt statement is used to input the component information and the context information into the large language model to traverse the developer's code line by line. In the operation of extracting context information, the context information includes: information surrounding the CAPTCHA component; wherein, the candidate information sources of the information surrounding the CAPTCHA component include: parent node control, leaf node control, controls on the same horizontal axis, and fragments of the current GUI page; For each candidate information source, extract the text field and resource-id field, and combine the separators of the text field and the resource-id field.

5. The mobile application CAPTCHA detection and automated verification method according to claim 1, characterized in that, In S4, automated verification is performed on text-based CAPTCHAs. Specifically, the CAPTCHA image is saved to a local folder, and the ddddOCR library is integrated to segment and capture the coordinates of the text in the CAPTCHA image. Using the large language model, the task information of selecting the text verification code is obtained from the screenshot of the verification code; the mobile device screenshot, task information, and the coordinate information of the segmented text are input into the large language model to design the question of the large language model; The large language model identifies and returns the coordinates of the text to be clicked based on the body temperature; it then retrieves the text coordinates from the console and uses ADB commands to perform the click operations sequentially.

6. The mobile application CAPTCHA detection and automated verification method according to claim 1, characterized in that, In step S4, automated verification of the slider CAPTCHA is performed. The slider parameters are obtained through the component information extraction module, including: initial coordinates (x0, y0), background image size (W×H), and screen positioning parameters (x1, y1, x2, y2) of the CAPTCHA area. Input the slider parameters and a screenshot of the mobile interface into the large language model, and set the problem for analysis and processing; The coordinates of the endpoint to be slid are obtained from the large language model; the sliding is performed using ADB commands according to the initial coordinates and the coordinates of the endpoint to be slid.

7. The mobile application CAPTCHA detection and automated verification method according to claim 1, characterized in that, In step S4, automated verification is performed on the rotating CAPTCHA; wherein, the slider parameters are obtained through the component information extraction module. Using the large language model, the task information of the rotating CAPTCHA is obtained from the screenshot of the rotating CAPTCHA, and the degree of rotation to be obtained is analyzed using the large language model. The relationship between the degree of rotation to be obtained and the distance the slider moves is processed to obtain the coordinates of the endpoint to be slid. The ADB command is then used to control the rotation of the verification code according to the coordinates of the endpoint to be slid.

8. The mobile application CAPTCHA detection and automated verification method according to claim 1, characterized in that, In step S4, automated verification is performed on the image-based CAPTCHA, wherein the screen positioning parameters (x1, y1, x2, y2) of the mobile device size W×H and the CAPTCHA area are obtained. Using the large language model, obtain the task information of the image-based CAPTCHA screenshot, set a question, and ask the large language model. The coordinates of the image to be selected are obtained from the large language model, and the image selection operation is performed sequentially according to the coordinates of the image to be selected using ADB commands.

9. The mobile application CAPTCHA detection and automated verification method according to claim 1, characterized in that, In S4, automated verification is performed on the Google reCAPTCHA CAPTCHA. The component information extraction module obtains the initial coordinates of the slider (x0, y0), the mobile device size W×H, and the screen positioning parameters (x1, y1, x2, y2) of the CAPTCHA area. Using the large language model, the task information of the Google reCAPTCHA CAPTCHA contained in the CAPTCHA screenshot is obtained, and questions are set to ask the large language model. The coordinates of the image to be reCAPTCHA operated on are obtained from the large language model. The reCAPTCHA operation is performed according to the coordinates of the image to be reCAPTCHA operated on using ADB commands. The operation is repeated according to a preset period until the interface jumps or the preset time limit is reached.

10. A mobile application CAPTCHA detection and automated verification system, characterized in that, The system includes: The information extraction module is used to extract component information from the mobile device's screen interface and the application's layout files; The verification code determination module is used to determine the verification code interface through a data dictionary determination method and a large language model determination method. Specifically, according to the data dictionary, the verification code interface is determined by retrieving the text information in the database. If the data dictionary determination method cannot determine the verification code interface, a screenshot of the GUI interface in the component information is input into the large language model to inquire whether the GUI interface contains a verification code. The verification code determination module is connected to the information extraction module. The verification code type determination module is used to extract the component information and verification code component context information of the input verification code from the GUI interface when the GUI interface contains a verification code, and input the component information and verification code component context information into the large language model using a preset prompt statement to determine the verification code type. The verification code type determination module is connected to the verification code determination module. The difference type verification module is used to perform rule matching operations on the difference type of the verification code in order to execute different verification schemes. The difference type verification module is connected to the verification code type determination module. The error analysis and system optimization module is used to extract error messages for cases where page transition is not triggered, extract error feedback information through difference analysis, and perform system optimization operations based on the error feedback information. The error analysis and system optimization module is connected to the difference type verification module.

Images