API information capturing method, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides an API information capturing method, electronic equipment, a storage medium and a program product. The method comprises the steps of obtaining a target sub-link queue of an initial page based on a link obtaining strategy and at least one prompt interaction with a large model; sequentially traversing the sub-links in the target sub-link queue based on the dynamic crawler program, establishing page models of pages corresponding to the traversed sub-links, and executing the following operations for each page model: simulating interaction between the client and the server in the initial page based on an interaction result and the dynamic crawler program; and obtaining API information about the initial page generated by interaction. According to the scheme of the embodiment, the problem that a large model cannot dynamically crawl is solved, the problem that a crawling scheme needs to be repeatedly designed for different webpage structures is also avoided, and the accuracy and comprehensiveness of API information crawling and the information crawling efficiency are improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of data processing technology, and in particular to an API information capture method, electronic device, storage medium, and program product. Background Art

[0002] API (Application Programming Interface) provides developers with a way to transfer data and perform operations between the front-end and back-end. In the process of software testing and development, API information collection plays a very important role. API information collection can help the testing team to have a more comprehensive understanding of the functions and behaviors of Web (page) applications, and help the development team evaluate the performance and scalability of the system. At the same time, various potential security vulnerabilities in the security testing process can often be discovered by testing the API information of the Web system. Therefore, the collection of API information for front-end and back-end interactions plays a very important role in the field of security testing. Through API information, testers can promptly discover the above potential vulnerabilities and fix them in a timely manner, thereby improving the security of the system. However, the current API information collection scheme is very time-consuming and labor-intensive. Summary of the Invention

[0003] The embodiments of the present disclosure provide an API information capture method, an electronic device, a storage medium, and a program product.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for capturing API information, which may include:

[0005] Based on a preset link acquisition strategy and at least one prompt interaction with a preset large model, a target sub-link queue of the initial page is obtained;

[0006] Based on a preset dynamic crawler program, the sub-links in the target sub-link queue are traversed in sequence, and page models of the pages corresponding to the traversed sub-links are established. The following operations are performed for each of the page models:

[0007] Perform at least one prompt interaction with the large model, simulate the interaction between the client and the server in the initial page based on the interaction result and the dynamic crawler program; and obtain application programming interface API information about the initial page generated by the interaction.

[0008] In a second aspect, an embodiment of the present disclosure provides an electronic device, characterized by including:

[0009] one or more processors;

[0010] A memory having one or more programs stored thereon, which, when executed by the one or more processors, enables the one or more processors to implement the API information capture method;

[0011] One or more input / output (I / O) interfaces are connected between the processor and the memory and configured to implement information interaction between the processor and the memory.

[0012] In a third aspect, an embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for capturing API information is implemented.

[0013] In a fourth aspect, an embodiment of the present disclosure further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the API information capture method is implemented.

[0014] The embodiment of the present disclosure obtains the target sub-link queue of the initial page based on a preset link acquisition strategy and at least one prompt interaction with a preset large model, thereby ensuring the comprehensiveness of the acquired sub-links; traverses the sub-links in the target sub-link queue in sequence based on a preset dynamic crawler program, and establishes a page model of the page corresponding to the traversed sub-link, which can fully utilize the information crawling ability of the dynamic crawler; performs at least one prompt interaction with the large model, simulates the interaction between the client and the server in the initial page based on the interaction result and the dynamic crawler program, and combines the above-mentioned solution of obtaining the target sub-link queue by interacting with the large model to realize the use of a combination of a large model static web page crawler and a dynamic crawler, which not only solves the problem that the large model cannot be crawled dynamically, but also avoids the problem of repeatedly designing crawling solutions when facing different web page structures, realizes the automatic crawling of API information, and improves the accuracy, comprehensiveness and information capture efficiency of API information crawling. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In the accompanying drawings of the embodiments of the present disclosure:

[0016] Figure 1 A flow chart of a method for capturing page information provided by an embodiment of the present disclosure;

[0017] Figure 2 A schematic diagram of a method for capturing page information provided by an embodiment of the present disclosure;

[0018] Figure 3 A schematic diagram of the page information crawling solution architecture provided by an embodiment of the present disclosure;

[0019] Figure 4 A schematic diagram illustrating a large model intelligently filling in a text box according to an embodiment of the present disclosure;

[0020] Figure 5 This is a block diagram of the electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the communication perception data processing method and computer-readable storage medium provided by the embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0022] The present disclosure will be described more fully hereinafter with reference to the accompanying drawings, but the illustrated embodiments may be embodied in different forms, and the present disclosure should not be construed as limited to the embodiments set forth below. Rather, these embodiments are provided so that the present disclosure will be thorough and complete and will fully understand the scope of the present disclosure to those skilled in the art.

[0023] The accompanying drawings of the embodiments of the present disclosure are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the detailed embodiments, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing the detailed embodiments with reference to the accompanying drawings.

[0024] The present disclosure may be described with reference to plan views and / or cross-sectional views by way of ideal schematic views of the present disclosure. Therefore, the exemplary illustrations may be modified according to manufacturing techniques and / or tolerances.

[0025] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0026] The terms used in this disclosure are only used to describe specific embodiments and are not intended to limit the disclosure. As used in this disclosure, the term "and / or" includes any and all combinations of one or more related enumerated items. As used in this disclosure, the singular forms "a" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. As used in this disclosure, the terms "comprising" and "made of" specify the presence of the features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof.

[0027] Unless otherwise defined, all terms (including technical and scientific terms) used in this disclosure have the same meanings as those commonly understood by those skilled in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined in this disclosure.

[0028] The present disclosure is not limited to the embodiments shown in the drawings, but includes modifications of the configurations formed based on the manufacturing process. Therefore, the regions illustrated in the drawings have schematic properties, and the shapes of the regions shown in the drawings illustrate the specific shapes of the regions of the elements, but are not intended to be limiting.

[0029] Some automated API scraping tools in the related art use web crawlers, which are automated programs used to retrieve information from the internet. They traverse web pages, collect data, and store it locally or on remote servers for subsequent processing or analysis. Web crawlers are programs or scripts that automatically crawl specific information from web pages according to certain rules.

[0030] Web crawlers can be divided into dynamic crawlers and static crawlers. Static crawlers are mainly used to collect and capture data from static web pages. They obtain data by simply downloading the web page content and parsing the HTML (Hypertext Markup Language) code, without executing JavaScript (a directly interpreted scripting language) code. Dynamic crawlers are more complex and need to simulate browser behavior and execute JavaScript code to obtain the complete page content. To crawl dynamic pages, tools such as Selenium and Playwright are often used to simulate browser operations, or to obtain data by analyzing the web page structure and XHR requests (used to exchange data with the server, the object that the Ajax function relies on) technology. However, during the data collection process, web crawlers need to design different rules based on the structure of different web pages, which increases the collection cost. Due to changes in the source web page code, the crawler may face failure during execution, so professionals are required to continuously maintain and adjust it, which also adds additional workload.

[0031] The specific implementations of some existing dynamic web crawlers only submit the configuration after filling in the page content, without taking into account the complex Web (network) system that includes selectors and multi-layer sub-page structures. When faced with complex network structures, their application is limited. Secondly, when dynamically clicking on a page, some existing web crawler solutions use regular expressions to filter the buttons on the page. This has the problem of incomplete filtering, resulting in only some buttons being recognized and the remaining sub-pages being unable to be triggered, resulting in a low data collection coverage rate. In addition, some existing solutions often fill in too simple content when filling in the page, filling in the same content in different text boxes, which may cause the problem of not being able to trigger the sub-page after filling in errors, and ultimately lead to incomplete API capture.

[0032] Large models (i.e., large-scale deep learning models) are a type of AI model that has attracted considerable attention in the field of natural language processing. Through pre-training, large models learn information such as grammatical structure, variable naming conventions, and function call relationships from large code bases, resulting in powerful code understanding and reasoning capabilities. By indexing and modeling code, large models can achieve more accurate and efficient code search and retrieval. The code retrieval and understanding capabilities of large models help improve the speed with which system applications can understand unfamiliar code and also facilitate understanding of diverse web page structures when crawling. In code retrieval scenarios, large models are very simple to use and are often more efficient than traditional, complex script designs. However, the output of large models does not have a fixed format, and its accuracy is strictly dependent on user-defined prompts. Existing web crawling solutions using large models directly use the code output of the large models, which suffers from low accuracy. Furthermore, when used as web crawlers, large models can only recognize static web page source code and cannot capture API data from dynamic web pages.

[0033] The embodiment of the present disclosure obtains the target sub-link queue of the initial page based on a preset link acquisition strategy and at least one prompt interaction with a preset large model, thereby ensuring the comprehensiveness of the acquired sub-links; traverses the sub-links in the target sub-link queue in sequence based on a preset dynamic crawler program, and establishes a page model of the page corresponding to the traversed sub-link, which can fully utilize the information crawling ability of the dynamic crawler; performs at least one prompt interaction with the large model, simulates the interaction between the client and the server in the initial page based on the interaction result and the dynamic crawler program, and combines the above-mentioned solution of obtaining the target sub-link queue by interacting with the large model to realize the use of a combination of a large model static web page crawler and a dynamic crawler, which not only solves the problem that the large model cannot be crawled dynamically, but also avoids the problem of repeatedly designing crawling solutions when facing different web page structures, thereby realizing automatic crawling of API information.

[0034] In summary, the method for automatically crawling API information in this embodiment combines the code retrieval capabilities of a large model with the dynamic execution capabilities of a dynamic crawler, making it suitable for complex web systems with multi-layered sub-page structures. Furthermore, it designs an efficient page button traversal scheme and intelligently populates page content, improving data collection coverage.

[0035] The embodiments of the present disclosure can be applied to various fields, including but not limited to the fields of artificial intelligence and data processing.

[0036] The API information capture method of the embodiments of the present disclosure can be applied to any terminal device, including but not limited to: in-vehicle devices, user equipment (UE), mobile devices, computing devices, wearable devices, etc., for example, including but not limited to cellular phones, cordless phones, personal digital assistants (PDAs), portable computers, etc. The API information capture method can be implemented by a processor calling computer-readable program instructions stored in a memory.

[0037] The embodiment of the present disclosure can be applied to a data processing device for collecting web API information, and can be implemented on any IT (Internet technology) product. For example, it can be deployed on a Linux kernel PC (personal computer), or other devices can be used to build an application environment through a virtual machine.

[0038] The following is a detailed introduction to the embodiments of the present disclosure.

[0039] The present disclosure provides a method for capturing API information. Figure 1 As shown, the method may include steps S11-S12:

[0040] S11. Based on a preset link acquisition strategy and at least one prompt interaction with a preset large model, a target sub-link queue of the initial page is acquired.

[0041] In the embodiment of the present disclosure, based on a preset link acquisition strategy and at least one prompt interaction with a preset macro model, a target sub-link queue of an initial page is acquired, including:

[0042] Find at least one sub-link existing in the initial page based on the link acquisition strategy;

[0043] Perform at least one prompt interaction with the large model to obtain at least one sub-link existing in the initial page output by the large model;

[0044] The at least one sub-link found is supplemented based on the at least one sub-link output by the large model, and all the sub-links obtained after the supplementation constitute a target sub-link queue.

[0045] In the embodiment of the present disclosure, the large model is set on a preset server. When prompt interaction with the large model is required, communication with the server can be performed to achieve interaction with the large model.

[0046] In an embodiment of the present disclosure, in order to obtain a target sub-link queue from an initial page, a preset link acquisition strategy can be used to find at least one sub-link existing in the initial page in advance, and then at least one prompt interaction can be performed to obtain at least one sub-link. The at least one sub-link obtained after the prompt interaction can be used as a supplement to the at least one sub-link queue found, and the target sub-link queue can be selected from the multiple sub-link queues obtained after the supplement.

[0047] In the disclosed embodiment, the above embodiment scheme realizes obtaining the target sub-link queue based on two schemes: prompt interaction with the big model and link acquisition strategy, thereby improving the comprehensiveness of the obtained sub-link queue and providing a technical basis for comprehensive crawling of API information.

[0048] In the embodiment of the present disclosure, Figure 2 、 Figure 3 As shown, the embodiment scheme is introduced in detail below.

[0049] In the embodiment of the present disclosure, the link acquisition strategy may include but is not limited to a regular matching method.

[0050] In the embodiment of the present disclosure, searching for at least one sub-link existing in the initial page based on the link acquisition strategy may include:

[0051] Get the webpage source code of the initial page;

[0052] Based on the web page source code, use the regular matching method to find at least one sub-link existing in the initial page.

[0053] In the embodiment of the present disclosure, the sub-link may include but is not limited to a URL (Uniform Resource Locator).

[0054] In the embodiment of the present disclosure, the sub-link acquisition stage of an initial web page can be entered first, that is, the web page source code of the initial page of the web page to be tested is first obtained, and all existing sub-links are found as much as possible from the initial page of the web page using regular matching.

[0055] In the disclosed embodiment, the initial page has relatively few executable elements, and some sub-links are hidden in select drop-down boxes or among other sub-links. Therefore, a direct dynamic crawl of the initial page cannot access most of the sub-pages. At this stage, a regular expression matching method is used to search for sub-links in the webpage source code, and at least one sub-link can be matched.

[0056] In the embodiment of the present disclosure, after obtaining at least one sub-link through the regular matching method, multiple prompt interactions can be performed with the big model through the first prompt interaction and the second prompt interaction, and the big model outputs multiple sub-links of the initial page through multiple interactions.

[0057] In an embodiment of the present disclosure, performing at least one prompt interaction with the large model to obtain at least one sub-link present in the initial page output by the large model may include:

[0058] Perform the first prompt interaction with the large model, prompting the large model to output multiple sub-links on the initial page;

[0059] A second prompt interaction is performed with the large model, prompting the large model to perform multiple rounds of voting on the multiple sub-links, and selecting at least one sub-link from the multiple sub-link queues for output according to the voting results.

[0060] In the embodiment of the present disclosure, by performing multiple rounds of interactions with the large model, generating multiple sub-link queues in a multi-round voting manner, and voting on the results, sub-links that cannot be matched by regular matching can be crawled, so that the embodiment of the present application can greatly improve the information crawling effect when facing heterogeneous web pages.

[0061] In the embodiment of the present disclosure, performing the first prompt interaction with the large model to prompt the large model to output multiple sub-links in the initial page may include:

[0062] The webpage source code of the initial page and the preset first prompt word are input into the large model to perform the first prompt interaction with the large model, prompting the large model to output multiple sub-links existing in the webpage source code.

[0063] In the embodiment of the present disclosure, by performing multiple rounds of interactions with the big model, the big model is prompted to output multiple sub-links existing in the web page source code, so that the required sub-links can be accurately obtained through the big model, thereby improving the efficiency and accuracy of obtaining the sub-links.

[0064] In an embodiment of the present disclosure, a first prompt word can be used to interact with the large model in the first prompt interaction. The first prompt word can include but is not limited to: separators, structured answers, a small number of successful examples provided to the large model, etc., so that the large model can understand the task requirements as much as possible.

[0065] In the embodiment of the present disclosure, a specific embodiment of the first prompt word is given below, for example: The first prompt word: {##Now you are a source code analyzer, and your purpose is to analyze the sub-links contained in the source code (such as the source code of a web page). ##Do not output the code that can analyze the sub-links, directly output the sub-link content, you need to analyze the following source code: +html_content+. ###Task flow: 1. You need to analyze the source code in detail, and find out the hidden sub-links as much as possible based on the unique structure of the source code. 2. Provided in JSON (JSON is a text-based, lightweight, and generally considered easy to read / write data exchange format.) format, which contains the following key: URL. 3. Output all sub-links that may exist in the code, do not miss any. If a sub-link only provides a resource path, but no protocol, host or port is provided, the resource path is output directly. 4. Output example:

[0066] Source code:

[0067] <link href= / static / css / chunk-libs.60cffd1a.css

[0068] rel=stylesheet> <div id＝ap>

[0069] Output:

[0070] {URL: / static / css / chunk-libs.60cffd1a.css}

[0071] }.

[0072] In the embodiment of the present disclosure, in the first prompt word, delimiters can be used to separate the various requirements in the task process to facilitate the understanding of the large model. The first instruction in the first prompt word is a summary of the overall command, so that the large model has a preliminary understanding. The second instruction is because the large model may mistakenly judge that the code that needs to be output is the code for analyzing the source code, so it is necessary to prohibit the large model from outputting code. html_content is the source code content. When the large model is required to output data in a fixed format, such as a sub-link queue, format requirements can be added to the prompt word, and the output can be in JSON format or list format. This ensures the consistency of the large model output and facilitates subsequent processing. Finally, a simple successful example is given. If the large model is simply asked a math problem, the large model is likely to output the wrong answer. However, before entering the problem into the large model, a simpler question can be entered first, and the problem-solving process and the correct answer can be given. In this way, the probability of the large model giving the correct answer will be greatly increased.

[0073] In the disclosed embodiment, after a round of prompt interaction with the large model, a sub-link queue is eventually output:

[0074] URL_lis1={url1,url2,…,url k };

[0075] Among them, URL_lis1 contains k (k is a positive integer) sub-links output by the large model. However, based on this, some sub-links may still be incorrect. In order to make the output results of this stage more accurate, the above interaction process with the large model is repeated. After N rounds of prompt interaction, N (N is a positive integer) sub-link queues will be obtained:

[0076] URL_lis={URL_lis1,URL_lis2,…,URL_lis N }.

[0077] In the embodiment of the present disclosure, the first prompt interaction may include one or more rounds of prompt interactions. Each round of prompt interaction may obtain a sub-link queue, and multiple rounds of prompt interactions may obtain multiple sub-link queues.

[0078] In the embodiment of the present disclosure, a second prompt interaction with the large model can then be performed using a multi-round voting method, so that the large model can analyze which queue is likely to have the most correct sub-links. The method is to design a second prompt word and input it to the large model, with the aim of allowing the large model to observe the commonalities, differences and logical relationships between these sub-link queues, and try to infer the option that is most likely to be the correct answer. The logic of this method is that even if the queue output by the large model each time has a certain degree of randomness or analysis errors, by repeating this process, a relatively comprehensive and diverse sample of answers can be obtained. By analyzing these answers, consistency or trends can be identified, so that a final decision or judgment can be made, and the final sub-link queue can be selected as the target sub-link queue. Multi-round voting improves the accuracy and credibility of decision-making through multi-party observation and reasoning. The following is a specific embodiment of the second prompt word.

[0079] In the disclosed embodiment, for example, the second prompt reads: {##You are now a URL analysis expert, able to analyze the most accurate sub-link queue from multiple sub-link queues on a webpage. ##Do not output the code that can analyze the sub-link queues; directly output the sub-link queue you believe is the most accurate. You need to analyze the following sub-link queue: +URL_lists+. ###Task Flow: 1. Analyze all sub-link queues in detail and select the final sub-link queue based on the URL structure and similarities between them. 2. Directly output the final selected sub-link queue.}

[0080] In the embodiment of the present disclosure, the sub-link queue output by the large model after the second prompt interaction is obtained, and the multiple sub-links found by the regular matching method can be supplemented to obtain the target sub-link queue.

[0081] In the embodiment of the present disclosure, supplementing the at least one found sub-link based on the at least one sub-link output by the large model may include:

[0082] A sub-link included in the at least one sub-link output by the large model but not included in the at least one sub-link found is added to the at least one sub-link found.

[0083] In the embodiment of the present disclosure, after the first prompt interaction and the second prompt interaction, the sub-links output by the large model are obtained, the sub-link queue output by the large model is compared with the sub-link queue obtained by regular matching, and the sub-links included in the sub-link queue output by the large model but not included in the sub-link queue obtained by regular matching are added to the sub-link queue obtained by regular matching to obtain the target sub-link queue finally used for multiple rounds of voting.

[0084] In the embodiment of the present disclosure, because there are many types of web page architectures, regular matching may not be satisfactory for crawling sub-links existing in some uncommon web page architectures, and adapting to different web page architectures also requires time and manpower. By utilizing the high universality of the large model, it is possible to crawl sub-links that regular matching cannot obtain, which serves as a good supplement to regular matching.

[0085] S12. Based on the preset dynamic crawler program, the sub-links in the target sub-link queue are traversed in sequence, and a page model of the page corresponding to the traversed sub-link is established. The following operations are performed for each page model: at least one prompt interaction is performed with the large model, and based on the interaction result and the dynamic crawler program, the interaction between the client and the server in the initial page is simulated; and the application programming interface API information about the initial page generated by the interaction is obtained.

[0086] In the embodiment of the present disclosure, the dynamic crawler program can be set on the terminal device.

[0087] In the embodiment of the present disclosure, Figure 2 、 Figure 3 As shown, after executing step S11, the initial stage of the dynamic crawler program can be entered. The meaning of this initial stage mainly refers to starting the dynamic crawler program so that the dynamic crawler program can automatically traverse the target sub-link queue obtained in the above step S11 and automatically fill in the form (including but not limited to any one or more of the following: text box information, selection box information, button information, etc.). Elements that require user execution (including but not limited to any one or more of the following: text boxes, selection boxes and buttons, etc.) will be simulated and executed by the dynamic crawler program to ensure that the dynamic crawler program can obtain valid information on the page based on simulating user interaction. For example, for the selection box element in the page, the dynamic crawler program will simulate the user's selection behavior; for the text box element, the dynamic crawler program will simulate the user's filling behavior. This automated method of simulating user behavior helps to improve the adaptability and intelligence level of the dynamic crawler program, enabling it to more effectively process various types of web page content.

[0088] In the embodiment of the present disclosure, at least one prompt interaction is performed with the large model, and based on the interaction result and the dynamic crawler program, the interaction between the client and the server in the initial page is simulated, including:

[0089] Perform at least one prompt interaction with the large model to determine the target content to be filled in the text box of the page model;

[0090] Fill in the text box based on the dynamic crawler and target content, and submit the form containing the information in the text box to simulate the interaction between the client and the server in the initial page.

[0091] In the disclosed embodiment, a large model is used to analyze the page, intelligently inferring the target content that needs to be filled in the text box, thus avoiding filling errors. In addition, the text box is filled in and submitted based on the target content to simulate the interaction between the client and the server on the initial page, and the API information generated by the interaction about the initial page is obtained. This achieves simulated interaction based on the page model and can be recursively executed or accessed even when new elements or new pages are added to the page model, thereby improving the coverage of information crawling.

[0092] In the embodiment of the present disclosure, performing at least one prompt interaction with the large model to determine target content to be filled in the text box of the page model includes:

[0093] Perform a third prompt interaction with the large model, prompting the large model to determine multiple alternative contents that should be filled in the current text box based on the preset text box prompt information;

[0094] The fourth prompt interaction is performed with the large model, prompting the large model to conduct multiple rounds of voting on the multiple candidate contents determined, and select the target content from the multiple candidate contents for output according to the voting results.

[0095] In the embodiment of the present disclosure, by conducting multiple rounds of interactions with the big model to obtain multiple alternative contents, the big model can fully understand the text box prompt information, thereby accurately analyzing the content that should be filled in the text box; selecting the target content from multiple alternative contents in a multi-round voting manner can further improve the accuracy of the filled-in content, reduce the impact of filling-in errors, and provide a technical basis for improving the efficiency and accuracy of crawling API information.

[0096] In the disclosed embodiment, the content filled in the text box is completely customized by the user, and simple filling is often not feasible. For text box elements in the page model, random filling may lead to filling errors, such as filling in different types of data, or data that does not conform to the settings, which in turn affects the execution of the subsequent dynamic crawler program. For example, when a text box needs to be filled with text containing only numbers, but the dynamic crawler program fills in "testname", the page will report an error and subsequent elements will not be clickable. Furthermore, verification code filling boxes often appear on the page. In this case, the page prompt information is a verification code image, and traditional methods cannot automatically identify the verification code content. To avoid these situations, it is necessary to consider the prompt information in the page. They are usually used to guide users on what type of data they should fill in. However, not all text boxes have clear prompt information, and sometimes the prompt information requires users to understand. To address this situation, it is possible to consider designing a large model prompt word to interact with the large model at least once, so that based on this at least one prompt interaction, the large model outputs the content that should be filled in each text box.

[0097] In the embodiment of the present disclosure, a third prompt word can be designed to perform a third prompt interaction with the large model, aiming to enable the large model to analyze the data that should be filled in the current text box based on the prompt information of the text box. The large model analyzes the possible format restrictions or character type restrictions of this text box based on the prompt information of the text box, and outputs the correct filling content that complies with the specifications.

[0098] In the disclosed embodiment, the following is an example of a third prompt word when the prompt information is text. For example, the third prompt word is: {##You are a text generation expert and can automatically generate corresponding text content based on the preset text box prompt. ##Output requirements: 1. Output in the format of the output example; 2. Do not output irrelevant content; 3. Do not output the example itself; 4. Example:

[0099] ###Text box prompt: Gender;

[0100] Output example: Male;

[0101] ###Text box prompt: start date;

[0102] Output example: 2024-02-10;

[0103] Your text box prompt is: +text_key.

[0104] }.

[0105] In the disclosed embodiment, the following is an example of the third prompt word when the prompt information is a verification code. For example, the third prompt word is: {##You are a verification code recognition expert and can automatically analyze the verification code content in the image based on the provided image. ##Output requirement: Do not output irrelevant content;

[0106] Your verification code picture is: +text_pic.

[0107] }.

[0108] In the disclosed embodiments, this method combines the prompt information in the text box on the page with the intelligent analysis capabilities of the large model, making the process of filling in the text box more intelligent and accurate, improving the dynamic crawler's adaptability to complex pages, reducing the impact of filling errors, and thus improving the program's stability and efficiency. Furthermore, when logging into a web page during the crawler process, verification code filling is often encountered. Using the large model for analysis can greatly improve the success rate of login.

[0109] In the embodiment of the present disclosure, in this stage, since it involves interaction with the large model, in order to ensure the accuracy of the large model output, multiple rounds of voting will also be used to analyze and optimize the results. However, for efficiency reasons, combined with the real production environment, the entire web crawler time cannot be too long. And because the output task here is relatively simple for the large model, the difference in the generated results will not be too large, so the number of interactions with the large model can be reduced here, and the large model is only allowed to accept the third prompt word M times (M is a positive integer), M is less than half of N, and eventually M results to be selected (i.e., M content to be selected) will be generated:

[0110]

[0111] In the disclosed embodiment, a fourth prompt word can then be designed, and a fourth prompt interaction can be performed with the large model. Multiple rounds of voting are used to vote on the M candidate results to ensure that the content entered into the text box is acceptable to the webpage. Furthermore, if the dynamic crawler program detects an error on the page after entering the content generated by the large model into the text box, it will return to the previous step and regenerate the content.

[0112] In the disclosed embodiment, a detailed embodiment of the fourth prompt word is given below. For example, the fourth prompt word: {##Now you are a text logic analysis expert. Enter multiple text contents and a text prompt. You can analyze which text content best matches the text prompt. ##Do not output the code that can analyze the text content. Directly output the text content you think is most consistent. You need to analyze the following text content: +texts+; The text prompt is: +prompt+; ###Task process: 1. You need to analyze in detail the type restrictions that may be contained in the text prompt. The text content you ultimately select must meet these restrictions; 2. Directly output the final selected text content.}

[0113] In the embodiment of the present disclosure, after obtaining the target content to be filled in the text box based on the above steps, the text box can be filled in to simulate the interaction process through the dynamic crawler program. The dynamic crawler program simulates the interaction process based on the page model corresponding to the current page.

[0114] In the embodiment of the present disclosure, Figure 4 As shown, it is a schematic diagram of the process of intelligently filling in text boxes of a large model. Among them, the URL page to be tested includes a text box to be filled in (i.e., a fill-in box for the user identity ID) and a corresponding button. The button cannot be clicked when the fill-in box for the user ID is not filled in or filled in incorrectly. The URL page to be tested also includes prompt information for the text box, such as "Fill in testname", and an example of filling in testname in the fill-in box for the user ID is given on one side, and a prompt is prompted that the text box is limited to text only, and a filling error is prompted when filling in non-text information, keeping the button unclickable. After adopting the embodiment of the present disclosure, the large model will perform intelligent analysis and semantic understanding of the prompt information of the text box and the prompt words entered by the user (i.e., the third prompt word and the fourth prompt word), output the target content that the text box needs to fill in, for example, output the analysis result 123, and fill in the text box. At this time, if the content filled in is correct, the button is in a clickable state.

[0115] In an embodiment of the present disclosure, after performing at least one prompt interaction with the large model and simulating the interaction between the client and the server on the initial page based on the interaction result and the dynamic crawler program, the method may further include:

[0116] When the interaction causes the page model to generate new elements, the new elements are executed recursively based on the preset sub-dynamic crawler program until the current page model is executed or a new page is entered; the new elements may include but are not limited to any one or more of the following: new text boxes, selection boxes and buttons.

[0117] In the embodiment of the present disclosure, when a new element is generated in the page model, the new element can be executed recursively to enable the dynamic crawler program to fully parse and process the page and crawl deeper information of the web page.

[0118] In the embodiment of the present disclosure, after accessing the sub-URL, the dynamic crawler program will establish a page model of the current page in order to analyze and process the various elements in the page. Due to the diversity of web page structures, executing an element may produce multiple results (for example, filling in a text box may generate a new text box or a new selection box), so the dynamic crawler process uniquely designs a page model to record all the elements in the simulated page. When the page model changes, such as when a new element appears in the page model, the program will start a recursive sub-dynamic crawler process under the current dynamic crawler process that only contains the new elements (the sub-dynamic crawler process performs corresponding operations for the new elements). This ensures the depth of the dynamic execution method proposed in this article to ensure that all elements on the page model can be executed until the web page enters the next new page or the current page model is executed.

[0119] In the disclosed embodiment, for example, assuming that a dynamic crawler process A is currently entered, the button list before execution is [a1, b1], and the text box list is [a2, b2]. After executing an element, a sub-page may pop up on the page. At this time, the page button list becomes [c1, f1], and the text box list becomes [c2, d2]. For the newly added button elements [c1, f1] and text box elements [c2, d2], since the priority of the sub-page is often higher, if these elements are not clicked, the remaining elements may not be able to continue to be executed. Therefore, in this case, it is necessary to recursively enter a sub-dynamic crawler process A_A. The sub-dynamic crawler process A_A will first fill in the text box elements [c2, d2] in sequence, and then click the newly added button elements [c1, f1] in sequence. During the clicking process, if a new page element is added, the sub-process will be entered again. If no new page element is added, execution will continue until the sub-dynamic crawler process A_A ends. When the child dynamic crawler process A_A ends, the sub-page that pops up under dynamic crawler process A will also disappear, and dynamic crawler process A can be continued. This process of recursively updating the page model ensures that the crawler program proposed in the embodiment of the present disclosure fully parses and processes the page. Compared with other methods of using dynamic crawlers for data crawling, the embodiment of the present disclosure can crawl deeper information on the web page.

[0120] In an embodiment of the present disclosure, after performing at least one prompt interaction with the large model and simulating the interaction between the client and the server on the initial page based on the interaction result and the dynamic crawler program, the method further includes:

[0121] When the interaction causes the page model to generate a new page, the new page is used as the initial page, and the steps of obtaining the target sub-link queue of the initial page based on the preset link acquisition strategy and at least one prompt interaction with the preset large model are returned to obtain the target sub-link queue corresponding to the new page, and based on the dynamic crawler program, the simulated interaction operation of the page model corresponding to each sub-link in the target sub-link queue corresponding to the new page is executed until all page models corresponding to the new page are executed.

[0122] In the embodiment of the present disclosure, when a new page is generated by the page model, by re-executing the embodiment of the present disclosure on the new page, it is ensured that all pages contained in each sub-link are executed, ensuring comprehensive parsing and crawling of page information, thereby ensuring the comprehensiveness of the crawled API information.

[0123] In the disclosed embodiment, the next step is to enter the submission and maintenance stage in the dynamic crawler program. After the text boxes in the initial stage of the above-mentioned dynamic crawler program are filled, the form is submitted in this stage, and the submission process includes simulating clicks on button elements in the page. When a new page is generated during the form submission process, the page model of the new page will be entered, and the current new page will be used as the initial page. Starting from the stage of the first prompt interaction with the large model, a new round of sub-crawling process will be carried out. This process will be carried out recursively to ensure access and parsing of all relevant pages. When new elements appear in the new page, the dynamic crawler program will update the page model corresponding to the new page, simulate the execution of the new elements, and submit the form.

[0124] In the embodiment of the present disclosure, when any new page appears during the dynamic crawler program's simulated interaction, the new page will be executed based on a preset link acquisition strategy and at least one prompt interaction with the preset large model to obtain the target sub-link queue of the new page, and the target sub-link queue corresponding to the new page will be added to the target sub-link queue of the original page. For example, assuming that the first target sub-link queue obtained based on the initial page contains sub-link A, sub-link B, and sub-link C, then sub-link A, sub-link B, and sub-link C are arranged in sequence in the first target sub-link queue, and each time link A, sub-link B, and sub-link C are traversed, a page model corresponding to the sub-link will be established, and based on the corresponding page model, the dynamic crawler program will perform interactive simulation to crawl the API information of the page corresponding to the corresponding sub-link. If the sub-link A is currently traversed, the page model a of the corresponding page is created based on the sub-link A. During the interactive simulation process performed by the dynamic crawler program based on the page model a, if a new page is generated, the preset link acquisition strategy and at least one prompt interaction with the preset large model are executed for the new page to obtain the target sub-link queue process of the new page. Assuming that the sub-links corresponding to the new page are A1, A2, and A3, the sub-links A1, A2, and A3 are added to the original target sub-link queues A, B, and C. Since the new page is a new page generated based on the page model corresponding to the sub-link A, the sub-links A1, A2, and A3 are added after the sub-link A, so that when the target sub-link queue is traversed next, A1, A2, and A3 are traversed first before B and C, so that all pages and elements in the page model corresponding to the sub-link A are executed first, and then the page models corresponding to the sub-links B and C are executed in sequence, thereby ensuring that all pages and all elements corresponding to each sub-link are executed, thereby ensuring the comprehensiveness of the information crawling.

[0125] In the embodiment of the present disclosure, during the entire dynamic crawling process, the API crawler will record the API format and corresponding data sent for front-end and back-end interaction as web page API information. The API crawler uses the web page behavior recording function of the automated testing tool, and the corresponding data recorded will include the URL of each request, the request method (such as GET or POST, POST packages the form and hides it in the background and sends it to the server; GET packages the form and appends it to the end of the URL before sending it), request parameters, and the server's response content and other information. These records will help monitor and analyze the request and response situations during the dynamic crawling process, as well as further data processing and analysis.

[0126] The embodiments of the present disclosure include at least the following advantages:

[0127] 1. By interacting with the large model in multiple rounds, generating multiple sub-link queues in a multi-round voting manner, and voting on the results, it is possible to crawl sub-links that cannot be matched by regular matching, and it can also greatly improve the crawling effect when facing heterogeneous web pages.

[0128] 2. Combining the methods of large-model static web crawlers and dynamic web crawlers, the problem of crawling large-model dynamic content is solved, and the repetitive work of designing different crawling solutions for different web page structures is avoided, thereby improving work efficiency.

[0129] 3. When filling dynamic web page content, such as recognizing and filling verification codes, the page is analyzed through a large model, and the content of the text box that needs to be filled is intelligently inferred. It is simple and easy to use, does not require complex code design, and avoids filling errors.

[0130] 4. When a new button (i.e., a new element) or a new page appears after clicking a button, the dynamic crawler program increases the coverage of information crawling through recursive clicks or accesses in the process of obtaining API information, thereby obtaining the required data more comprehensively.

[0131] The present disclosure also provides an electronic device 100, such as Figure 5 Shown, including:

[0132] One or more processors 101;

[0133] A memory 102 storing one or more programs, which, when executed by the one or more processors 101, enable the one or more processors 101 to implement the API information capture method;

[0134] One or more input / output (I / O) interfaces 103 are connected between the processor 101 and the memory 102 and configured to implement information exchange between the processor 101 and the memory 102 .

[0135] In the embodiment of the present disclosure, the electronic device may include but is not limited to a terminal device.

[0136] The embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for capturing API information is implemented.

[0137] The embodiment of the present disclosure also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the API information capture method.

[0138] Those skilled in the art will appreciate that all or some of the functional modules / units disclosed above may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0139] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation.

[0140] Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit (CPU), a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH) or other disk storage; compact disc (CD-ROM), digital versatile disc (DVD) or other optical disc storage; magnetic cassettes, tapes, disk storage or other magnetic storage; any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0141] The present disclosure has disclosed example embodiments, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for purposes of limitation. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly indicated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A method for capturing API information, characterized in that: The method comprises: Based on a preset link acquisition strategy and at least one prompt interaction with a preset large model, a target sub-link queue of the initial page is acquired; Based on a preset dynamic crawler program, the sub-links in the target sub-link queue are traversed in sequence, and page models of the pages corresponding to the traversed sub-links are established. The following operations are performed for each of the page models: Perform at least one prompt interaction with the large model, simulate the interaction between the client and the server in the initial page based on the interaction result and the dynamic crawler program; and obtain application programming interface API information about the initial page generated by the interaction.

2. The method for capturing API information according to claim 1, wherein: The step of obtaining a target sub-link queue of the initial page based on a preset link acquisition strategy and at least one prompt interaction with a preset large model includes: Find at least one sub-link existing in the initial page based on the link acquisition strategy; Perform at least one prompt interaction with the large model to obtain at least one sub-link existing in the initial page output by the large model; The at least one sub-link found is supplemented based on the at least one sub-link output by the large model, and all sub-links obtained after the supplementation constitute the target sub-link queue.

3. The API information capture method according to claim 2, wherein: The performing at least one prompt interaction with the large model to obtain at least one sub-link existing in the initial page output by the large model includes: Performing a first prompt interaction with the large model, prompting the large model to output a plurality of sub-links in the initial page; Perform a second prompt interaction with the large model, prompting the large model to perform multiple rounds of voting on the multiple sub-links, and select at least one sub-link from the multiple sub-links for output according to the voting results.

4. The API information capture method according to claim 3, wherein: The first prompt interaction with the large model, prompting the large model to output multiple sub-links in the initial page, includes: The webpage source code of the initial page and a preset first prompt word are input into the large model to perform a first prompt interaction with the large model, prompting the large model to output multiple sub-links existing in the webpage source code.

5. The API information capture method according to claim 1, wherein: The step of performing at least one prompt interaction with the large model and simulating the interaction between the client and the server on the initial page based on the interaction result and the dynamic crawler program includes: Perform at least one prompt interaction with the large model to determine target content to be filled in the text box of the page model; The text box is filled in based on the dynamic crawler program and the target content, and a form containing the information in the text box is submitted to simulate the interaction between the client and the server in the initial page.

6. The API information capture method according to claim 5, wherein: The performing at least one prompt interaction with the large model to determine target content to be filled in the text box of the page model includes: Performing a third prompt interaction with the large model, prompting the large model to determine multiple candidate contents that should be filled in the current text box according to preset text box prompt information; Perform a fourth prompt interaction with the large model, prompting the large model to perform multiple rounds of voting on the multiple candidate contents determined, and select the target content from the multiple candidate contents determined according to the voting results to output.

7. The method for capturing API information according to claim 1, wherein: After performing at least one prompt interaction with the large model and simulating the interaction between the client and the server on the initial page based on the interaction result and the dynamic crawler program, the method further includes: When the interaction causes the page model to generate new elements, the new elements are executed recursively based on the preset sub-dynamic crawler program until the current page model is executed or a new page is entered; the new elements include any one or more of the following: new text boxes, selection boxes and buttons.

8. The method for capturing API information according to claim 1, wherein: After performing at least one prompt interaction with the large model and simulating the interaction between the client and the server on the initial page based on the interaction result and the dynamic crawler program, the method further includes: In the case where the interaction causes the page model to generate a new page, the new page is used as the initial page, and the step of obtaining the target sub-link queue of the initial page based on the preset link acquisition strategy and at least one prompt interaction with the preset large model is returned to obtain the target sub-link queue corresponding to the new page, and based on the dynamic crawler program, the simulated interaction operation of the page model corresponding to each sub-link in the target sub-link queue corresponding to the new page is executed until all the page models corresponding to the new page are executed.

9. An electronic device, characterized in that: include: one or more processors; A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the API information crawling method according to any one of claims 1 to 8; One or more input / output (I / O) interfaces are connected between the processor and the memory and configured to implement information interaction between the processor and the memory.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the API information capture method according to any one of claims 1 to 8.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for capturing API information according to any one of claims 1 to 8 is implemented.

Images