A method, system, device, and medium for detecting APIs that adapt to changes in website structure.

Abstract

This invention provides an API detection method, system, device, and medium that adapts to changes in website structure, belonging to the field of internet technology. The method comprises the following steps: collecting API call data from various types of websites and extracting API call features; constructing a machine learning model, training the model using the extracted API call features, and automatically adjusting the recognition strategy within the model; using the machine learning model to identify APIs on new or updated websites, generating API information, detecting API changes between different versions of each website, compiling the changed API information, and generating a change report; and structuring the identified API information to generate an API directory. This invention automatically adjusts the recognition strategy to adapt to the structure and coding habits of different websites, and can also automatically filter out newly added or removed API interfaces, improving API identification accuracy and maintenance efficiency.

Description

Technical Field

[0001] This invention belongs to the field of Internet technology, specifically relating to an API detection method, system, device, and medium that adapts to changes in website structure. Background Technology

[0002] API stands for Application Programming Interface.

[0003] APIs play a crucial role in software development and maintenance, serving as a hub for effective communication and data exchange between different software systems or components. With the rapid development of internet technology, especially in the context of cloud computing, microservice architecture, and CI / CD, software systems require frequent iterative updates, including the addition, modification, or deletion of APIs. API changes are common during software iteration, driven by factors such as changing business requirements, technology stack updates, performance optimization, or security improvements. However, these changes bring the following problems: First, poor compatibility; API changes may affect applications and systems that rely on these APIs, leading to functional failures or system crashes. Second, high maintenance difficulty; developers need to track API changes to update corresponding application code and documentation, a process often manual and error-prone. Third, update delays; in large systems, API changes may require simultaneous updates across multiple components and services, potentially causing update delays and deployment setbacks.

[0004] Traditional API identification and tracing methods primarily rely on manually maintained documentation or static code analysis techniques, which often fail to effectively address rapidly changing requirements. For example, manual documentation may not be updated in a timely manner, while static code analysis may not accurately identify dynamically generated or runtime-modified APIs. Furthermore, traditional methods struggle to automatically identify and adapt to changes in website or application architecture, limiting their effectiveness in today's rapidly iterating environments.

[0005] This is a shortcoming of the existing technology. Therefore, it is very necessary to provide an API detection method, system, device and medium that adapts to changes in website structure in order to address the above-mentioned defects in the existing technology. Summary of the Invention

[0006] To address the shortcomings of traditional API identification and tracking methods, which are unable to effectively respond to rapidly changing needs and struggle to automatically identify and adapt to changes in website or application structures, this invention provides an API detection method, system, device, and medium that adapts to changes in website structure, thereby resolving the aforementioned technical problems.

[0007] In a first aspect, the present invention provides an API detection method that adapts to changes in website structure, comprising the following steps:

[0008] S1. Collect API call data from various types of websites and extract API call characteristics from the API call data;

[0009] S2. Build a machine learning model, train the machine learning model using the extracted API call features, learn the API recognition patterns of different websites, and automatically adjust the recognition strategy in the machine learning model;

[0010] S3. Use the trained machine learning model to identify APIs for new or updated websites, generate API information, detect API changes between different versions of each website, organize the changed API information, and generate a change report.

[0011] S4. The identified API information is structured to generate an API directory.

[0012] Furthermore, the specific steps of step S1 are as follows:

[0013] S11. Designate each type of website as a target website and use a web crawler to access the target websites and capture page content;

[0014] S12. Perform HTML analysis and parsing on the crawled page content to extract relevant API information;

[0015] S13. Extract and analyze the dynamic content of JavaScript code to identify API calls;

[0016] S14. Monitor and record network requests between the browser and the server.

[0017] S15. Save all collected data as API call data to a database or file system;

[0018] S16. Extract features from the API call data to obtain API call features.

[0019] Furthermore, step S16 is detailed as follows:

[0020] S161. Clean the collected API call data to remove data with relevance below the threshold and noisy data;

[0021] S162. Use regular expressions and natural language processing algorithms to identify URL patterns, HTTP methods, and request and response formats as characteristics of API calls;

[0022] S163. The extracted API call features are stored in a structured manner in a database or file system.

[0023] Furthermore, the URL pattern includes the URL path of the API and the query parameters of the API;

[0024] The HTTP methods include GET, POST, PUT, and DELETE methods used for API calls;

[0025] The request and response formats include the formats of request parameters, request headers, and response files.

[0026] Furthermore, the specific steps of step S2 are as follows:

[0027] S21. Determine if the model update cycle has been reached;

[0028] If so, proceed to step S22;

[0029] If not, use the existing trained machine learning model and proceed to step S3;

[0030] S22. Load API call characteristics from the database or file system and set the labels;

[0031] S23. Generate a dataset from the API call features and their labels, and divide the dataset into a training set and a validation set;

[0032] S24. Use machine learning algorithms to train the training set to obtain an adaptive machine learning model that can recognize API calls;

[0033] S25. Calculate evaluation metrics for the machine learning model using the validation set, evaluate the performance of the machine learning model using the evaluation metrics, and iteratively train and optimize the machine learning model based on the evaluation results.

[0034] S26. Replace the original machine learning model with the optimized machine learning model and save it.

[0035] Furthermore, the specific steps of step S3 are as follows:

[0036] S31. Use the new or updated website as the website to be tested, collect API call data of the website to be tested, and extract API call features;

[0037] S32. Load the saved machine learning model, input the API call features of the website under test into the loaded machine learning model, identify the API interface and API interface information of the current version of the website under test, and save them to the database;

[0038] S33. Retrieve the API interface of the previous version of the website under test from the database;

[0039] S34. Use set operation algorithms to compare the API interfaces of the two versions of the website under test, mark the API interfaces that exist in the current version but do not exist in the previous version as newly added API interfaces, and mark the API interfaces that exist in the previous version but do not exist in the current version as removed API interfaces.

[0040] S35. Organize information on newly added and removed API interfaces and generate a change report.

[0041] Furthermore, the specific steps of step S4 are as follows:

[0042] S41. Load the current API interface information of the website under test;

[0043] S42. Load the API interface information for the current and previous versions of the website under test, including any additions and deletions.

[0044] S43. Integrate the current API interface information of the website under test with the newly added and removed API interface information, generate a structured API directory based on the integration result, and mark the newly added and removed API interfaces in the API directory;

[0045] S44. Export the generated API directory as a file in a preset format.

[0046] Secondly, the present invention provides an API detection system that adapts to changes in website structure, comprising:

[0047] The data acquisition and feature extraction module is used to collect API call data from various types of websites and extract API call features from the API call data.

[0048] The adaptive model training module is used to build machine learning models. It uses extracted API call features to train the machine learning models, learns the API recognition patterns of different websites, and automatically adjusts the recognition strategies in the machine learning models.

[0049] The API detection module is used to identify APIs of new or updated websites using trained machine learning models, generate API information, detect API changes between different versions of each website, organize the changed API information, and generate change reports.

[0050] The API directory generation module is used to perform structured processing on the identified API information and generate an API directory.

[0051] Thirdly, the present invention provides an electronic device, including a processor and a memory;

[0052] The memory is used to store a computer program, and the processor is used to retrieve and run the computer program from the memory, causing the electronic device to perform the method described in the first aspect above.

[0053] Fourthly, the present invention provides an electronic device, including a processor and a memory;

[0054] The memory is used to store a computer program, and the processor is used to retrieve and run the computer program from the memory, causing the electronic device to perform the method described in the first aspect above.

[0055] The beneficial effects of this invention are as follows:

[0056] The API detection method, system, device, and medium for adaptive website structure changes provided by this invention can not only automatically adjust the identification strategy to adapt to the structure and coding habits of different websites, but also automatically filter out newly added or removed API interfaces during software iteration and updates, thereby improving the accuracy of API identification and the efficiency of maintenance.

[0057] Furthermore, the design principle of this invention is reliable, the structure is simple, and it has a very wide range of application prospects.

[0058] Therefore, it is evident that the present invention has outstanding substantive features and significant progress compared with the prior art, and the beneficial effects of its implementation are also obvious. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a flowchart illustrating the API detection method for adaptive website structure changes according to the present invention.

[0061] Figure 2 This is a schematic diagram of the API detection system for adaptive website structure changes according to the present invention. Detailed Implementation

[0062] The API detection method for adaptive website structure changes can be applied to one or more electronic devices. The electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0063] Electronic devices can be any electronic product that allows for human-computer interaction, such as personal computers, tablets, smartphones, personal digital assistants (PDAs), and interactive network television (IPTV). The networks in which electronic devices operate include, but are not limited to, the Internet, wide area networks (WANs), metropolitan area networks (MANs), local area networks (LANs), and virtual private networks (VPNs).

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0065] Please see Figure 1 The diagram illustrates a specific implementation of an API detection method that adapts to changes in website structure, comprising the following steps:

[0066] S1. Collect API call data from various types of websites and extract API call characteristics from the API call data;

[0067] S2. Build a machine learning model, train the machine learning model using the extracted API call features, learn the API recognition patterns of different websites, and automatically adjust the recognition strategy in the machine learning model;

[0068] S3. Use the trained machine learning model to identify APIs for new or updated websites, generate API information, detect API changes between different versions of each website, organize the changed API information, and generate a change report.

[0069] S4. The identified API information is structured to generate an API directory.

[0070] In this embodiment, step S1 is specifically as follows:

[0071] S11. Designate each type of website as a target website and use a web crawler to access the target websites and capture page content;

[0072] S12. Perform HTML analysis and parsing on the crawled page content to extract relevant API information;

[0073] S13. Extract and analyze the dynamic content of JavaScript code to identify API calls;

[0074] S14. Monitor and record network requests between the browser and the server.

[0075] S15. Save all collected data as API call data to a database or file system;

[0076] S16. Extract features from the API call data to obtain API call features; the specific steps of step S16 are as follows:

[0077] S161. Clean the collected API call data to remove data with relevance below the threshold and noisy data;

[0078] S162. Use regular expressions and natural language processing algorithms to identify URL patterns, HTTP methods, and request and response formats as characteristics of API calls; the URL patterns include the API's URL path and the API's query parameters;

[0079] The HTTP methods include GET, POST, PUT, and DELETE methods used for API calls;

[0080] Request and response formats include the format of request parameters, request headers, and response files; response file formats include JSON and XML formats.

[0081] S163. The extracted API call features are stored in a structured manner in a database or file system;

[0082] The specific steps of step S2 are as follows:

[0083] S21. Determine if the model update cycle has been reached;

[0084] If so, proceed to step S22;

[0085] If not, use the existing trained machine learning model and proceed to step S3;

[0086] S22. Load API call characteristics from the database or file system and set the labels;

[0087] For example, 1 represents an API call, and 0 represents a non-API call;

[0088] S23. Generate a dataset from the API call features and their labels, and divide the dataset into a training set and a validation set;

[0089] For example, the dataset can be divided into a training set and a validation set in an 8:2 ratio;

[0090] S24. Use machine learning algorithms to train the training set to obtain an adaptive machine learning model that can recognize API calls;

[0091] Machine learning algorithms can be trained on API call features using any of the following methods: decision trees, random forests, or deep learning models. Among them, the random forest algorithm is an ensemble learning method based on multiple decision trees, which can effectively handle high-dimensional data and prevent overfitting. The following explanation will use the random forest algorithm as an example of choosing a machine learning algorithm.

[0092] Configure the parameters for the random forest algorithm:

[0093] The number of decision trees, n_estimators, can be set to 100, for example, to build 100 decision trees;

[0094] The maximum depth, max_depth, can be set to None, indicating that the maximum depth of the tree is unlimited, so as to fully learn the data features;

[0095] The minimum number of splittable samples, min_samples_split, can be set to 2, meaning that each node must have at least two samples before it can continue splitting.

[0096] Using data and labels from the training set, a random forest model is trained. During the training process, multiple decision trees are constructed by randomly selecting subsets of data and features multiple times.

[0097] During training, the random forest model gradually learns the relationship between features and labels, with each decision tree trained independently;

[0098] For each decision tree, the same number of samples as the original training set are randomly drawn from the training set with replacement as the training data for that decision tree.

[0099] At each node of the decision tree, a subset of features are randomly selected from all features, and the best splitting method is chosen based on these selected features.

[0100] Each decision tree grows independently to the maximum depth max_depth, or until a stopping condition is met, such as when the number of samples contained in a node is less than the minimum number of splittable samples min_samples_split.

[0101] S25. Calculate evaluation metrics for the machine learning model using the validation set, evaluate the performance of the machine learning model using the evaluation metrics, and iteratively train and optimize the machine learning model based on the evaluation results.

[0102] For example, when new data in the validation set is used as input samples, each decision tree will give a classification result, and the final classification result is obtained by majority voting or averaging the results of all decision trees;

[0103] The performance of the random forest model is evaluated by calculating accuracy, recall, and F1 score using the validation set.

[0104] Based on the evaluation results, parameters such as the number of decision trees n_estimators, the maximum depth max_depth, and the minimum number of splittable samples min_samples_split were tuned to improve the model's generalization ability, and the stability and robustness of the model were further verified by using the K-fold cross-validation method.

[0105] S26. Replace the original machine learning model with the optimized machine learning model and save it;

[0106] For example, the trained random forest model can be saved as a reusable file (such as a Pickle file) for easy loading and application later;

[0107] The trained random forest model can then be used on new or updated websites to identify API call features in real time, thereby improving identification efficiency and accuracy.

[0108] The specific steps of step S3 are as follows:

[0109] S31. Use the new or updated website as the website to be tested, collect API call data of the website to be tested, and extract API call features;

[0110] S32. Load the saved machine learning model, input the API call features of the website under test into the loaded machine learning model, identify the API interface and API interface information of the current version of the website under test, and save them to the database;

[0111] S33. Retrieve the API interface of the previous version of the website under test from the database;

[0112] S34. Use set operation algorithms to compare the API interfaces of the two versions of the website under test, mark the API interfaces that exist in the current version but do not exist in the previous version as newly added API interfaces, and mark the API interfaces that exist in the previous version but do not exist in the current version as removed API interfaces.

[0113] S35. Organize information on newly added and removed API interfaces and generate a change report;

[0114] The specific steps of step S4 are as follows:

[0115] S41. Load the current API interface information of the website under test;

[0116] S42. Load the API interface information for the current and previous versions of the website under test, including any additions and deletions.

[0117] S43. Integrate the current API interface information of the website under test with the newly added and removed API interface information, generate a structured API directory based on the integration result, and mark the newly added and removed API interfaces in the API directory;

[0118] S44. Export the generated API directory as a file in a preset format.

[0119] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0120] The following are embodiments of the API detection system for adaptive website structure changes provided in this disclosure. This device and the API detection method for adaptive website structure changes in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the API detection system for adaptive website structure changes, please refer to the embodiments of the API detection method for adaptive website structure changes described above.

[0121] like Figure 2 As shown, the system includes:

[0122] The data acquisition and feature extraction module is used to collect API call data from various types of websites and extract API call features from the API call data.

[0123] The adaptive model training module is used to build machine learning models. It uses extracted API call features to train the machine learning models, learns the API recognition patterns of different websites, and automatically adjusts the recognition strategies in the machine learning models.

[0124] The API detection module is used to identify APIs of new or updated websites using trained machine learning models, generate API information, detect API changes between different versions of each website, organize the changed API information, and generate change reports.

[0125] The API directory generation module is used to perform structured processing on the identified API information and generate an API directory.

[0126] The API detection method for adaptive website structure changes provided by this invention comprises modules and algorithm steps of various examples described in conjunction with the embodiments disclosed herein. These modules and steps can be implemented using electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0127] Those skilled in the art will understand that the API detection method for adaptive website structure changes provided by this invention can be implemented as a system, method, or program product. Therefore, the various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0128] In a non-transitory computer-readable storage medium storing an API detection method for adaptive website structure changes, the storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0129] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. A method for detecting API changes that adapt to changes in website structure, characterized in that, Includes the following steps: S1. Collect API call data from various types of websites and extract API call characteristics from the API call data; S2. Build a machine learning model, train the machine learning model using the extracted API call features, learn the API recognition patterns of different websites, and automatically adjust the recognition strategy in the machine learning model; S3. Use the trained machine learning model to identify APIs for new or updated websites, generate API information, detect API changes between different versions of each website, organize the changed API information, and generate a change report. S4. The identified API information is processed in a structured manner to generate an API directory; The specific steps of step S1 are as follows: S11. Designate each type of website as a target website and use a web crawler to access the target websites and capture page content; S12. Perform HTML analysis and parsing on the crawled page content to extract relevant API information; S13. Extract and analyze the dynamic content of JavaScript code to identify API calls; S14. Monitor and record network requests between the browser and the server. S15. Save all collected data as API call data to a database or file system; S16. Extract features from the API call data to obtain API call features; The specific steps of step S3 are as follows: S31. Use the new or updated website as the website to be tested, collect API call data of the website to be tested, and extract API call features; S32. Load the saved machine learning model, input the API call features of the website under test into the loaded machine learning model, identify the API interface and API interface information of the current version of the website under test, and save them to the database; S33. Retrieve the API interface of the previous version of the website under test from the database; S34. Use set operation algorithms to compare the API interfaces of the two versions of the website under test, mark the API interfaces that exist in the current version but do not exist in the previous version as newly added API interfaces, and mark the API interfaces that exist in the previous version but do not exist in the current version as removed API interfaces. S35. Organize information on newly added and removed API interfaces and generate a change report.

2. The API detection method for adaptive website structure changes as described in claim 1, characterized in that, The specific steps of step S16 are as follows: S161. Clean the collected API call data to remove data with relevance below the threshold and noisy data; S162. Use regular expressions and natural language processing algorithms to identify URL patterns, HTTP methods, and request and response formats as characteristics of API calls; S163. The extracted API call features are stored in a structured manner in a database or file system.

3. The API detection method for adaptive website structure changes as described in claim 2, characterized in that, The URL pattern includes the API's URL path and the API's query parameters; The HTTP methods include GET, POST, PUT, and DELETE methods used for API calls; The request and response formats include the formats of request parameters, request headers, and response files.

4. The API detection method for adaptive website structure changes as described in claim 1, characterized in that, The specific steps of step S2 are as follows: S21. Determine if the model update cycle has been reached; If so, proceed to step S22; If not, use the existing trained machine learning model and proceed to step S3; S22. Load API call characteristics from the database or file system and set the labels; S23. Generate a dataset from the API call features and their labels, and divide the dataset into a training set and a validation set; S24. Use machine learning algorithms to train the training set to obtain an adaptive machine learning model that can recognize API calls; S25. Calculate evaluation metrics for the machine learning model using the validation set, evaluate the performance of the machine learning model using the evaluation metrics, and iteratively train and optimize the machine learning model based on the evaluation results. S26. Replace the original machine learning model with the optimized machine learning model and save it.

5. The API detection method for adaptive website structure changes as described in claim 4, characterized in that, The specific steps of step S4 are as follows: S41. Load the current API interface information of the website under test; S42. Load the API interface information for the current version and the previous version of the website under test, including any additions and deletions; S43. Integrate the current API interface information of the website under test with the newly added and removed API interface information, generate a structured API directory based on the integration result, and mark the newly added and removed API interfaces in the API directory; S44. Export the generated API directory as a file in a preset format.

6. An API detection system that adapts to changes in website structure, used to implement the method described in any one of claims 1-5, characterized in that, include: The data acquisition and feature extraction module is used to collect API call data from various types of websites and extract API call features from the API call data. The adaptive model training module is used to build machine learning models. It uses extracted API call features to train the machine learning models, learns the API recognition patterns of different websites, and automatically adjusts the recognition strategies in the machine learning models. The API detection module is used to identify APIs of new or updated websites using trained machine learning models, generate API information, detect API changes between different versions of each website, organize the changed API information, and generate change reports. The API directory generation module is used to perform structured processing on the identified API information and generate an API directory.

7. An electronic device, characterized in that, Including processor and memory; The memory is used to store a computer program, and the processor is used to retrieve and run the computer program from the memory, causing the electronic device to perform the method described in any one of claims 1-5.

8. A storage medium, characterized in that, The storage medium stores instructions that, when run on a computer, cause the computer to perform the method described in any one of claims 1-5.

Images