API (Application Program Interface) detection method, system and equipment adaptive to website structure change and medium

Abstract

The invention provides an API (Application Program Interface) detection method, system and equipment adaptive to website structure change and a medium, and belongs to the technical field of Internet. The method comprises the following steps: collecting API calling data from various types of websites, and extracting API calling characteristics; constructing a machine learning model, training the machine learning model by using the extracted API calling features, and automatically adjusting a recognition strategy in the machine learning model; a machine learning model is used for carrying out API identification on the new website or the updated website, API information is generated, API changes between different versions of each website are detected, the changed API information is arranged, and a change report is generated; and performing structured processing on the identified API information to generate an API directory. The method adapts to the structures and coding habits of different websites to automatically adjust the identification strategy, can automatically screen out newly added or reduced APIs, and improves the API identification accuracy and maintenance efficiency.

Description

Background Art

[0002] API is the abbreviation of Application Programming Interface, application programming interface.

[0003] API plays a more important role in the process of software development and maintenance. It is the hub for effective communication and data exchange between different software systems or components. With the rapid development of Internet technology, especially in the context of cloud computing, microservice architecture and CI / CD, software systems need to be updated frequently, including the addition, modification or deletion of APIs. In the process of software iteration, changes in APIs are a common phenomenon. These changes may be driven by changes in business needs, updates to the technology stack, performance optimization or security improvements. However, these changes bring the following problems: First, poor compatibility. API changes may affect applications and systems that rely on these APIs, resulting in functional failures or system crashes. Second, maintenance is difficult. Developers need to track API changes to update the corresponding application code and documents. This process is often manual and error-prone. Third, update delays. In large systems, API changes may need to be updated simultaneously in multiple components and services, which may lead to update delays and deployment delays.

[0004] Traditional API identification and tracking methods mainly rely on manually maintained documents or static code analysis techniques, which often cannot effectively cope with rapidly changing requirements. For example, manual documents may not be updated immediately, and static code analysis may not accurately identify APIs that are dynamically generated or modified at runtime. In addition, traditional methods have difficulty automatically identifying and adapting to changes in website or application structure, which limits their effectiveness in modern rapid iteration environments.

[0005] This is a shortcoming of the prior art. Therefore, in view of the above-mentioned defects in the prior art, it is very necessary to provide an API detection method, system, device and medium that are adaptive to website structure changes. Summary of the invention

[0006] In view of the defects that the above-mentioned traditional API identification and tracking cannot effectively respond to rapidly changing needs, and traditional methods are difficult to automatically identify and adapt to changes in website or application structure, the present invention provides an API detection method, system, device and medium that are adaptive to website structure changes to solve the above-mentioned technical problems.

[0007] In a first aspect, the present invention provides an API detection method for adaptive website structure changes, comprising the following steps: S1. Collect API call data from various types of websites and extract API call features from the API call data; S2. Build a machine learning model, use the extracted API call features to train the machine learning model, learn the API recognition rules of different websites, and automatically adjust the recognition strategy in the machine learning model; S3. Use the trained machine learning model to identify APIs of new or updated websites, generate API information, detect API changes between different versions of each website, organize the changed API information, and generate a change report; S4. Structuralize the identified API information and generate an API directory. Furthermore, the specific steps of step S1 are as follows: S11. Take each type of website as the target website, and use a web crawler to access the target website and crawl the page content; S12. Perform HTML analysis and parsing on the captured page content to extract relevant API information; S13. Extract and analyze the dynamic content of the JavaScript code and identify API calls; S14. Monitor and record network requests from browsers and server supports; S15. Save all collected data as API call data in a database or file system; S16. Perform feature extraction on the API call data to obtain API call features.

[0008] Furthermore, the specific steps of step S16 are as follows: S161. Clean the collected API call data to remove data with correlation below a threshold and noise data; S162. Use regular expressions and natural language processing algorithms to identify URL patterns, HTTP methods, and request and response formats that are characteristic of API calls; S163. Save the extracted API call features in a structured manner in a database or file system.

[0009] Further, the URL pattern includes the URL path of the API and the query parameters of the API; The HTTP methods include GET, POST, PUT and DELETE methods used by API calls; The request and response formats include request parameters, request headers, and response file formats.

[0010] Furthermore, the specific steps of step S2 are as follows: S21. Determine whether the model update cycle has been reached; If yes, go to step S22; If not, use the previously trained machine learning model and proceed to step S3; S22. Load API call features from a database or file system and set labels; S23. Generate a data set using the API call features and their labels, and divide the data set into a training set and a validation set; S24. Use a machine learning algorithm to train the training set to obtain an adaptive machine learning model capable of identifying API calls; S25. Calculate evaluation indicators for the machine learning model through the validation set, use the evaluation indicators to evaluate the performance of the machine learning model, and iteratively train and optimize the machine learning model based on the evaluation results; S26. Replace the original machine learning model with the optimized machine learning model and save it.

[0011] Furthermore, the specific steps of step S3 are as follows: S31. Take the new website or updated website as the website to be tested, collect API call data of the website to be tested and extract API call features; S32. Load the saved machine learning model, and input the API call features of the website to be tested into the loaded machine learning model, identify the API interface and API interface information of the current version of the website to be tested, and save them to the database; S33. Obtain the API interface of the previous version of the website to be tested from the database; S34. Use a set operation algorithm to compare the API interfaces of the two versions of the tested website, mark the API interfaces that exist in the current version but not in the previous version as new API interfaces, and mark the API interfaces that exist in the previous version but not in the current version as reduced API interfaces; S35. Organize the newly added and reduced API interface information and generate a change report.

[0012] Furthermore, the specific steps of step S4 are as follows: S41. Load the current API interface information of the website to be tested; S42. Load the new and reduced API interface information of the current version and the previous version of the website to be tested; S43. Integrate the current API interface information of the tested website with the newly added and reduced API interface information, generate a structured API directory based on the integration results, and annotate the newly added and reduced API interfaces in the API directory; S44. Export the generated API directory to a file in a preset format.

[0013] In a second aspect, the present invention provides an API detection system for adaptive website structure changes, comprising: The data collection and feature extraction module is used to collect API call data from various types of websites and extract API call features from the API call data; The adaptive model training module is used to build a machine learning model, train the machine learning model using the extracted API call features, learn the API recognition rules of different websites, and automatically adjust the recognition strategy in the machine learning model; The API detection module is used to use the trained machine learning model to identify the APIs of new or updated websites, generate API information, detect API changes between different versions of each website, organize the changed API information, and generate change reports; The API directory generation module is used to structure the identified API information and generate an API directory.

[0014] In a third aspect, the present invention provides an electronic device, including a processor and a memory; The memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the electronic device executes the method described in the first aspect above.

[0015] In a fourth aspect, the present invention provides an electronic device, comprising a processor and a memory; The memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the electronic device executes the method described in the first aspect above.

[0016] The beneficial effects of the present invention are: The API detection method, system, device and medium for adaptive website structure changes provided by the present invention can not only automatically adjust the recognition strategy to adapt to the structure and coding habits of different websites, but also automatically screen out new or reduced API interfaces during the software iteration update process, thereby improving the accuracy of API recognition and the efficiency of maintenance.

[0017] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect.

[0018] It can be seen that compared with the prior art, the present invention has outstanding substantive features and significant progress, and the beneficial effects of its implementation are also obvious. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 It is a flow chart of the API detection method for adaptive website structure changes of the present invention.

[0021] Figure 2 It is a schematic diagram of the API detection system for adaptive website structure changes of the present invention. DETAILED DESCRIPTION

[0022] The API detection method for adaptive website structure changes can be applied to one or more electronic devices, wherein the electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0023] An electronic device may be any electronic product that can interact with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), an interactive network television (IPTV), etc. The network in which the electronic device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] See also Figure 1 The following is a specific implementation of an API detection method for adaptive website structure changes, including the following steps: S1. Collect API call data from various types of websites and extract API call features from the API call data; S2. Build a machine learning model, use the extracted API call features to train the machine learning model, learn the API recognition rules of different websites, and automatically adjust the recognition strategy in the machine learning model; S3. Use the trained machine learning model to identify APIs of new or updated websites, generate API information, detect API changes between different versions of each website, organize the changed API information, and generate a change report; S4. Structuralize the identified API information and generate an API directory.

[0026] In this embodiment, the specific steps of step S1 are as follows: S11. Take each type of website as the target website, and use a web crawler to access the target website and crawl the page content; S12. Perform HTML analysis and parsing on the captured page content to extract relevant API information; S13. Extract and analyze the dynamic content of the JavaScript code and identify API calls; S14. Monitor and record network requests from browsers and server supports; S15. Save all collected data as API call data in a database or file system; S16. Extract features from the API call data to obtain API call features. The specific steps of step S16 are as follows: S161. Clean the collected API call data to remove data with correlation below a threshold and noise data; S162. Use regular expressions and natural language processing algorithms to identify URL patterns, HTTP methods, and request and response formats as API call features; the URL pattern includes the API URL path and API query parameters; The HTTP methods include GET, POST, PUT and DELETE methods used by API calls; The request and response formats include request parameters, request headers, and response file formats; response file formats include JSON and XML formats; S163. The extracted API call features are stored in a database or file system in a structured manner; The specific steps of step S2 are as follows: S21. Determine whether the model update cycle has been reached; If yes, go to step S22; If not, use the previously trained machine learning model and proceed to step S3; S22. Load API call features from a database or file system and set labels; For example, use 1 to indicate an API call and 0 to indicate a non-API call; S23. Generate a data set using the API call features and their labels, and divide the data set into a training set and a validation set; For example, the data set is divided into a training set and a validation set in a ratio of 8:2; S24. Use a machine learning algorithm to train the training set to obtain an adaptive machine learning model capable of identifying API calls; The machine learning algorithm can use any one of the decision trees, random forests or deep learning models to train the API call features. Among them, the random forest algorithm is an integrated learning method based on multiple decision trees, which can effectively process high-dimensional data and prevent overfitting. The following will take the random forest algorithm as an example to explain; Set the parameters of the random forest algorithm: The number of decision trees n_estimators can be set to 100, for example, to build 100 decision trees; The maximum depth max_depth can be set to None, for example, which means that the maximum depth of the tree is unlimited so as to fully learn the data features; The minimum splittable samples min_samples_split can be set to 2, for example, each node must have at least two samples to continue splitting; Use the data and labels in the training set to train the random forest model. During the training process, multiple decision trees are constructed by randomly selecting data subsets and feature subsets multiple times. During the training process, the random forest model gradually learns the relationship between features and labels, and each decision tree is trained independently; For each decision tree, the same number of samples as the original training set are randomly extracted with replacement from the training set as the training data of the decision tree; At each node of the decision tree, a subset of features is randomly selected from all features and then the best split is chosen based on these selected features; Each decision tree grows independently to the maximum depth max_depth, or until a stopping condition is met, such as the number of samples contained in the node is less than the minimum splittable sample min_samples_split; S25. Calculate evaluation indicators for the machine learning model through the validation set, use the evaluation indicators to evaluate the performance of the machine learning model, and iteratively train and optimize the machine learning model based on the evaluation results; For example, for new data in the validation set as input samples, each decision tree will give a classification result, and the final classification result is obtained by majority voting or averaging the results of all decision trees; Use the validation set to calculate the accuracy, recall, and F1 score to evaluate the performance of the random forest model; According to the evaluation results, we tuned the number of decision trees n_estimators, the maximum depth max_depth, the minimum splittable samples min_samples_split and other parameters to improve the generalization ability of the model, and used the K-fold cross-validation method to further verify the stability and robustness of the model. S26. Replace the original machine learning model with the optimized machine learning model and save it; For example, saving the trained random forest model as a reusable file (such as a Pickle file) is convenient for subsequent loading and application; The trained random forest model is subsequently used in new or updated websites to identify API call features in real time, thus improving recognition efficiency and accuracy. The specific steps of step S3 are as follows: S31. Take the new website or updated website as the website to be tested, collect API call data of the website to be tested and extract API call features; S32. Load the saved machine learning model, and input the API call features of the website to be tested into the loaded machine learning model, identify the API interface and API interface information of the current version of the website to be tested, and save them to the database; S33. Obtain the API interface of the previous version of the website to be tested from the database; S34. Use a set operation algorithm to compare the API interfaces of the two versions of the tested website, mark the API interfaces that exist in the current version but not in the previous version as new API interfaces, and mark the API interfaces that exist in the previous version but not in the current version as reduced API interfaces; S35. Organize the newly added and reduced API interface information and generate a change report; The specific steps of step S4 are as follows: S41. Load the current API interface information of the website to be tested; S42. Load the new and reduced API interface information of the current version and the previous version of the website to be tested; S43. Integrate the current API interface information of the tested website with the newly added and reduced API interface information, generate a structured API directory based on the integration results, and annotate the newly added and reduced API interfaces in the API directory; S44. Export the generated API directory to a file in a preset format.

[0027] It should be understood that the order of execution of the steps in the above embodiment does not mean that the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0028] The following is an embodiment of the API detection system for adaptive website structure changes provided by the embodiments of the present disclosure. The device and the API detection method for adaptive website structure changes in the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the API detection system for adaptive website structure changes, please refer to the embodiments of the API detection method for adaptive website structure changes mentioned above.

[0029] like Figure 2 As shown, the system includes: The data collection and feature extraction module is used to collect API call data from various types of websites and extract API call features from the API call data; The adaptive model training module is used to build a machine learning model, train the machine learning model using the extracted API call features, learn the API recognition rules of different websites, and automatically adjust the recognition strategy in the machine learning model; The API detection module is used to use the trained machine learning model to identify the APIs of new or updated websites, generate API information, detect API changes between different versions of each website, organize the changed API information, and generate change reports; The API directory generation module is used to structure the identified API information and generate an API directory.

[0030] The API detection method for adaptive website structure changes provided by the present invention is a module and algorithm step of each example described in combination with the embodiments disclosed herein, and can be implemented by electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0031] Those skilled in the art will appreciate that the various aspects of the method for detecting adaptive website structure changes provided by the present invention can be implemented as a system, method or program product. Therefore, various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which can be collectively referred to as "circuit", "module" or "system" here.

[0032] In the non-transitory computer-readable storage medium storing the API detection method for adaptive website structure changes, it can be but not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0033] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person of ordinary skill in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.

Claims

1. An API detection method for adaptive website structure changes, characterized in that: The steps include: S1. Collect API call data from various types of websites and extract API call features from the API call data; S2. Build a machine learning model, use the extracted API call features to train the machine learning model, learn the API recognition rules of different websites, and automatically adjust the recognition strategy in the machine learning model; S3. Use the trained machine learning model to identify APIs of new or updated websites, generate API information, detect API changes between different versions of each website, organize the changed API information, and generate a change report; S4. Structuralize the identified API information and generate an API directory.

2. The API detection method for adaptive website structure changes according to claim 1, characterized in that: The specific steps of step S1 are as follows: S11. Take each type of website as the target website, and use a web crawler to access the target website and crawl the page content; S12. Perform HTML analysis and parsing on the captured page content to extract relevant API information; S13. Extract and analyze the dynamic content of the JavaScript code and identify API calls; S14. Monitor and record network requests from browsers and server supports; S15. Save all collected data as API call data in a database or file system; S16. Perform feature extraction on the API call data to obtain API call features.

3. The API detection method for adaptive website structure changes according to claim 2, characterized in that: The specific steps of step S16 are as follows: S161. Clean the collected API call data to remove data with correlation below a threshold and noise data; S162. Use regular expressions and natural language processing algorithms to identify URL patterns, HTTP methods, and request and response formats that are characteristic of API calls; S163. Save the extracted API call features in a structured manner in a database or file system.

4. The API detection method for adaptive website structure changes according to claim 3, characterized in that: The URL pattern includes the API URL path and the API query parameters; The HTTP methods include GET, POST, PUT and DELETE methods used by API calls; The request and response formats include request parameters, request headers, and response file formats.

5. The API detection method for adaptive website structure changes according to claim 2, characterized in that: The specific steps of step S2 are as follows: S21. Determine whether the model update cycle has been reached; If yes, go to step S22; If not, use the previously trained machine learning model and proceed to step S3; S22. Load API call features from a database or file system and set labels; S23. Generate a data set using the API call features and their labels, and divide the data set into a training set and a validation set; S24. Use a machine learning algorithm to train the training set to obtain an adaptive machine learning model capable of identifying API calls; S25. Calculate evaluation indicators for the machine learning model through the validation set, use the evaluation indicators to evaluate the performance of the machine learning model, and iteratively train and optimize the machine learning model based on the evaluation results; S26. Replace the original machine learning model with the optimized machine learning model and save it.

6. The API detection method for adaptive website structure changes according to claim 5, characterized in that: The specific steps of step S3 are as follows: S31. Take the new website or updated website as the website to be tested, collect API call data of the website to be tested and extract API call features; S32. Load the saved machine learning model, and input the API call features of the website to be tested into the loaded machine learning model, identify the API interface and API interface information of the current version of the website to be tested, and save them to the database; S33. Obtain the API interface of the previous version of the website to be tested from the database; S34. Use a set operation algorithm to compare the API interfaces of the two versions of the tested website, mark the API interfaces that exist in the current version but not in the previous version as new API interfaces, and mark the API interfaces that exist in the previous version but not in the current version as reduced API interfaces; S35. Organize the newly added and reduced API interface information and generate a change report.

7. The API detection method for adaptive website structure changes according to claim 6, characterized in that: The specific steps of step S4 are as follows: S41. Load the current API interface information of the website to be tested; S42. Load the new and reduced API interface information of the current version and the previous version of the website to be tested; S43. Integrate the current API interface information of the tested website with the newly added and reduced API interface information, generate a structured API directory based on the integration results, and annotate the newly added and reduced API interfaces in the API directory; S44. Export the generated API directory to a file in a preset format.

8. An API detection system that is adaptive to website structure changes, characterized in that: include: The data collection and feature extraction module is used to collect API call data from various types of websites and extract API call features from the API call data; The adaptive model training module is used to build a machine learning model, train the machine learning model using the extracted API call features, learn the API recognition rules of different websites, and automatically adjust the recognition strategy in the machine learning model; The API detection module is used to use the trained machine learning model to identify the APIs of new or updated websites, generate API information, detect API changes between different versions of each website, organize the changed API information, and generate change reports; The API directory generation module is used to structure the identified API information and generate an API directory.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the electronic device executes the method described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method described in any one of claims 1 to 7.

Images