Data classification and grading method and device, electronic equipment and computer storage medium

Abstract

The invention discloses a data classification and grading method and device, a storage medium and electronic equipment. The method comprises the following steps: acquiring application program interface (API) data to be processed; sampling and filtering the to-be-processed API data according to the user identification information and the interface identification information by taking a preset time period as a period to obtain sampled and filtered API data; and classifying and grading the sampled and filtered API data, and storing the classified and graded API data in a first data index database. In the whole data classification and grading process, before the API data is classified and graded, periodic sampling and filtering are performed on the to-be-processed API data according to the user identification information and the interface identification information, so that repeated interface calling data when the same user calls the same interface within the preset time period is effectively removed, and the data processing efficiency is improved. According to the method, the data processing amount when the system classifies and grades the API data is reduced, the classification and grading efficiency of the API data is improved, and then the security risk detection efficiency of the API data is improved.

Description

Technical Field

[0001] The present application relates to the field of data security technology, and in particular to a data classification and grading method, device, electronic device and computer storage medium. Background Art

[0002] With the popularity of mobile applications and the rise of cloud computing, the number of APIs (Application Programming Interface) has increased dramatically, and the data security risks associated with them have also increased simultaneously. In order to deal with possible attack threats in API data (such as data leakage, unauthorized access, injection attacks, etc.), it is necessary to perform sensitive discovery and classification of API data in order to better understand the functions, usage, data exposure and potential security risks of APIs.

[0003] Due to the huge amount of API data and the complex data content, traditional data classification and grading methods are limited by classification and grading performance. The computing resources and time costs consumed during classification and grading are too large, and there is a problem of insufficient efficiency in data classification and grading, which in turn affects the efficiency of security risk detection for API data. Summary of the invention

[0004] The embodiments of the present application provide a data classification and grading method, device, electronic device and computer storage medium, which can efficiently classify and grade API data, thereby improving the efficiency of security risk detection of API data.

[0005] In a first aspect, an embodiment of the present application provides a data classification and grading method, including:

[0006] Get the API data to be processed;

[0007] Taking a preset time period as a cycle, sampling and filtering the API data to be processed according to the user identification information and the interface identification information to obtain the sampled and filtered API data;

[0008] The sampled and filtered API data is classified and graded, and the classified and graded API data is stored in the first data index library.

[0009] In a possible implementation, the API data to be processed includes multiple interface call data;

[0010] The API data to be processed is sampled and filtered according to the user identification information and the interface identification information in a preset time period, and the sampled and filtered API data is obtained, including:

[0011] Acquire multiple pieces of interface call data from the API data to be processed in a preset time period;

[0012] The interface call data having the same user identification information and interface identification information among the multiple interface call data are removed to obtain the sampled and filtered API data.

[0013] In a possible implementation, the sampled and filtered API data includes at least one piece of interface call data;

[0014] Classifying and grading the sampled and filtered API data, and storing the classified and graded API data in a first data index library, including:

[0015] Identify sensitive information contained in each interface call data;

[0016] According to the preset label configuration rules, label rules are matched on the sensitive information contained in each interface call data to obtain the labeled interface call data;

[0017] The labeled interface call data is stored in the first data index library.

[0018] In one possible implementation, the preset label configuration rules include one or more of API function level label configuration rules, API data exposure level label configuration rules, API status level label configuration rules, and API sensitivity level level label configuration rules.

[0019] In a possible implementation, obtaining application program interface (API) data to be processed includes:

[0020] Use the collection probe to collect traffic transmitted based on the Hypertext Transfer Protocol (HTTP) in the application environment;

[0021] Analyze the traffic and obtain traffic analysis data;

[0022] Perform intranet filtering, network segment filtering and interface filtering on the traffic analysis data to obtain the API data to be processed.

[0023] In a possible implementation, interface filtering includes:

[0024] Remove page static resources and binary file transfer data from traffic parsing data, and retain structured text data from traffic parsing data.

[0025] In a possible implementation, after obtaining the application program interface API data to be processed, before sampling and filtering the API data to be processed according to the user identification information and the interface identification information at a preset time period, the method further includes:

[0026] Based on the API data to be processed, obtain API detailed data; the API detailed data includes detailed information of multiple API calls;

[0027] The detailed information of the multiple API calls are classified and stored in the second data index library.

[0028] In a possible implementation, after sampling and filtering the API data to be processed according to the user identification information and the interface identification information with a preset time period as a cycle and obtaining the sampled and filtered API data, the method further includes:

[0029] Store the sampled and filtered API data into the message middleware;

[0030] Before classifying and grading the sampled and filtered API data and storing the classified and graded API data in the first data index library, the method further includes:

[0031] Get sampled and filtered API data from the message middleware.

[0032] In a second aspect, an embodiment of the present application provides a data classification and grading device, including:

[0033] A data acquisition module is used to acquire the application program interface API data to be processed;

[0034] A sampling and filtering module is used to sample and filter the API data to be processed according to the user identification information and the interface identification information in a preset time period to obtain the API data after sampling and filtering;

[0035] The classification and grading module is used to classify and grade the API data after sampling and filtering, and store the classified and graded API data in the first data index library.

[0036] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps provided in the first aspect of the embodiment of the present application.

[0037] In a fourth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the method steps provided in the first aspect of the embodiment of the present application.

[0038] The above-mentioned data classification and grading method, device, electronic device and computer storage medium obtain the application program interface API data to be processed; sample and filter the API data to be processed according to the user identification information and the interface identification information in a preset time period to obtain the sampled and filtered API data; classify and grade the sampled and filtered API data, and store the classified and graded API data in the first data index library. During the entire data classification and grading process, before classifying and grading the API data, the API data to be processed is periodically sampled and filtered according to the user identification information and the interface identification information, which effectively removes the repeated interface call data when the same user calls the same interface within the preset time period, reduces the amount of data processing when the system classifies and grades the API data, improves the efficiency of API data classification and grading, and further improves the efficiency of security risk detection of API data. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A schematic diagram of an application environment of a data classification and grading method provided by an exemplary embodiment of the present application;

[0041] Figure 2 A flowchart of a data classification and grading method provided for an exemplary embodiment of the present application;

[0042] Figure 3 A schematic diagram of an architecture for identifying and classifying interface call data provided by an exemplary embodiment of the present application;

[0043] Figure 4 A schematic diagram showing the results of data classification and grading provided by an exemplary embodiment of the present application;

[0044] Figure 5 A flowchart of another data classification and grading method provided for an exemplary embodiment of the present application;

[0045] Figure 6 A schematic diagram of an architecture for filtering traffic parsing data provided by an exemplary embodiment of the present application;

[0046] Figure 7 A schematic diagram of a data sampling and filtering architecture provided for an exemplary embodiment of the present application;

[0047] Figure 8A schematic diagram of the architecture of another data classification and grading method provided for an exemplary embodiment of the present application;

[0048] Fig. 9 A schematic diagram of the structure of a data classification and grading device in one embodiment;

[0049] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] In the description of the present application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0052] The data classification and grading method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 10 communicates with the server 20 through the network. The data storage system can store the data that the server 20 needs to process. The data storage system can be integrated on the server 20, or it can be placed on the cloud or other network servers. The server 20 obtains the application program interface API data to be processed; samples and filters the API data to be processed according to the user identification information and the interface identification information in a preset time period to obtain the sampled and filtered API data; classifies and grades the sampled and filtered API data, and stores the classified and graded API data in the first data index library.

[0053] The terminal 10 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 20 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0054] In one embodiment, Figure 2 As shown, a data classification and grading method is provided, which is applied to Figure 1 The server 20 in the example is used as an example to illustrate, and the following steps are included:

[0055] S201: Acquire application programming interface (API) data to be processed.

[0056] Optionally, before sampling, filtering and classifying the API data, the server 20 will perform data standardization processing such as parsing and interface filtering on the API traffic collected in the application environment. The specific process of obtaining the application interface API data to be processed will be described in the following steps and will not be repeated here.

[0057] S202: sampling and filtering the API data to be processed according to the user identification information and the interface identification information in a preset time period to obtain the sampled and filtered API data.

[0058] The preset time period is a sampling period set manually, for example, it can be set to 1 hour, and can also be adjusted according to actual needs, for example, it can be adjusted to half an hour, 2 hours, etc. The smaller the preset time period is set, the higher the sampling frequency for API data, the stronger the timeliness and the higher the accuracy of the API data after sampling and filtering; but it also means that the data processing volume of the server 20 when classifying and grading API data increases. If the preset time period is set too small, it may lead to excessive computing resources and time costs consumed when classifying and grading API data; if the preset time period is set too large, it may lead to low timeliness and reduced accuracy of the API data obtained after sampling and filtering.

[0059] The above user identification information may be, but is not limited to, user name, user ID number, user mobile phone number, etc., and may be set and adjusted according to the actual business scenario. The above interface identification information may be, but is not limited to, interface address (including protocol, domain name, port and path), etc., and the interface identification information may uniquely distinguish an interface.

[0060] Optionally, the present application sets the preset time period to 1 hour, and uses a traffic sampling task engine in combination with user identification information (such as user name) and interface identification information (such as interface address) to sample and filter the API data to be processed. For multiple interface call data with the same user identification information and interface identification information within 1 hour, only one of the multiple interface call data is retained, thereby effectively removing the repeated interface call data when the same user calls the same interface within the preset time period, reducing the data processing volume and computing resources and time cost consumed when the server 20 subsequently classifies and grades the API data, thereby improving the efficiency of API data classification and grading, and thereby improving the efficiency of API data security risk detection.

[0061] S203: Classify and grade the API data after sampling and filtering, and store the classified and graded API data in the first data index library.

[0062] The first data index library may be, but is not limited to, an ES index library (Elasticsearch). The ES index library supports dynamic mapping, can automatically detect field types and create corresponding mappings, and simplifies index management. At the same time, it can store complex data structures, such as JSON documents, and adapt to diverse data models. In addition, it can also provide near real-time data analysis, which can quickly index new data and make it searchable within seconds.

[0063] See also Figure 3 The server 20 uses the classification and grading task engine integrated thereon to perform content recognition and classification on the sampled and filtered interface call data (including API requests and API responses). The interface call data sampled and filtered by the traffic sampling task engine is then labeled and stored in the first data index library for subsequent offline analysis such as API data security risk detection.

[0064] See also Figure 4, the server 20 can classify and grade APIs used in different scenarios such as websites, open platforms, applications (Application, APP), mini-programs and Internet of Things (IoT) devices from five levels: API format, API type, API sensitivity level, API data exposure surface and API status according to the preset tag configuration rules. Specifically, from the API format level, it can be divided into RESTful format API, SOAP format API, GraphQL format API or gRPC format API, etc.; from the API type level, it can be divided into data exposure API, login API, registration API, file download API, etc.; from the API sensitivity level level, it can be divided into high sensitivity API, medium sensitivity API, low sensitivity API, non-sensitive API, etc.; from the data exposure surface level, it can be divided into mobile phone number exposure API, ID card number exposure API, email exposure API, amount exposure API, company name exposure API, etc.; from the API status level, it can be divided into new API, active API, deactivated API or resurrection API, etc. By classifying and grading API data from the above levels, we can timely understand and manage API changes, verify and evaluate the corresponding API data, and ensure that the API list in the ES index library is synchronized with the actual situation, maintaining the timeliness and accuracy of the API list.

[0065] The above-mentioned data classification and grading method obtains the application program interface API data to be processed; samples and filters the API data to be processed according to the user identification information and the interface identification information in a preset time period to obtain the sampled and filtered API data; classifies and grades the sampled and filtered API data, and stores the classified and graded API data in the first data index library. During the entire data classification and grading process, before classifying and grading the API data, the API data to be processed is periodically sampled and filtered according to the user identification information and the interface identification information, which effectively removes the repeated interface call data when the same user calls the same interface within the preset time period, reduces the amount of data processing when the system classifies and grades the API data, improves the efficiency of API data classification and grading, and further improves the efficiency of API data security risk detection.

[0066] In one embodiment, Figure 5 As shown in the figure, another data classification and grading method is provided, which is applied to Figure 1 The server 20 in the example is used as an example to illustrate, and the following steps are included:

[0067] S501: Collecting traffic transmitted based on the Hypertext Transfer Protocol (HTTP) in an application environment through a collection probe.

[0068] Optionally, a collection probe (Agent) is deployed in the application's running environment. The collection probe has the ability to mirror traffic transmitted based on the Hypertext Transfer Protocol HTTP. It can capture complete API call data by copying all HTTP requests and responses in the application's running environment, so as to enable subsequent analysis and processing of the API data.

[0069] S502: Analyze the traffic flow to obtain traffic analysis data.

[0070] Optionally, the server 20 parses the collected traffic based on the HTTP protocol to obtain traffic parsing data. The traffic parsing data includes API request data including the request URL (Uniform Resource Locator), the request HTTP method, the request header, and the request body, as well as API response data including the response status code, the response header, and the response body. It is worth noting that in addition to API request data and API response data, which are API interface call data, the traffic parsing data also includes page static resources, binary file transfers, non-API requests (other types of network requests) and other data.

[0071] S503: Perform intranet filtering, network segment filtering and interface filtering on the traffic analysis data to obtain API data to be processed.

[0072] See also Figure 6 Before processing the API data, the server 20 filters the above-mentioned traffic parsing data based on HTTP transmission. The entire filtering process specifically includes intranet filtering, network segment filtering and interface filtering. Among them, intranet filtering is mainly to filter out API calls that are not allowed by the enterprise's internal network, and ensure that only those API calls related to the enterprise's internal network are analyzed. Through intranet filtering, irrelevant traffic from the external network can be prevented from entering the analysis system, reducing noise data, and improving the accuracy and efficiency of API traffic analysis. Network segment filtering refers to filtering traffic based on the network segment of the IP address, which can be to exclude the network segment to which a specific malicious IP address belongs, or to retain the traffic of a specific network segment allowed by the intranet. Network segment filtering helps to further refine traffic management and increase access control to sensitive APIs.

[0073] In one embodiment, interface filtering includes: removing page static resources and binary file transfer data in the traffic parsing data, and retaining structured text data in the traffic parsing data.

[0074] It is understandable that, since the traffic parsing data contains other data besides the API interface call data, the server 20 needs to filter the traffic parsing data before analyzing and processing the API data, remove the page static resources and text binary file transmission data in the traffic parsing data, and focus on the real API interface call data. The API data to be processed after the interface filtering is generally structured text data in the form of JSON format, and the API data to be processed includes multiple interface call data, each of which includes one or more of the request HTTP method, request URL, request header, request body, response status code, response header, and response body.

[0075] In this embodiment, the interface filtering technology ensures that when the API data is subsequently classified and graded, the analysis system can focus on the real interface call data, reduce the amount of unnecessary data processing during subsequent data classification and grading, and improve the efficiency of API data classification and grading.

[0076] S504: Based on the API data to be processed, obtain API detailed data; the API detailed data includes multiple API call detailed information.

[0077] Optionally, when processing large-scale API traffic data, the server 20 usually performs sampling and filtering on the data in order to balance performance and storage requirements to reduce the amount of data that needs to be processed and stored. However, sampling and filtering may cause incomplete data, thereby affecting the accuracy and comprehensiveness of data analysis. In order to avoid this problem, this embodiment obtains complete API detailed data through the traffic sampling task engine integrated on the server 20 before performing sampling and filtering, so that even if sampling and filtering are performed later, the original complete data is still available for emergency use.

[0078] Specifically, the traffic sampling task engine extracts the API call details corresponding to each API interface from the API data to be processed. The API call details corresponding to each API interface include but are not limited to the request method, request URL, request header, request body, response status code, response header, response body, response time, etc. These API call details are crucial to understanding the performance, security, and availability of the API interface in the original traffic.

[0079] In addition, the traffic sampling task engine will also perform root node information statistics, service information statistics, and client information statistics for all API data to be processed. Specifically, it includes statistics on the first access time, the most recent access time, the total number of assets, the total number of newly discovered assets, the total number of services, the total number of newly discovered services, the number of API interfaces, the number of clients, the number of unconfirmed customers, the number of confirmed customers, the number of blacklists, the number of whitelists, the number of visits, the number of failures, the reasons for failures, the number of error codes, the size of the requested traffic, the size of the traffic to be signed, the distribution of visits in the past 24 hours, and the top five API interfaces in the past 24 hours, etc., which are used to provide data portraits of different dimensions after classifying and grading the API data.

[0080] S505: Categorize and store the multiple API call detailed information into the second data index library.

[0081] Optionally, the traffic sampling task engine integrated on the server 20 obtains the API detailed data based on the API data to be processed, and then stores the obtained full amount of API detailed data, that is, the request method, request URL, request header, request body, response status code, response header, response body, response time and other API call details contained in each API interface to the ES index library for subsequent real-time search and analysis. The ES index library is a distributed search and data analysis engine that can provide near real-time data retrieval and can be horizontally expanded. It is very suitable for storing structured API call data. The use of the ES index library facilitates users to search and analyze a large number of API call details, thereby achieving efficient monitoring and management of API data. In this embodiment, the validity period of the API detailed data stored in the ES index library is set to seven days. By limiting the shelf life of the data, the demand for storage space can be reduced, and the related storage costs can be reduced, while improving the query speed and performance of new data.

[0082] It is worth noting that this embodiment only describes a preferred implementation mode of the present application. The validity period of the API detailed data is not limited to seven days, and can also be adjusted according to actual needs, such as five days, one month, etc. The specific duration of the validity period of the API detailed data is not limited in this embodiment of the present application.

[0083] In addition, the traffic sampling task engine will store the above-mentioned root node information, service information, and client information in the Redis library through the three keys of api_root, api_service, and api_client, and set the validity period of these three types of statistical data to long-term for subsequent generation of data portraits of different dimensions.

[0084] S506: Acquire multiple pieces of interface call data in the API data to be processed in a preset time period.

[0085] Optionally, the API data to be processed includes multiple interface call data, that is, a collection of all API requests and API responses received or processed by the server 20 within a period of time, and each interface call data includes one or more of the request HTTP method, request URL, request header, request body, response status code, response header, and response body. The traffic sampling task engine integrated on the server 20 obtains multiple interface call data in the API data to be processed with a preset time period as a cycle. Among them, the preset time period is a sampling period set manually, for example, it can be set to 1 hour, and can also be adjusted according to actual needs, for example, it can be adjusted to half an hour, 2 hours, etc. The smaller the preset time period is set, the higher the sampling frequency for the API data, the stronger the timeliness and the higher the accuracy of the API data after sampling and filtering; but at the same time, it means that the data processing volume of the server 20 when classifying and grading the API data increases. If the preset time period is set too small, it may cause the computing resources and time cost consumed when classifying and grading the API data to be too large; if the preset time period is set too large, it may cause the API data obtained after sampling and filtering to be not high in timeliness and reduce accuracy.

[0086] S507: Remove the interface calling data with the same user identification information and interface identification information from the multiple pieces of interface calling data to obtain sampled and filtered API data.

[0087] The sampled and filtered API data includes at least one piece of interface call data.

[0088] Optionally, the traffic sampling task engine integrated on the server 20 performs sampling and filtering on multiple interface call data contained in the API data to be processed based on the deduplication rules stored in the Redis library, with a preset time period as a cycle. The deduplication rules stored in the Redis library are specifically: only retain the API call requests issued when the same user of the same client accesses the same port on the same server. That is, the traffic sampling task engine combines the user identification information such as the user name contained in the API request, and the interface identification information such as the interface address contained in the request URL to perform sampling and filtering, remove the interface call data with the same user identification information and interface identification information in the multiple interface call data, and obtain the sampled and filtered API data including at least one interface call data.

[0089] See also Figure 7, assuming that the preset time period is 1 hour, a class of interface call data with the same user name and interface address are referred to by the same letter, and the interface call data referred to by different letters have different user names or different interface addresses. If the data before sampling in the first hour is: A1, A2, B1, B1, B1, B1, A1, A1, A1, A1, then the interface call data after sampling and filtering by the traffic sampling task engine is: A1, A2, B1. It can be understood that if sampling and filtering are not performed, the subsequent system needs to classify and grade the 11 interface call data, and after sampling, only the 3 interface call data A1, A2, and B1 need to be classified and graded. Similarly, if the data before sampling in the second hour is: A1, A1, A1, A1, B1, B1, B1, B1, B1, then the interface call data after sampling and filtering by the traffic sampling task engine is: A1, B1. That is, after executing the sampling and filtering in the second hour, only the two interface call data A1 and B1 need to be classified and graded.

[0090] S508: Store the sampled and filtered API data in the message middleware.

[0091] Optionally, this embodiment introduces the sampled and filtered API data into the Kafka message queue. As a distributed messaging system, Kafka provides high throughput, low latency, and persistent messaging services, and is suitable for processing large amounts of real-time data streams. By storing the sampled and filtered API data in the Kafka message queue, an efficient, reliable, and scalable data processing pipeline can be built to support a variety of application scenarios such as real-time analysis, batch processing jobs, and data integration.

[0092] S509: Obtain the sampled and filtered API data from the message middleware.

[0093] Optionally, the classification and grading task engine integrated on the server 20 obtains the sampled and filtered API data in the Kafka message queue through the Kafka Topic subscribed by the Kafka instance configured thereon, and the sampled and filtered API data includes at least one interface call data.

[0094] S510: Identify sensitive information contained in each interface call data.

[0095] Optionally, after obtaining the sampled and filtered API data, the classification and grading task engine identifies the sensitive information contained in each interface call data in the sampled and filtered API data. Specifically, the classification and grading task engine uses automated API identification technology to structure the API requests and API responses contained in each interface call data of the network, identifies the format of each interface call data, and applies predefined rules and regular expressions to identify parameters in each interface call data that may contain sensitive information, thereby performing sensitive discovery on the content of the API data.

[0096] In this embodiment, the classification and grading task engine continuously monitors and analyzes API interactions, identifies sensitive data contained in API requests and return content, and promptly updates the details of sensitive data exposure, thereby better understanding the API's functions, usage, data exposure, and potential security risks.

[0097] S511: Perform label rule matching on the sensitive information contained in each interface call data according to the preset label configuration rule to obtain each interface call data after labeling.

[0098] Optionally, the classification and grading task engine integrated on the server 20 loads preset label rules from the MySQL database in real time to classify and grade the sensitive information contained in each interface call data. Specifically, the classification and grading task engine matches the label rules of the sensitive information according to the preset label configuration rules to obtain the labeled interface call data.

[0099] In one embodiment, the preset label configuration rules include one or more of API function level label configuration rules, API data exposure level label configuration rules, API status level label configuration rules, and API sensitivity level level label configuration rules.

[0100] Specifically, the classification and grading task engine can classify and grade the sampled and filtered API data from the functional level, data exposure level, interaction mode level, sensitivity level, and status level according to the preset label configuration rules. From the functional level, API data types include login API, registration API, SMS verification code sending API, export API, file upload API, file download API, etc.; from the API data exposure level, API data types include data exposure API, data collection API, etc.; from the interaction mode level, API data types include service call API, human-machine access API, etc.; from the sensitivity level, API data levels include high sensitivity, medium sensitivity, low sensitivity, and non-sensitive; from the status level, API data status includes new API, active API, deactivated API, and resurrected API.

[0101] In this embodiment, based on the classification and grading task engine, the sensitive information contained in each interface call data is matched with label rules according to the preset label configuration rules, so that the changes of API can be understood and managed in a timely manner, and the corresponding API data can be verified and evaluated, thereby ensuring that the API list in the ES index library is synchronized with the actual situation, and maintaining the timeliness and accuracy of the API list.

[0102] S512: Storing the labeled interface call data in the first data index library.

[0103] Among them, the first data index library is the ES index library, which is different from the second data index library mentioned above that stores the full amount of API detail data. It can be understood that the classification and grading task engine stores the labeled interface call data in the ES index library, and sets the validity period of the labeled interface call data stored in the ES index library to long-term validity, so as to provide data portraits of different dimensions.

[0104] Optionally, this embodiment combines the labeled interface call data in the first data index library and the full API detail data in the second data index library to perform offline analysis such as API data security risk detection, and provides data portraits of different dimensions and information display and configuration. After classifying and grading the API data, this embodiment can provide API root node portraits, API asset and service portraits, API application portraits, API list and API details display, API activity display, API custom configuration, etc.

[0105] Specifically, the above API root node portrait includes: first access time, last access time, total number of assets, total number of newly discovered assets, total number of services, total number of newly discovered services, number of API interfaces, number of clients, number of unconfirmed customers, number of confirmed customers, sensitive classification, sensitivity level, cross-border access, API interface label display, black / white list number, number of visits, number of failures, reasons for failure, size of request / accepted traffic, and distribution of visits in the past 24 hours. The above API asset and service portrait includes: asset IP, port, service name, total number of API interfaces, number of successful interfaces, number of failures, sensitive classification, and sensitivity level. The above API application portrait includes: number of APIs, number of sensitive APIs, cross-border communication, API access trends in the past week, access distribution in the past 24 hours, the top five API interfaces in terms of interface access in the past 24 hours, percentage of sensitivity levels, the top five API interfaces in terms of client access times, the top five API interfaces in terms of sensitive types, and the top ten API interfaces in terms of user access to sensitive quantities. The above API list and API details display include: request method display, request URL display, request header display, request body display, response status code display, response header display, response body display, response time display, etc. The above API activity display includes new API display, active API display, inactive API display and resurrected API display. The above API custom configuration includes: custom API label, API interface merging configuration, API interface splitting configuration, API classification and grading sampling rate configuration, API access relationship learning period configuration, API user account identification configuration, etc.

[0106] See also Figure 8, which is a schematic diagram of the architecture corresponding to this embodiment. The traffic parsing data transmitted based on HTTP becomes the API data to be processed after intranet filtering, network segment filtering and interface filtering. Before performing sampling and filtering, the traffic sampling task engine obtains API detail data from the API data to be processed and stores it in the ES original traffic index library. The storage validity period of the API detail data is 7 days. At the same time, the traffic sampling task engine also counts the root node information, service information and client information based on the API data to be processed, and stores them in the Redis library through the three keys of api_root, api_service and api_client respectively. The validity period of these three types of statistical data is set to long-term for subsequent generation of data portraits of different dimensions. Then the traffic sampling task engine calls the deduplication rules stored in the Redis library, samples and filters the multiple interface call data included in the API data to be processed, and obtains the API data after sampling and filtering. The API data after sampling and filtering is introduced into the message middleware Kafka queue. The classification and grading task engine obtains the API data after sampling and filtering (i.e. deduplication) from the Kafka queue, and identifies the sensitive information contained in the API data after sampling and filtering. Then, the preset label rules are loaded from the MySQL database in real time to classify and grade the sensitive information contained in each interface call data. Finally, the classification and grading task engine stores the labeled interface call data, that is, the classified and graded API data, in the ES index library to perform offline analysis such as security risk detection in combination with the full amount of API detail data stored in the ES original traffic index library.

[0107] In this embodiment, before the API data to be processed is classified and graded, the full amount of API detailed data is stored in the first data index library in advance, which is convenient for subsequent data retrieval, analysis and list generation, and improves the efficiency of data analysis. Using message middleware such as Kafka, real-time data processing is realized, which is convenient for more in-depth data monitoring and data analysis. The filtering sampling mechanism based on the traffic sampling task engine reduces the amount of data stored and processed, avoids the processing of useless or duplicate data, improves system performance and resource utilization, and realizes more efficient data classification and grading. Based on the classification and grading task engine, sensitive information in the interface call data is identified and marked, and the label rule matching is performed on the sensitive information contained in each interface call data according to the preset label configuration rules, thereby providing a basis for the security risk detection of API data and realizing more refined data classification and grading. By storing the labeled interface call data in the first data index library, and combining the full amount of API detailed data stored in the first data index library for offline analysis such as API data security risk detection, it provides API root node portrait, API asset and service portrait, API application portrait, API list and API details display, API activity display, API custom configuration and other services, realizing more comprehensive data classification and grading.

[0108] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0109] Based on the invention concept of the above data classification and grading method, Fig. 9 As shown, the embodiment of the present application further provides a data classification and grading device 900 for implementing the above-mentioned data classification and grading method. The data classification and grading device 900 includes:

[0110] The data acquisition module 901 is used to acquire the application program interface API data to be processed;

[0111] The sampling and filtering module 902 is used to perform sampling and filtering on the API data to be processed according to the user identification information and the interface identification information at a preset time period to obtain the API data after sampling and filtering;

[0112] The classification and grading module 903 is used to classify and grade the API data after sampling and filtering, and store the classified and graded API data in the first data index library.

[0113] In one possible implementation, the API data to be processed includes multiple interface call data; the sampling and filtering module 902 is specifically used to obtain multiple interface call data in the API data to be processed with a preset time period as a cycle; remove the interface call data with the same user identification information and interface identification information in the multiple interface call data, and obtain the sampled and filtered API data.

[0114] In one possible implementation, the sampled and filtered API data includes at least one interface call data; the classification and grading module 903 is specifically used to identify the sensitive information contained in each interface call data; the sensitive information contained in each interface call data is matched with the label rules according to the preset label configuration rules to obtain the labeled interface call data; and the labeled interface call data is stored in the first data index library.

[0115] In one possible implementation, the preset label configuration rules include one or more of API function level label configuration rules, API data exposure level label configuration rules, API status level label configuration rules, and API sensitivity level level label configuration rules.

[0116] In one possible implementation, the data acquisition module 901 is specifically used to collect traffic transmitted based on the Hypertext Transfer Protocol HTTP in an application environment through a collection probe; parse the traffic to obtain traffic analysis data; perform intranet filtering, network segment filtering, and interface filtering on the traffic analysis data to obtain API data to be processed.

[0117] In a possible implementation, the data acquisition module 901 is specifically used to remove page static resources and binary file transmission data in the traffic parsing data, and retain the structured text data in the traffic parsing data.

[0118] In a possible implementation, the classification and grading device 900 also includes a data storage module for obtaining API detail data based on the API data to be processed; the API detail data includes multiple API call details; and the multiple API call details are classified and stored in the second data index library.

[0119] In a possible implementation, the data storage module is also used to sample and filter the API data to be processed based on user identification information and interface identification information at a preset time period, and after obtaining the sampled and filtered API data, store the sampled and filtered API data in the message middleware; classify and grade the sampled and filtered API data, and before storing the classified and graded API data in the first data index library, obtain the sampled and filtered API data from the message middleware.

[0120] Each module in the above-mentioned data classification and grading device 900 can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0121] The embodiment of the present application also provides an electronic device, which may be a server, and its internal structure diagram may be as follows: Fig.10 As shown. The electronic device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the electronic device is used to store duplicate removal rule data, label configuration rule data, full API detail data and labeled interface call data. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal through a network connection. The processor of the electronic device executes a computer program to implement a data classification and grading method.

[0122] Those skilled in the art will understand that Fig.10 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0123] In a possible implementation, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0124] Get the API data to be processed;

[0125] Taking a preset time period as a cycle, sampling and filtering the API data to be processed according to the user identification information and the interface identification information to obtain the sampled and filtered API data;

[0126] The sampled and filtered API data is classified and graded, and the classified and graded API data is stored in the first data index library.

[0127] In one possible implementation, the API data to be processed includes multiple interface call data; when the processor executes the computer program, it also implements the following steps: obtaining multiple interface call data in the API data to be processed with a preset time period as a cycle; removing the interface call data with the same user identification information and interface identification information from the multiple interface call data to obtain sampled and filtered API data.

[0128] In one possible implementation, the sampled and filtered API data includes at least one interface call data; when the processor executes the computer program, the processor also implements the following steps: identifying sensitive information contained in each interface call data; matching the sensitive information contained in each interface call data with a label rule according to a preset label configuration rule to obtain each labeled interface call data; and storing each labeled interface call data in a first data index library.

[0129] In one possible implementation, the preset label configuration rules include one or more of API function level label configuration rules, API data exposure level label configuration rules, API status level label configuration rules, and API sensitivity level level label configuration rules.

[0130] In one possible implementation, the processor also implements the following steps when executing the computer program: collecting traffic transmitted based on the Hypertext Transfer Protocol HTTP in the application environment through a collection probe; parsing the traffic to obtain traffic parsing data; performing intranet filtering, network segment filtering and interface filtering on the traffic parsing data to obtain API data to be processed.

[0131] In a possible implementation, when the computer program is executed by a processor, the following steps are implemented: removing page static resources and binary file transfer data in the traffic parsing data, and retaining structured text data in the traffic parsing data.

[0132] In a possible implementation, the processor further implements the following steps when executing the computer program: based on the API data to be processed, obtaining API detail data; the API detail data includes multiple API call details; and classifying and storing the multiple API call details in a second data index library.

[0133] In one possible implementation, the processor also implements the following steps when executing the computer program: sampling and filtering the API data to be processed based on the user identification information and the interface identification information in a preset time period, and after obtaining the sampled and filtered API data, storing the sampled and filtered API data in the message middleware; classifying and grading the sampled and filtered API data, and obtaining the sampled and filtered API data from the message middleware before storing the classified and graded API data in the first data index library.

[0134] The present application also provides a computer storage medium, which stores instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes one or more steps in the above embodiment. If the components of the above electronic device are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer storage medium.

[0135] In a possible implementation, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0136] Get the API data to be processed;

[0137] Taking a preset time period as a cycle, sampling and filtering the API data to be processed according to the user identification information and the interface identification information to obtain the sampled and filtered API data;

[0138] The sampled and filtered API data is classified and graded, and the classified and graded API data is stored in the first data index library.

[0139] In one possible implementation, the API data to be processed includes multiple interface call data; when the computer program is executed by the processor, the following steps are also implemented: obtaining multiple interface call data in the API data to be processed with a preset time period as a cycle; removing the interface call data with the same user identification information and interface identification information from the multiple interface call data to obtain sampled and filtered API data.

[0140] In one possible implementation, the sampled and filtered API data includes at least one interface call data; when the computer program is executed by the processor, the following steps are also implemented: identifying sensitive information contained in each interface call data; matching the sensitive information contained in each interface call data with a label rule according to a preset label configuration rule to obtain each labeled interface call data; and storing each labeled interface call data in a first data index library.

[0141] In one possible implementation, the preset label configuration rules include one or more of API function level label configuration rules, API data exposure level label configuration rules, API status level label configuration rules, and API sensitivity level level label configuration rules.

[0142] In one possible implementation, when the computer program is executed by the processor, the following steps are also implemented: collecting traffic transmitted based on the Hypertext Transfer Protocol (HTTP) in the application environment through a collection probe; parsing the traffic to obtain traffic parsing data; performing intranet filtering, network segment filtering, and interface filtering on the traffic parsing data to obtain API data to be processed.

[0143] In a possible implementation, when the computer program is executed by the processor, the following steps are also implemented: removing page static resources and binary file transfer data in the traffic parsing data, and retaining structured text data in the traffic parsing data.

[0144] In one possible implementation, when the computer program is executed by the processor, the following steps are also implemented: based on the API data to be processed, API detail data is obtained; the API detail data includes multiple API call details; and the multiple API call details are classified and stored in the second data index library.

[0145] In one possible implementation, when the computer program is executed by the processor, the following steps are also implemented: sampling and filtering the API data to be processed based on the user identification information and the interface identification information in a preset time period, and after obtaining the sampled and filtered API data, storing the sampled and filtered API data in the message middleware; classifying and grading the sampled and filtered API data, and obtaining the sampled and filtered API data from the message middleware before storing the classified and graded API data in the first data index library.

[0146] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted by the computer-readable storage medium. The computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes. In the absence of conflict, the technical features in this embodiment and the implementation scheme can be combined arbitrarily.

[0148] The embodiments described above are merely preferred embodiments of the present application and are not intended to limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements made to the technical solutions of the present application by ordinary technicians in this field should fall within the protection scope determined by the claims.

[0149] The above describes a specific embodiment of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A data classification and grading method, characterized in that: The method comprises: Get the API data to be processed; The API data to be processed is sampled and filtered according to the user identification information and the interface identification information in a preset time period to obtain the sampled and filtered API data; The sampled and filtered API data is classified and graded, and the classified and graded API data is stored in a first data index library.

2. The method according to claim 1, characterized in that The API data to be processed includes multiple interface call data; The step of sampling and filtering the API data to be processed according to the user identification information and the interface identification information in a preset time period to obtain the sampled and filtered API data includes: Acquire multiple pieces of interface call data from the API data to be processed in a preset time period; The interface calling data having the same user identification information as the interface identification information is removed from the plurality of interface calling data to obtain the sampled and filtered API data.

3. The method according to claim 1, characterized in that The sampled and filtered API data includes at least one piece of interface call data; The classifying and grading the sampled and filtered API data, and storing the classified and graded API data in a first data index library, comprises: Identifying sensitive information contained in each of the interface call data; According to the preset label configuration rules, label rule matching is performed on the sensitive information contained in each of the interface call data to obtain the labeled interface call data; The labeled interface call data are stored in the first data index library.

4. The method according to claim 3, characterized in that The preset label configuration rules include one or more of API function level label configuration rules, API data exposure level label configuration rules, API status level label configuration rules and API sensitivity level label configuration rules.

5. The method according to claim 1, characterized in that The obtaining of the application program interface API data to be processed includes: Use the collection probe to collect traffic transmitted based on the Hypertext Transfer Protocol (HTTP) in the application environment; Parsing the traffic to obtain traffic parsing data; The traffic analysis data is subjected to intranet filtering, network segment filtering and interface filtering to obtain API data to be processed.

6. The method according to claim 5, characterized in that The interface filtering includes: The page static resources and binary file transmission data in the traffic parsing data are removed, and the structured text data in the traffic parsing data is retained.

7. The method according to claim 1, characterized in that After acquiring the application program interface API data to be processed, and before sampling and filtering the API data to be processed according to the user identification information and the interface identification information in a preset time period, the method further includes: Based on the API data to be processed, obtaining API detailed data; the API detailed data includes multiple API call detailed information; The multiple API call detailed information are classified and stored in the second data index library.

8. A data classification and grading device, characterized in that: The device comprises: A data acquisition module is used to acquire the application program interface API data to be processed; A sampling and filtering module, used to perform sampling and filtering on the API data to be processed according to the user identification information and the interface identification information at a preset time period, to obtain the API data after sampling and filtering; The classification and grading module is used to classify and grade the API data after sampling and filtering, and store the classified and graded API data in the first data index library.

9. An electronic device, characterized in that: include: A processor and a memory; the memory stores a computer program, and the processor implements the method steps of any one of claims 1 to 7 when executing the computer program.

10. A computer storage medium, characterized in that: The computer storage medium stores a plurality of instructions, which are suitable for being loaded by a processor and executing the method steps according to any one of claims 1 to 7.

Images