Method and system for security evaluation of self-developed information system code in large enterprises

Abstract

The application provides a large enterprise internal self-development information system code security evaluation method and system. The application uses code security visualization integrated management technology, distributed message queue technology, flow collection technology and vulnerability detection technology to realize security detection of enterprise internal self-development system code, high-risk vulnerability scanning, effectively solve the problem of frequent security vulnerabilities of self-development information system after online, and provide real-time reliable data support for the code security situation and information security situation of the development department at all levels for the company information management layer through the visualization integrated management technology. In addition, the application can be applied to the large enterprise internal self-development information system development team, and the information security vulnerability detection is carried out on the code side before the information system is online, so that the information security vulnerability can be found in time and effectively.

Description

TECHNICAL FIELD

[0001] The present application relates to the security field of self-developed information system code, in particular to a large enterprise internal self-developed information system code security evaluation method and system. BACKGROUND

[0002] While the disruptive trends of cloud computing, big data, and the like continue to play a role in the application economy, business rapid iteration and other needs have become indispensable in business decision-making. In today's application-driven, cloud computing, and mobile environment, business rapid iteration will help business value-added.

[0003] Considering the need for business rapid iteration, many aspects of traditional security processes have become obstacles that must be removed, and unfortunately many enterprises are unaware of these problems. The operation of traditional security is based on the system being released, and then the security personnel confirm the security risks of the system. The process designed using this method is only suitable for waterfall mode business activities. Unfortunately, with the introduction of iteration, this way of operating security is flawed and introduces inherent risks into the system, as business decisions need to balance in-line and keep up with the speed of business.

[0004] With the rapid changes in business decisions, traditional security is no longer an option. In the development cycle, it is too late, and when it is coordinated with iterative design and system release, it is not fast enough. Traditional security is in a relatively awkward position under the demand of business rapid iteration.

[0005] Usually, the security team cannot collect all the information needed to make meaningful security decisions. In order to provide iteration values that closely map customer needs, value creation processes are accelerating. A decision at the end of the cycle or a complete system test can have devastating results. In fact, most of these security decisions are rarely adopted and are often rejected by business executives, but when a security incident occurs, the security team is the first to be questioned.

[0006] Therefore, it has become the consensus of security personnel that security issues can be discovered and fixed in time before the system is released, even in the testing and development stages. However, the current mainstream security products either cannot automatically, quickly, and comprehensively discover vulnerabilities, or cannot intuitively present vulnerabilities to developers; with business iteration, security personnel need to constantly initiate scanning tasks, manually organize each scanning result into a relatively easy-to-understand scanning report and send it to the relevant personnel, and after repair, security personnel review, greatly reducing the work efficiency of security personnel and failing to meet the needs of business rapid iteration. How to conduct security detection on new business systems before they go online has become a key concern for each enterprise.

[0007] The Chinese patent application with publication date of July 8, 2015 and publication number CN104767757A discloses a multi-dimension security monitoring method and system based on WEB service, which comprehensively monitors website security from three dimensions of availability, security event and web vulnerability. The monitoring scanning module detects data of a target website by using a port detection engine, and then transmits scanning information to a data center. The data center effectively evaluates website security condition by correlation analysis of information, and classifies the website as a high-risk website, a medium-risk website, a low-risk website or a safe website. The application realizes comprehensive monitoring of website security from multiple dimensions, has higher efficiency, dynamically displays monitoring results, and realizes security monitoring of each business node in a scanned network. However, the application patent application does not involve pre-release security testing of self-developed information system code in a large enterprise.

[0008] The Chinese patent application with publication date of July 29, 2015 and publication number CN104809404A discloses a data layer system of an information security attack and defense platform, which is provided with a tool library module, a scene configuration library module, a courseware library module, a security intelligence library module, a log library module, an attack behavior library module and a platform library module. The seven resource library modules are independently operated, linked between modules and linked with external data, so as to provide a complete, efficient and unified standard data layer system for the information security attack and defense platform in a virtual device and entity device combination manner, and help the information security attack and defense platform to effectively realize network security evaluation, attack and defense confrontation, new product inspection and testing, and a series of evaluations and tests on network architecture, design process, host security and data security. However, the application patent application does not involve pre-release security testing of self-developed information system code in a large enterprise. SUMMARY

[0009] The present application aims to solve at least one of the above-mentioned deficiencies in the prior art. For example, one of the objectives of the present application is to solve the problem of frequent information security vulnerabilities caused by the fast iteration speed of self-developed information systems in a large enterprise during the development cycle, the late intervention of the security testing process and the insufficient personnel of the internal security team.

[0010] To achieve the above-mentioned objectives, one aspect of the present application provides a security evaluation method for self-developed information system code in a large enterprise, which comprises the following steps:

[0011] Step one: when performing a routine function test on an information system, a traffic collection module automatically captures http / https traffic in a completely transparent manner for users, and finally saves all collected traffic to a storage center database;

[0012] Step two, analyze the request flow, data flow in the storage center database through the flow analysis module, analyze the request mode, interface URL and request body of the target site, and finally store them in the storage center database in a structured manner for subsequent module calling;

[0013] Step three, call the crawler module, according to the target website domain name or address extracted in the distributed message queue, try to initiate HTTP request to the target address for HTML text crawling, then parse the URI endpoint in the response HTML text, and store the request and response HTML text and URI endpoint of the crawler into the storage center database for subsequent module calling;

[0014] Step four, the vulnerability detection engine takes out the information system URL that has been identified from the storage center database, performs multi-threaded scanning on all URLs of the information system, and inserts the scanning results into the visual integrated management platform database;

[0015] Step five, analyze the code security vulnerability scanning result data stored in the storage center database, and perform vulnerability analysis and chart display.

[0016] Another aspect of the application provides a large enterprise internal self-developed information system code security evaluation system, the security evaluation system comprises a flow collection module, a flow analysis module, a crawler module, a vulnerability detection engine, a storage center database, and a visual integrated management platform database, wherein,

[0017] The flow collection module can automatically capture http / https flow in a completely transparent manner for users when performing routine function test on the information system, and finally all the collected flow is saved to the storage center database;

[0018] The flow analysis module can analyze the request flow, data flow in the storage center database, analyze the request mode, interface URL and request body of the target site, and finally store them in the storage center database in a structured manner for subsequent module calling;

[0019] The crawler module can be called to realize according to the target website domain name or address extracted in the distributed message queue, try to initiate HTTP request to the target address for HTML text crawling, then parse the URI endpoint in the response HTML text, and store the request and response HTML text and URI endpoint of the crawler into the storage center database for subsequent module calling;

[0020] The vulnerability detection engine can take out the information system URL that has been identified from the storage center database, perform multi-threaded scanning on all URLs of the information system, and insert the scanning results into the visual integrated management platform database;

[0021] The visualization integrated management platform database can analyze and chart the scanning results.

[0022] Compared with the prior art, the beneficial effects of the present application include at least one of the following:

[0023] By moving the security test process to the development stage, information security vulnerability detection can be performed on the code side before the information system goes online, and information security vulnerabilities can be found in time and effectively, solving the problem of frequent information security vulnerabilities caused by the fast iteration speed of the self-developed information system in the development cycle, the late intervention of the security test process, and the lack of personnel in the enterprise internal security team.

[0024] By using code security visualization integrated management technology, distributed message queue technology, traffic collection technology and vulnerability detection technology, security detection and high-risk vulnerability scanning of the self-developed system code in the enterprise can be realized, effectively solving the problem of frequent security vulnerabilities of the self-developed information system after going online.

[0025] By using the visualization integrated management technology, real-time and reliable data support can be provided for the company's information management layer to understand the code security and information security of the development departments at all levels. BRIEF DESCRIPTION OF DRAWINGS

[0026] The above and other objects and / or characteristics of the present application will become more apparent from the following description with reference to the attached drawings, wherein:

[0027] Fig. 1 A flowchart of an exemplary embodiment of the present application is shown;

[0028] Fig. 2 A GET request connection diagram in an exemplary embodiment of the present application is shown;

[0029] Fig. 3 A POST request connection diagram in an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] In the following, the large enterprise internal self-developed information system code security evaluation method and system of the present application will be described in detail in conjunction with exemplary embodiments. Here, the large enterprise internal self-developed information system is a large system supported by information technology and communication technology, with a large scale, wide distribution, multi-level network structure, spanning multiple security domains, processing massive, complex and various data, and providing multiple types of applications.

[0031] Overall, the large enterprise internal self-development information system code security evaluation method and system provided by the present application can be applied to the large enterprise internal self-development information system development team, the security test process is moved to the development stage, the information security vulnerability detection is carried out on the code side before the information system is online, and the information security vulnerability is found in time and effectively. The present application realizes the security detection of the enterprise internal self-development system code, the high-risk vulnerability scanning by using the code security visual integrated management technology, the distributed message queue technology, the flow collection technology and the vulnerability detection technology, effectively solves the problem of frequent security vulnerabilities of the self-development information system after online, and simultaneously provides real-time reliable data support for the code security situation and the information security situation of the development departments at all levels for the company informatization management layer through the visual integrated management technology.

[0032] Fig. 1 The flow diagram of one exemplary embodiment of the present application is shown; Fig. 2 The GET request connection diagram in one exemplary embodiment of the present application is shown, wherein the GET request is to request data from a specified resource, the SYN is a synchronization sequence number, the ACK is an acknowledgement character, and the GET Head is a response message header for requesting to obtain a resource. Fig. 3 The POST request connection diagram in one exemplary embodiment of the present application is shown, wherein the POST request is to submit data to be processed to a specified resource, the SYN is a synchronization sequence number, the ACK is an acknowledgement character, and the DATA is data.

[0033] As shown in Figs. 1 to 3 In one exemplary embodiment of the present application, the large enterprise internal self-development information system code security evaluation method is realized by the following steps:

[0034] Step one, first, collect http / https flow, the enterprise internal tester collects the flow for security test when carrying out the conventional function test on the information system, which is completely transparent to the user, and finally all the collected flow is saved to the storage center database.

[0035] The module of flow collection is divided into a proxy module, a plug-in module, a flow sniffing module and a log import module.

[0036] 1) The agent module is a function module using gateway agent technology and clustering. It performs identity authentication and replicates business traffic to a storage center. The clustered agent mode can achieve effective load balancing in high concurrency situations when business access volume is concentrated. The clustered agent can be self-scheduled to process requests from clients and forward them to the storage center. Through the application, the agent cluster can reduce the number of single-point failures and achieve high availability of clustered resources. It can effectively solve the problems of excessive data flow and heavy network load without incurring the expense of purchasing high-performance servers. It makes full use of existing equipment and avoids data flow loss caused by server single-point failure. It has flexible and diverse balancing strategies to reasonably distribute data flow to servers in the server cluster to share the burden. Even if the existing server is expanded and upgraded, it is only necessary to simply add a new server to the service cluster without changing the existing network structure and stopping the existing service.

[0037] 2) The instrumentation module is a passive security testing technology and active security testing technology linkage detection. Through the application, passive security testing needs to deploy an Agent probe in the Web Server, inject tracking code into the bytecode program using instrumentation technology to form instrumented bytecode, and directly obtain data flow and collect relevant information when the program runs. The instrumentation mode only needs to be started on the business server to be tested with one key, and the code data flow, HTTP request / response packet, and other information of the tested project can be obtained and sent to the storage center database.

[0038] 3) The traffic sniffing module is mainly used for PC and mobile terminal test environment under complex conditions. The traffic collection Agent is deployed on the business server to obtain HTTP traffic through sniffing and send it to the storage center database. Through the application, the test personnel are completely unaware, the additional configuration work is removed, and the test personnel's work process is completely unchanged. At the same time, the traffic collection Agent uses the sniffing method to obtain HTTP traffic, and the business data link is a bypass relationship, which does not affect the business data flow.

[0039] 4) The log import module mainly aims at enterprises with a self-built perfect log platform that records complete WEB log conditions. The log import module has a complete log import interface. Through the application, the enterprise imports the logs in the log platform into the distributed message queue system according to the agreed format. The log analysis module extracts the Header, Cookie, User-Agent, Body and other elements in the log, reorganizes the log based on the HTTP protocol, restores the original request and sends the restored request to the storage center database. By converting the rich access logs, function test logs, user behavior logs and other logs recorded in the log platform into original requests and storing them and handing them over to the vulnerability detection engine for detection, the maximum coverage of the interface can be achieved, and the detection effect can be optimized.

[0040] Each collection module corresponds to different business scenarios, and there are differences in usage. The plug-in module can support general vulnerabilities, business logic vulnerabilities, third-party components and other risk detection, and can also cover encryption, CAPTCHA and other one-time interface scenarios, truly achieving full coverage of business scenarios. The plug-in module does not need to replay the request, and there is no dirty data, almost achieving 0 false positives, improving the detection precision, and directly locating the code location, code content and data flow information, fully presenting the whole process of the vulnerability from input, propagation to final execution, greatly reducing the difficulty of vulnerability reproduction.

[0041] Step two, analyze the request flow and data flow in the storage center database through the flow analysis module.

[0042] The flow analysis module mainly analyzes the HTTP protocol flow. In normal functional testing, normal request and response data are collected, the HTTP request protocol package is parsed, including the method, protocol version and resource URI in the request line, the protocol field and field value in the request header, and the parameter and parameter value in the request body. The HTTP response protocol package is parsed, including the protocol version and status code request flow in the response line, the parameter and parameter value in the response header, and the HTML text in the response body.

[0043] Step three, call the crawler module, according to the target website domain name or address extracted from the distributed message queue, try to initiate HTTP request to the target address for HTML text crawling, then parse the URI endpoint in the response HTML text, and store the request and response HTML text of the crawler and the URI endpoint in the storage center database for subsequent module calling.

[0044] The working process of the crawler module is as follows:

[0045] 1) Crawler module gets the URL address string of the information system from the distributed message queue system, splits the URL address string by regular matching, and then gets the independent protocol type string (http / https) and address (IP or domain name) and parameter information. In this way, the recognition and segmentation of the URL address are completed.

[0046] 2) By adding a Header request header and a request parameter, a Request request object is constructed, and then a standard HTTP method call (GET) is initiated to the segmented target address. The target server responds with a standard HTML text, and in this way, the step of obtaining the target site HTML text is completed.

[0047] 3) The HTML text is converted into a DOM tree, the corresponding tag content and attributes and attribute values are obtained through the corresponding HTML tags, and then the URI endpoints related to the target address are extracted and saved into the storage center database through matching screening, and in this way, the steps of HTML text parsing and URI endpoint extraction are completed.

[0048] Step four, the vulnerability detection engine takes out the information system URL that has been identified from the storage center database, performs multi-threaded scanning on all URLs of the information system, and inserts the scanning results into the visual integrated management platform database.

[0049] The vulnerability detection engine quickly detects project vulnerabilities through cluster detection. Cluster technology can obtain relatively high performance, reliability, and flexibility at a relatively low cost. Task scheduling is the core technology in a cluster system. At the same time, the detection cluster obtains the request flow data from the storage center, simulates multiple vulnerability detection methods to comprehensively detect the requests in the user configuration project, and quickly generates a vulnerability report. The detection cluster automatically detects and updates the new detection results to the vulnerability report as the business requests change.

[0050] Step five, analyze the code security vulnerability scanning result data stored in the storage center database, perform vulnerability number statistics, project vulnerability ranking, and vulnerability trend distribution, and display the results through the code security visual integrated management console to provide decision support for information management.

[0051] After the definition of the security evaluation method of the above exemplary embodiment, the development test personnel and the professional code security audit team use the security evaluation method of the exemplary embodiment to perform full-quantity code audit on the following systems, and the related result summary report is as follows:

[0052] Example 1 integrated platform summary report

[0053] I. Source Code Information

[0054]

[0055] II. Grade Statistics

[0056]

[0057] III. Category Statistics

[0058]

[0059]

[0060]

[0061] Example 2 Contractor Management System Summary Report I. Source Code Information

[0062]

[0063] II. Grade Statistics

[0064]

[0065] III. Category Statistics

[0066]

[0067]

[0068]

[0069] Example 3 Technology System Summary Report

[0070] I. Source Code Information

[0071]

[0072] II. Grade Statistics

[0073]

[0074]

[0075] III. Category Statistics

[0076]

[0077] In another exemplary embodiment of the present application, the security evaluation system of self-developed information system code in large enterprises is composed of a traffic collection module, a traffic analysis module, a crawler module, a vulnerability detection engine, a storage center database, and a visual integrated management platform database. The system can realize the security evaluation method of self-developed information system code in large enterprises in the above exemplary embodiment.

[0078] The traffic collection module can automatically capture the http / https traffic in a completely transparent manner for the user when performing routine function tests on the information system, and finally all the collected traffic is saved to the storage center database. The traffic collection module includes a proxy module, a plug-in module, a traffic sniffing module, and a log import module. The plug-in module can support general vulnerability, business logic vulnerability, and third-party component risk detection, and can also cover encryption, one-time interface scenarios of verification codes; the plug-in module does not need to replay requests and has no dirty data, and can directly locate to code location, code content, and data flow information, and completely show the whole process of the vulnerability from input, propagation to final execution.

[0079] The traffic analysis module can analyze the request traffic and data traffic in the storage center database, analyze the request method, interface URL, and request body of the target site, and finally structure the storage to the storage center database for subsequent module calling. The traffic analysis module can analyze the HTTP protocol traffic, collect request and response data in functional testing, parse the HTTP request protocol package and the HTTP response protocol package, the HTTP response protocol includes the method in the request line, the protocol version and the resource URI, the protocol field and the field value in the request header, and the parameter and the parameter value in the request body; the HTTP response protocol includes the protocol version and the status code returned in the response line, the parameter and the parameter value in the response header, and the HTML text in the response body.

[0080] The crawler module can be called to realize the extraction of the target website domain name or address in the distributed message queue, attempt to initiate an HTTP request to the target address for HTML text crawling, then parse the URI endpoint in the response HTML text, and store the request and response HTML text of the crawler and the URI endpoint in the storage center database for subsequent module calling.

[0081] The working process of the crawler module is as follows:

[0082] 1) The crawler module obtains the URL address string of the information system from the distributed message queue system, divides the URL address string by regular matching, then obtains independent protocol type string and address and parameter information to complete the identification and segmentation of the URL address;

[0083] 2) by adding Header request header and request parameters, construct a Request request object, then initiate a standard HTTP method call to the target address divided out, the target server responds to the standard HTML text, thereby completing the steps of obtaining the target site HTML text;

[0084] 3) the HTML text is converted into a DOM tree, the corresponding tag content and attributes and attribute values are obtained through the corresponding HTML tag, then the URI endpoint related to the target address is extracted and saved into the storage center database through matching screening, thereby completing the steps of HTML text analysis and URI endpoint extraction.

[0085] The vulnerability detection engine can take out the information system URL that has been identified from the storage center database, perform multi-thread scanning on all URLs of the information system, and insert the scanning result into the visual integrated management platform database.

[0086] The visual integrated management platform database can analyze and graphically display the scanning result.

[0087] In summary, the present application is used for the security detection of the online new business system by the inventor, and a mode and method for the security evaluation of the self-developed information system code in large enterprises are researched. Through the method, the development and test personnel can quickly locate the problems in code writing, immediately adjust the code content structure, ensure the safety and reliability of the entire code structure, and the information management layer of the company can provide real-time and reliable data support for the code security and information security of the development departments at all levels.

[0088] Although the present application has been described above with reference to the example embodiments and the accompanying drawings, it should be clear to those skilled in the art that various modifications can be made to the above embodiments without departing from the spirit and scope of the claims.

Claims

1. A large enterprise internal self-development information system code security evaluation method, the security evaluation method comprising the following steps: Step one, when performing routine function test on the information system, the traffic collection module automatically captures the http / https traffic in a completely transparent manner for the user, and finally all the collected traffic is saved to the storage center database; Step two, analyze the request traffic and data traffic in the storage center database through the traffic analysis module, analyze the request method, interface URL and request body of the target site, and finally store them in a structured manner to the storage center database for subsequent module calling; Step three, call the crawler module, according to the target website domain name or address extracted from the distributed message queue, try to initiate HTTP request to the target address for HTML text crawling, then parse the URI endpoint in the response HTML text, and store the request and response HTML text of the crawler and the URI endpoint into the storage center database for subsequent module calling; Step four, the vulnerability detection engine takes out the information system URL that has been identified from the storage center database, performs multi-thread scanning on all URLs of the information system, and inserts the scanning results into the visual integrated management platform database; Step five, analyze the code security vulnerability scanning result data stored in the storage center database, and perform vulnerability analysis and chart display.

2. The method for security assessment of in-house self-developed information system code of large enterprises according to claim 1, characterized by, The traffic collection module includes proxy module, plug-in module, traffic sniffing module, and log import module.

3. The method for security assessment of in-house self-developed information system codes of large enterprises according to claim 2, characterized in that, The plug-in module can support general vulnerability, business logic vulnerability and third-party component risk detection, and can also cover encryption, one-time interface scene of verification code; The plug-in module does not need to replay the request, and there is no dirty data, which can directly locate to the code location, code content and data flow information, and completely show the whole process of the vulnerability from input, propagation to final execution.

4. The method for security assessment of in-house self-developed information system codes of large enterprises according to claim 1, characterized by, The traffic analysis module can analyze HTTP protocol traffic, collect request and response data in functional test, and parse HTTP request protocol package and HTTP response protocol package.

5. The method for security assessment of in-house self-developed information system codes of large enterprises according to claim 4, characterized in that, The HTTP response protocol includes the method in the request line, the protocol version and the resource URI, the protocol field and the field value in the request header, and the parameter and the parameter value in the request body.

6. The method for security assessment of in-house self-developed information system codes of large enterprises according to claim 4, characterized by, The HTTP response protocol includes the protocol version and the status code returned in the response line, the parameter and the parameter value in the response header, and the HTML text in the response body.

7. The method for security assessment of in-house self-developed information system codes of large enterprises according to claim 1, characterized by, The workflow of the crawler module is as follows: 1) The crawler module obtains the URL address string of the information system from the distributed message queue system, divides the URL address string through regular matching, and then obtains independent protocol type string and address and parameter information, to complete the identification and segmentation of the URL address; 2) By adding Header request header and request parameter, a Request request object is constructed, then a standard HTTP method call is initiated to the segmented target address, the target server responds to the standard HTML text, thereby completing the step of obtaining the target site HTML text; 3) The HTML text is converted into a DOM tree, the corresponding tag content and attributes and attribute values are obtained through the corresponding HTML tag, and then the URI endpoint related to the target address is extracted and saved into the storage center database through matching screening, so as to complete the steps of HTML text analysis and URI endpoint extraction.

8. The method for security assessment of in-house self-developed information system codes of large enterprises according to claim 1, characterized by, The vulnerability detection engine detects the project vulnerabilities quickly through the method of the cluster.

9. The method for security assessment of in-house self-developed information system codes of large enterprises according to claim 1, characterized by, The vulnerability analysis includes vulnerability statistics, project vulnerability ranking, and vulnerability trend distribution, and the chart is displayed through the code security visualization integrated management console.

10. A security evaluation system for in-house self-developed information system codes in large enterprises, characterized by The security evaluation system comprises a traffic collection module, a traffic analysis module, a crawler module, a vulnerability detection engine, a storage center database, and a visualization integrated management platform database. The traffic collection module can automatically capture http / https traffic in a completely transparent manner for users during regular functional testing of the information system, and finally all collected traffic is saved to the storage center database. The traffic analysis module can analyze the request traffic and data traffic in the storage center database, analyze the request method, interface URL and request body of the target site, and finally structure the storage to the storage center database for subsequent module calling. The crawler module can be called to extract the target website domain name or address from the distributed message queue, attempt to initiate an HTTP request to the target address for HTML text crawling, then analyze the URI endpoint in the response HTML text, and store the crawler request and response HTML text and URI endpoint into the storage center database for subsequent module calling. The vulnerability detection engine can take out the information system URL that has been identified from the storage center database, perform multi-thread scanning on all URLs of the information system, and insert the scanning results into the visualization integrated management platform database. The visualization integrated management platform database can analyze the scanning results and display the chart.

11. The system for security assessment of in-house self-developed information system codes of large enterprises according to claim 10, characterized in that, The traffic collection module comprises a proxy module, a plug-in module, a traffic sniffing module, and a log import module.

12. The system for security assessment of in-house self-developed information system codes of large enterprises according to claim 10, wherein The traffic analysis module can analyze HTTP protocol traffic, collect request and response data in functional testing, analyze HTTP request protocol packets and HTTP response protocol packets, and the HTTP response protocol contains the method, protocol version and resource URI in the request line, the protocol field and field value in the request header, and the parameter and parameter value in the request body.

13. The system for security assessment of in-house self-developed information system codes of large enterprises according to claim 10, wherein The workflow of the crawler module is as follows: 1) The crawler module obtains the URL address string of the information system from the distributed message queue system, splits the URL address string by regular matching, and then obtains independent protocol type string and address and parameter information, to complete the recognition and segmentation of the URL address; 2) By adding the Header request header and request parameters, a Request request object is constructed, and then the target address segmented out is called by a standard HTTP method, and the target server responds to the standard HTML text, thereby completing the step of obtaining the target site HTML text; 3) The HTML text is converted into a DOM tree, the corresponding label content and attributes and attribute values are obtained through the corresponding HTML label, and then the URI endpoints related to the target address are extracted and saved into the storage center database through matching screening, thereby completing the steps of HTML text analysis and URI endpoint extraction.

Images