Malicious crawler identification and display method, system, electronic device and storage medium

Abstract

The present invention discloses a method, system, electronic device and storage medium for identifying and displaying malicious crawlers. The method comprises: obtaining characteristic elements of request information related to an IP address within a certain time range; judging whether a request related to each IP address is an abnormal request based on a predetermined threshold value of any characteristic element in the request information or two or more threshold values after a combination of any associated characteristic elements, and generating a first cascade display area; filtering non-abnormal requests to generate a second cascade display area; generating a third cascade display area for each abnormal request, obtaining interface information under the abnormal IP address to generate a fourth cascade display area; and generating a fifth cascade display area based on the log information of the IP address or interface under the abnormal request. The method of the present invention can quickly perform fine-grained indicator monitoring and analysis on various request information indicators, thereby improving the accuracy of malicious crawler identification, making viewing more intuitive, improving identification efficiency, and maintaining data security.

Description

Technical Field

[0001] The present invention relates to the technical field of malicious crawlers, and in particular to a method, system, electronic device and storage medium for monitoring and displaying malicious crawlers. Background Art

[0002] In recent years, with the rapid development of internet technology, web crawler technology has been widely used in various fields, such as search engines, data analysis, and business intelligence. However, with the popularization of web crawler technology, malicious crawler activities have become increasingly rampant. Malicious crawlers use automated means to illegally obtain and abuse website data, leading to frequent data leaks and user privacy violations, seriously affecting enterprise data security and significantly compromising enterprise security and user experience.

[0003] Currently, anti-crawler technology generally uses the access traffic value of the corresponding IP address, that is, the click traffic value, as a single indicator to determine whether it is malicious crawler behavior. Identifying crawler behavior by rough access traffic value may lead to misjudgment of malicious crawler behavior. This is because malicious crawlers usually use camouflage and evasion techniques. They may disguise themselves as normal users to access the website, or evade anti-crawler monitoring by modifying request headers, using proxy IPs, etc., which makes identification and analysis difficult and poses a threat to data security. In addition, there is no way to intuitively display various indicators of crawler behavior and their related information and other fine-grained analysis indicators to users. Users cannot quickly understand the business situation, which may eventually lead to data leakage. Summary of the Invention

[0004] The present invention provides a malicious crawler identification and display method, system, electronic device and storage medium to solve the problems in existing anti-crawler technology such as inaccurate crawler identification, low recognition efficiency and lack of comprehensive indicator display, which result in users being unable to quickly understand business conditions.

[0005] In order to solve the above technical problems, in a first aspect, the present invention provides a method for identifying and displaying malicious crawlers, comprising:

[0006] Obtain request information related to an IP address within a certain time range. The characteristic elements of the request information include the number of requests, the percentage of request times, the number of abnormal requests, the percentage of abnormal requests, the number of failed requests, the percentage of failed requests, the maximum session duration, and the maximum session duration request speed.

[0007] Determine whether the request related to each IP address is an abnormal request based on any preset threshold of a characteristic element in the request information or two or more thresholds of a combination of any associated characteristic elements, and generate a first cascade display area to display the request information related to each IP address and the abnormality result in the first cascade display area;

[0008] Filter non-abnormal requests and generate a second cascade display area based on the IP address of the abnormal request and the request information under the IP address;

[0009] For each abnormal request, a third cascade display area is generated under the IP address, and the third cascade display area is used to display the IP portrait of the request information under the IP address;

[0010] Obtain the interface information under the abnormal IP address and generate the fourth cascade display area;

[0011] Based on the log information of the IP address or interface under the abnormal request, a fifth cascade display area is generated for querying and verifying the abnormal request information.

[0012] Optionally, the content displayed in the first cascade display area includes IP address information within a certain time range and the following list under the IP address: number of requests, percentage of request times, number of request anomalies, percentage of request anomalies, number of request failures, percentage of request failures, maximum session duration, maximum session duration, request speed information and abnormal indicator identification.

[0013] Optionally, whether the request related to each IP address is an abnormal request is determined based on any pre-set characteristic element threshold in the request information or two or more thresholds after the combination of any related characteristic elements, including setting thresholds for the number of requests, the proportion of request times, the number of abnormal requests, the proportion of abnormal requests, the number of request failures, the proportion of request failures, and the maximum session duration in the request information. When any characteristic element threshold or any one of the two or more thresholds after the combination of any related characteristic elements is not met, it is determined to be an abnormal request.

[0014] Optionally, the characteristic elements of the request information also include sensitive words. Correspondingly, whether the request related to each IP address is an abnormal request is judged based on any preset characteristic element threshold in the request information or two or more thresholds after the combination of any associated characteristic elements. It also includes only comparing the sensitive words in the request information with the preset sensitive words to determine whether the request is an abnormal request.

[0015] Optionally, the IP portrait includes basic information of the IP, threat situation, and traffic trend chart.

[0016] Optionally, the fourth cascade display area displays the IP address under the abnormal indicator and the corresponding list of the IP address, and the list name includes the interface name, interface code, interface priority, whether it is overstepping authority, whether it is sensitive, number of requests, proportion of request times, daily request peak ratio, number of request anomalies, and abnormal indicator identification.

[0017] Optionally, the fifth cascade display area includes generating different log time distribution graphs and log detail distribution lists according to different query condition combinations, wherein the different query conditions include time, interface information, IP address, regional address, and requester information.

[0018] In a second aspect, the present invention provides a malicious crawler identification and display system, comprising:

[0019] A request information acquisition unit, configured to acquire request information related to an IP address within a certain time range, wherein characteristic elements of the request information include the number of requests, the percentage of request times, the number of abnormal requests, the percentage of abnormal requests, the number of failed requests, the percentage of failed requests, the maximum session duration, and the maximum session duration request speed;

[0020] A first cascade display area generating unit, configured to determine whether a request associated with each IP address is an abnormal request based on a preset threshold value of any characteristic element in the request information or two or more threshold values formed by combining any associated characteristic elements, and to generate a first cascade display area to display the request information associated with each IP address and the abnormality determination result in the first cascade display area;

[0021] A second cascade display area generating unit is used to filter non-abnormal requests and generate a second cascade display area according to the IP address of the abnormal request and the request information under the IP address;

[0022] A third cascade display area generating unit is used to generate a third cascade display area under the IP address for each abnormal request, wherein the third cascade display area is used to display the IP portrait of the request information under the IP address;

[0023] A fourth cascade display area generating unit, configured to obtain interface information under an abnormal IP address and generate a fourth cascade display area;

[0024] The fifth cascade display area generating unit is used to generate the fifth cascade display area according to the log information of the IP address or interface under the abnormal request, and query and verify the abnormal request information.

[0025] In a third aspect, the present invention provides a malicious crawler identification and display device, including a memory and a processor, wherein:

[0026] The memory is used to store computer programs;

[0027] The processor is used to read the computer program in the memory and execute the steps of the malicious crawler identification and display method provided in the first aspect above.

[0028] In a fourth aspect, the present invention provides a computer-readable storage medium having a readable computer program stored thereon, which, when executed by a processor, implements the steps of the malicious crawler identification and display method provided in the first aspect above.

[0029] Compared with the prior art, the malicious crawler identification and display method, system, electronic device, and storage medium provided by the present invention have the following beneficial effects:

[0030] By obtaining request information related to an IP address within a certain time range, the characteristic elements of the request information include the number of requests, the proportion of the number of requests, the number of abnormal requests, the proportion of abnormal requests, the number of request failures, the proportion of request failures, the maximum session duration, and the maximum session duration request speed; judging whether the request related to each IP address is an abnormal request based on any preset characteristic element threshold in the request information or two or more thresholds after any combination of associated characteristic elements, and generating a first cascade display area, displaying the request information related to each IP address and the result of whether it is abnormal through the first cascade display area; filtering non-abnormal requests, based on the IP address where the abnormal request is located and the request information under the IP address Generate a second cascade display area; for each abnormal request, generate a third cascade display area under the IP address, the third cascade display area is used to display the IP portrait of the request information under the IP address; obtain the interface information under the abnormal IP address, and generate a fourth cascade display area; based on the log information of the IP address or interface under the abnormal request, generate a fifth cascade display area, which is a technical solution for abnormal request information query and verification, and can quickly perform fine-grained indicator monitoring and analysis on various request information indicators, improve the accuracy of malicious crawler identification, make analysis and viewing more intuitive, and present more diverse request indicators, so that users can quickly understand business conditions, improve identification efficiency, prevent malicious crawler leakage, and maintain data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only part of the embodiments of the present invention, rather than all the embodiments. For ordinary technicians in this field, without paying any creative work, other drawings obtained based on these drawings are all within the scope of protection of this application.

[0032] Figure 1 This is a flow chart of a method for identifying and displaying malicious crawlers provided by an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram of a user interaction interface displayed under the indicator dashboard provided by an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of an IP portrait provided by an embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram of the fourth cascade display area provided by an embodiment of the present invention;

[0036] Figure 5 This is a schematic diagram of the structure of a malicious crawler identification and display system provided by an embodiment of the present invention;

[0037] Figure 6 1 is a schematic structural diagram of a malicious crawler identification and display device provided by an embodiment of the present invention;

[0038] Figure 7 It is a schematic diagram of the structure of a computer-readable storage medium provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0040] In order to make the description of the present disclosure more detailed and complete, the following is an illustrative description of the implementation methods and specific examples of the present invention; however, this is not the only form of implementing or using the specific embodiments of the present invention. The implementation methods cover the features of multiple specific embodiments and the method steps and their sequence for constructing and operating these specific embodiments. However, other specific embodiments can also be used to achieve the same or equal functions and step sequences. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0041] Implementation Case 1

[0042] In order to achieve the purpose of the present invention, the data collection and calculation of this embodiment adopts Apache flink as the cornerstone of real-time computing, uses kafka as the middleware, and is written to the ad hoc query starrocks. By combining the ad hoc query capability of starrocks with the pre-calculation and window computing capabilities of flink, the corresponding indicator calculation is controlled to respond in seconds, providing efficient queries for users' real-time observation of abnormal requests. At the same time, it also supports real-time configuration rules and dynamically tags the data to facilitate subsequent analysis and corresponding calculation processing of the labeled content. Apache Flink is an open source stream processing framework developed by the Apache Software Foundation. Its core is a distributed streaming data flow engine written in Java and Scala. Flink executes any streaming data program in a data-parallel and pipeline manner. Flink's pipeline runtime can execute batch and stream processing programs.

[0043] like Figure 1 As shown, a malicious crawler identification and display method provided by an embodiment of the present invention includes:

[0044] S101, obtaining request information related to an IP address within a certain time range, wherein characteristic elements of the request information include the number of requests, the percentage of request times, the number of abnormal requests, the percentage of abnormal requests, the number of failed requests, the percentage of failed requests, the maximum session duration, and the maximum session duration request speed;

[0045] In this step, the time range for obtaining data can be set according to actual needs, and the time can be accurate to seconds. The characteristic elements of the request information related to the obtained IP address are shown above, among which the number of requests, the number of request exceptions, the number of request failures, the maximum session duration, and the maximum session duration request speed can all be directly presented in numbers. For example, the number of requests is 1000 times, which means that the IP address has 1000 requests within the time period. The request number ratio, request exception ratio, or request failure ratio can be presented in the form of numbers + percentages. For example, the request exception percentage is the number of request exceptions / number of requests × 100%.

[0046] S102: Determine whether a request associated with each IP address is an abnormal request based on a preset threshold of any characteristic element in the request information or two or more thresholds formed by combining any associated characteristic elements, and generate a first cascade display area to display the request information associated with each IP address and the abnormality determination result in the first cascade display area;

[0047] As an optional implementation, whether the request related to each IP address is an abnormal request is determined based on any pre-set characteristic element threshold in the request information or two or more thresholds after the combination of any related characteristic elements, including setting thresholds for the number of requests, the proportion of request times, the number of abnormal requests, the proportion of abnormal requests, the number of request failures, the proportion of request failures, and the maximum session duration in the request information. When any characteristic element threshold or any one of the two or more thresholds after the combination of any related characteristic elements is not met, it is determined to be an abnormal request.

[0048] For example, within a certain time period, if the number of requests exceeds 1,000, the request ratio exceeds 0.5%, the number of request failures exceeds 100, and the maximum session duration exceeds 100 minutes, and there are 5 maximum session durations exceeding 100 minutes within the time period, it is set as an abnormality. In addition, users can further observe the request abnormality ratio and request failure ratio, and grasp the fine-grained and multi-dimensional information of the request abnormality, so as to facilitate the overall understanding of the abnormality.

[0049] For example, if the maximum session duration exceeds 100 minutes within a certain time period, it will be set as an anomaly, regardless of other indicators. After setting this single indicator as an anomaly, users can observe other indicators such as the number of requests, the percentage of requests, the number of abnormal requests, the percentage of abnormal requests, the percentage of failed requests, the number of times the maximum session duration exceeds 100 minutes, etc., to obtain fine-grained, multi-dimensional information on request anomalies, making it easier to understand the overall anomaly situation.

[0050] For example, within a certain time period, if the number of requests exceeds 1,000, the request ratio exceeds 0.5%, the number of request failures exceeds 100, and the maximum session duration exceeds 100 minutes, and both the number of request failures exceeds 100 and the maximum session duration exceeds 100 minutes are met within the time period, it is set as an anomaly, regardless of other characteristic factors. In addition, users can further observe other characteristic factors such as the request anomaly ratio and the request failure ratio to obtain fine-grained, multi-dimensional information about the request anomaly, making it easier to understand the overall situation of the anomaly.

[0051] As an optional implementation, the characteristic elements of the request information also include sensitive words. Correspondingly, whether the request related to each IP address is an abnormal request is determined based on any preset threshold of the characteristic elements in the request information or two or more thresholds of any combination of related characteristic elements. It also includes comparing only the sensitive words in the request information with the preset sensitive words to determine whether the request is an abnormal request. For example, as long as the sensitive words involve "customer data" or the waybill number begins with "AS", as long as such sensitive words are identified, it is determined to be abnormal.

[0052] As an optional implementation, the content displayed in the first cascade display area includes IP address information within a certain time range and the following list under the IP address: number of requests, percentage of request times, number of request exceptions, percentage of request exceptions, number of request failures, percentage of request failures, maximum session duration, maximum session duration request speed information.

[0053] For example, the first cascade display area is as follows Figure 2 The user interaction interface displayed under the indicator dashboard shown in the figure has a function selection box on the left, an indicator window of time, interface label, etc. on the upper right, and a result display corresponding to the function selection box on the left, such as the request information related to the IP address of this embodiment and whether it is abnormal. The function selection box includes indicator dashboard, IP portrait, log query, alarm management, abnormal indicators, alarm configuration, alarm processing, label management, label definition, interface classification, sensitive fields, incremental fields, management background, user management, etc. For example, if you select the indicator dashboard selection box, the following will appear: Figure 2 In the user interface shown, after entering the time period to be monitored in the upper right part of the indicator panel, the IP analysis display area in the lower right part will appear. For example, if the time period for obtaining data is from 18:40:15 on June 28, 2024 to 20:40:15 on June 28, 2024, the IP analysis display area includes a sequence number column, an IP address column, a request count column, a request count percentage column, a request anomaly count column, a request anomaly percentage column, a request failure count column, a request failure percentage column, a maximum session duration column, and a maximum session duration request speed column. Each row is the relevant request information under the IP address, including the values of the request count, request count percentage, request anomaly count, request anomaly percentage, request failure count, request failure percentage, maximum session duration, and maximum session duration request speed.

[0054] For example, after identifying sensitive words, the sensitive column marker in the first cascade display area can be used to output an exception indication while displaying the IP address and the number of requests under it, the percentage of requests, the number of request anomalies, the percentage of request anomalies, the number of request failures, the percentage of request failures, the maximum session duration and other request information, so as to facilitate users to grasp the multi-dimensional information of the sensitive exception situation.

[0055] Obtain fine-grained, multi-dimensional detailed information on the accessed object, and display fine-grained, multi-dimensional request information under each IP address to users through the first cascade display area, making it easier for users to understand the access situation. It also provides a more accurate and detailed data source for subsequent malicious crawler judgments, providing a basis for improving the accuracy of malicious crawler judgments.

[0056] S103, filtering non-abnormal requests and generating a second cascade display area based on the IP address of the abnormal request and the request information under the IP address;

[0057] In order to further grasp the abnormal request situation in detail and save the display area page space, the information that is not abnormal is filtered out according to the method of steps S101 and S102, and only the abnormal request information is displayed in the second cascade display area.

[0058] An abnormality indicator frame is provided on the first cascade display area interface, and the abnormality indicator frame can be clicked to enter the second cascade display area. The content displayed in the second cascade display area is the request information displayed in the first cascade display area under abnormal conditions.

[0059] S104: For each abnormal request, generate a third cascade display area under the IP address, wherein the third cascade display area is used to display the IP profile of the request information under the IP address;

[0060] Access the third cascade display area via the IP Profile function selection box on the left side of the dashboard's electronic user interface. IP Profile allows you to drill down into individual IP addresses in the dashboard, displaying detailed basic information, threat profiles, traffic trends, and a list of metrics across various dimensions. Users can analyze and assess the request behavior of an anomalous IP address based on its request history.

[0061] As an optional implementation, the IP portrait includes basic information of the IP, threat situation, and traffic trend chart. Figure 3 As shown, the basic information of the IP portrait includes time information and IP address information, and the threat situation includes IP traffic information, including the number of abnormal requests, interface type, UA type, reference (Referer) type, user account type, maximum session duration, request abnormality ratio, interface duplication, UA duplication ratio, reference duplication ratio, user account duplication ratio and other information; the traffic trend chart is a traffic trend pulse chart display of relevant indicators in the threat traffic situation on the timeline, of course, it can also be a chart or other graphical display.

[0062] Through IP profiling, users can analyze and understand IP request information more intuitively.

[0063] S105, obtaining interface information under the abnormal IP address and generating a fourth cascade display area;

[0064] As an optional implementation, the fourth cascade display area is entered through the interface label (not shown in the figure) at the lower right corner of the electronic user interface of the third cascade display area. Figure 4As shown, the fourth cascade display area displays the IP address under the abnormal indicator and the corresponding list of the IP address. The list name includes the interface name, interface code, interface priority, whether it is unauthorized, whether it is sensitive, number of requests, percentage of request times, daily request volume peak ratio, number of abnormal requests, and abnormal indicator identification. By obtaining the above interface information under the abnormal IP address, you can better understand the network communication channel of the abnormal IP address, providing a more accurate information source for subsequent abnormal information processing.

[0065] S106: Obtain log information of the IP address or interface under the abnormal request and generate a fifth cascade display area.

[0066] Enter the fifth cascade display area through the log query window in the indicator dashboard to perform log query for abnormal request information query and verification.

[0067] As an optional implementation, the fifth cascade display area includes generating different log time distribution graphs and log detail distribution lists according to different query condition combinations, where the different query conditions include time, interface information, IP address, regional address, personnel information, etc.

[0068] For example, by querying the log time distribution pulse chart and log detail distribution list of the abnormal information output by the human resources management application, the log detail distribution list includes field information such as the specific time of the event, IP address, region, interface, interface name, application name (human resources management application), etc. In addition, the fifth cascade display area also includes a field search box, through which you can select the fields that appear in the list and hide field information that does not need to appear.

[0069] Through the display of the fifth cascade display area, any combination of log data can be traced and analyzed, and the corresponding fields will appear in the list according to the field search. This quickly, effectively and intuitively meets user needs while also saving data storage resources and improving query speed.

[0070] It should be noted that the indicator dashboard is implemented based on StarRocks ad hoc queries. By dividing the entire indicator system into general statistics, deduplication statistics, and session window statistics, different categories use real-time calculation and Flink pre-calculation respectively. Among them, general statistics, such as IP request information, use real-time calculation. Deduplication statistics deduplicate IP request information and use detailed table real-time statistics. Considering performance, deduplication statistics are implemented using StarRocks bitmap optimization. Pre-aggregate tables, such as the formation of various form display areas, use Flink to calculate and output the corresponding window values and session data in the front-end link. Ultimately, 15+ dimensions and 100+ indicators can be constructed.

[0071] Implementation Case 2

[0072] Based on the above malicious crawler identification and display method, the embodiment of the present invention provides a malicious crawler identification and display system, such as Figure 5 shown.

[0073] For other details about how the above technical solutions are implemented by each module in the above malicious crawler identification and display system, please refer to the description of the malicious crawler identification and display method provided in the above invention embodiment, which will not be repeated here.

[0074] Based on the above malicious crawler identification and display method, such as Figure 6 As shown, an embodiment of the present invention further provides a schematic diagram of the structure of a malicious crawler identification and display device, wherein the device 6 includes a processor 61 and a memory 62 coupled to the processor 61. The memory 62 stores a computer program, and when the computer program is executed by the processor 61, the processor 61 performs the steps of the malicious crawler identification and display method in the above embodiment.

[0075] For other details about how the processor 61 in the malicious crawler identification and display device implements the above technical solution, please refer to the description of the malicious crawler identification and display method provided in the above invention embodiment, which will not be repeated here.

[0076] Among them, the processor 61 can also be called a CPU (Central Processing Unit), and the processor 61 may be an integrated circuit chip with signal processing capabilities; the processor 61 can also be a general-purpose processor, DSP (Digital Signal Process), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, among which the general-purpose processor can be a microprocessor or the processor 61 can also be any conventional processor, etc.

[0077] like Figure 7As shown, an embodiment of the present invention further provides a schematic diagram of the structure of a computer-readable storage medium, on which a readable computer program 71 is stored; wherein the computer program 71 can be stored in the above-mentioned storage medium in the form of a software product, including a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk, a ROM (Read-Only Memory), a RAM (Random Access Memory), and other media that can store program code, or a terminal device such as a computer, server, mobile phone, or tablet.

[0078] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0079] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the present embodiment according to actual needs.

[0080] In addition, the functional modules in the various embodiments of the present application may be integrated into a single processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may be stored in a computer-readable storage medium.

[0081] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0082] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in accordance with the embodiments of the present application are fully or partially generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that can be stored by a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, hard disk, or tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0083] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used in the present application to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0084] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0086] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0088] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for identifying and displaying malicious crawlers, characterized in that: include: Obtain request information related to an IP address within a certain time range. The characteristic elements of the request information include the number of requests, the percentage of request times, the number of abnormal requests, the percentage of abnormal requests, the number of failed requests, the percentage of failed requests, the maximum session duration, and the maximum session duration request speed. Determine whether the request related to each IP address is an abnormal request based on any preset threshold of a characteristic element in the request information or two or more thresholds of a combination of any associated characteristic elements, and generate a first cascade display area to display the request information related to each IP address and the abnormality result in the first cascade display area; Filter non-abnormal requests and generate a second cascade display area based on the IP address of the abnormal request and the request information under the IP address; For each abnormal request, a third cascade display area is generated under the IP address, and the third cascade display area is used to display the IP portrait of the request information under the IP address; Obtain the interface information under the abnormal IP address and generate the fourth cascade display area; Based on the log information of the IP address or interface under the abnormal request, a fifth cascade display area is generated for querying and verifying the abnormal request information.

2. The method for identifying and displaying malicious crawlers according to claim 1, characterized in that: The content displayed in the first cascade display area includes IP address information within a certain time range and the following list under the IP address: number of requests, percentage of request times, number of abnormal requests, percentage of abnormal requests, number of failed requests, percentage of failed requests, maximum session duration, maximum session duration request speed and abnormal indicator identification.

3. The method for identifying and displaying malicious crawlers according to claim 2, wherein: Whether the request related to each IP address is an abnormal request is determined based on any pre-set characteristic element threshold in the request information or two or more thresholds after the combination of any related characteristic elements, including setting thresholds for the number of requests, the proportion of request times, the number of abnormal requests, the proportion of abnormal requests, the number of failed requests, the proportion of failed requests, and the maximum session duration in the request information. When any characteristic element threshold or any one of the two or more thresholds after the combination of any related characteristic elements is not met, it is determined to be an abnormal request.

4. The method for identifying and displaying malicious crawlers according to claim 3, wherein: The characteristic elements of the request information also include sensitive words. Correspondingly, whether the request related to each IP address is an abnormal request is judged based on any preset characteristic element threshold in the request information or two or more thresholds after the combination of any associated characteristic elements, including only comparing the sensitive words in the request information with the preset sensitive words to determine whether the request is an abnormal request.

5. The method for identifying and displaying malicious crawlers according to claim 1, wherein: The IP profile includes basic information of the IP requesting information, threat conditions, and traffic trend charts. The basic information of the IP profile includes time information and IP address information. The threat conditions include IP traffic information, specifically including the number of request anomalies, interface type, UA type, reference type, user account type, maximum session duration, request anomaly ratio, interface repetition rate, UA repetition ratio, reference repetition ratio, and user account repetition ratio. The traffic trend chart is a traffic trend chart of related indicators in the threat traffic situation on the timeline.

6. The method for identifying and displaying malicious crawlers according to claim 1, wherein: The display in the fourth cascade display area includes the IP address under the abnormal indicator and the corresponding list of the IP address. The list name includes the interface name, interface code, interface priority, whether it is overstepping authority, whether it is sensitive, number of requests, proportion of request times, daily request peak ratio, number of request anomalies, and abnormal indicator identification.

7. The method for identifying and displaying malicious crawlers according to claim 1, wherein: The fifth cascade display area includes generating different log time distribution graphs and log detail distribution lists according to different query condition combinations, and the query conditions include time, interface information, IP address, regional address, and requester information.

8. A malicious crawler identification and display system, characterized in that: include: A request information acquisition unit, configured to acquire request information related to an IP address within a certain time range, wherein characteristic elements of the request information include the number of requests, the percentage of request times, the number of abnormal requests, the percentage of abnormal requests, the number of failed requests, the percentage of failed requests, the maximum session duration, and the maximum session duration request speed; A first cascade display area generating unit, configured to determine whether a request associated with each IP address is an abnormal request based on a preset threshold value of any characteristic element in the request information or two or more threshold values formed by combining any associated characteristic elements, and to generate a first cascade display area to display the request information associated with each IP address and the abnormality determination result in the first cascade display area; A second cascade display area generating unit is used to filter non-abnormal requests and generate a second cascade display area according to the IP address of the abnormal request and the request information under the IP address; A third cascade display area generating unit is used to generate a third cascade display area under the IP address for each abnormal request, wherein the third cascade display area is used to display the IP portrait of the request information under the IP address; A fourth cascade display area generating unit, configured to obtain interface information under an abnormal IP address and generate a fourth cascade display area; The fifth cascade display area generating unit is used to generate the fifth cascade display area according to the log information of the IP address or interface under the abnormal request, and query and verify the abnormal request information.

9. An electronic device capable of identifying and displaying malicious crawlers, characterized in that: comprising a memory and a processor, wherein: The memory is used to store computer programs; The processor is used to read the computer program in the memory and execute the malicious crawler identification and display method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: A readable computer program is stored thereon, and when the program is executed by a processor, the malicious crawler identification and display method as described in any one of claims 1-7 is implemented.

Images