Malicious crawler identification display method and system, electronic equipment and storage medium

Abstract

The invention discloses a malicious crawler identification and display method and system, electronic equipment and a storage medium. The method comprises the steps of obtaining feature elements of request information related to an IP address within a certain time range; judging whether the request related to each IP address is an abnormal request or not according to any feature element threshold value in preset request information or two or more threshold values after any associated feature element combination, and generating a first cascade display area; non-abnormal requests are filtered, and a second cascade display area is generated; for each abnormal request, generating a third cascade display area, obtaining interface information under the abnormal IP address, and generating a fourth cascade display area; and generating a fifth cascade display area according to the log information of the IP address or the interface under the abnormal request. According to the method, fine-grained index monitoring analysis can be rapidly carried out on various request information indexes, the malicious crawler identification accuracy is improved, checking is more visual, the identification efficiency is improved, and the data security is maintained.

Description

Technical Field

[0001] The present invention relates to the technical field of malicious crawlers, and particularly to a method, system, electronic device and storage medium for monitoring and displaying malicious crawlers. Background Art

[0002] In recent years, with the rapid development of Internet technology, web crawler technology has been widely used in various fields, such as search engines, data analysis, business intelligence, etc. However, with the popularization of web crawler technology, malicious crawler activities have become increasingly rampant. Malicious crawlers illegally obtain and abuse website data through automated means, resulting in frequent problems such as data leakage and infringement of user privacy, seriously affecting the data security of enterprises. It greatly damages the security of enterprises and the user experience.

[0003] Currently, when anti-crawler technology identifies crawler behavior, it generally judges whether it is a malicious crawler behavior through the numerical value of the access traffic of the corresponding IP address, that is, a single index of click traffic value. Identifying crawler behavior through a rough access traffic value will result in misjudgment of malicious crawler behavior. This is because malicious crawlers usually adopt disguise and avoidance techniques. They may disguise themselves as normal users to access the website, or avoid anti-crawler monitoring by modifying the request header, using proxy IPs, etc. The identification and analysis are difficult, posing a threat to data security. In addition, it is also impossible to intuitively display various indicators of crawler behavior and their associated information and other fine-grained analysis indicators to users. Users cannot quickly understand the business situation, which may ultimately lead to data leakage. Summary of the Invention

[0004] The present invention provides a method, system, electronic device and storage medium for identifying and displaying malicious crawlers, aiming to solve the problems in the existing anti-crawler technology, such as inaccurate crawler identification, low identification efficiency, and no comprehensive indicator display, resulting in users being unable to quickly understand the business situation.

[0005] To solve the above technical problems, in a first aspect, the present invention provides a method for identifying and displaying malicious crawlers, including: Obtain request information related to an IP address within a certain time range. The characteristic elements of the request information include the number of requests, the proportion of the number of requests, the number of abnormal requests, the proportion of abnormal requests, the number of failed requests, the proportion of failed requests, the maximum session duration, and the request speed of the maximum session duration. Judge whether the requests related to each IP address are abnormal requests according to any threshold of the request information set in advance or two or more thresholds after combining any associated characteristic elements, and generate a first cascading display area. Display the request information related to each IP address and the result of whether it is abnormal through the first cascading display area. Filter non-exception requests, and generate a second-level cascade display area based on the IP address where the exception request is located and the request information under this IP address; For each exception request, generate a third-level cascade display area under this IP address, and the third-level cascade display area is used to display the IP portrait of the request information under this IP address; Obtain the interface information under the exception IP address and generate a fourth-level cascade display area; Generate a fifth-level cascade display area based on the log information of the IP address or interface under the exception request, which is used for querying and verifying exception request information.

[0006] Optionally, the content displayed in the first-level cascade display area includes IP address information within a certain time range and the following list under this IP address: number of requests, proportion of request times, number of request exceptions, proportion of request exceptions, number of request failures, proportion of request failures, maximum session duration, maximum session duration request speed information, and exception index identifier.

[0007] Optionally, judge whether the requests related to each IP address are exception requests according to any feature element threshold in the preset request information or two or more thresholds after combining any associated feature elements. Specifically, set thresholds for the number of requests, proportion of request times, number of request exceptions, proportion of request exceptions, number of request failures, proportion of request failures, and maximum session duration in the request information respectively. When any one of the thresholds of any feature element or two or more thresholds after combining any associated feature elements is not met, it is judged as an exception request.

[0008] Optionally, the feature elements of the request information further include sensitive words. Correspondingly, judge whether the requests related to each IP address are exception requests according to any feature element threshold in the preset request information or two or more thresholds after combining any associated feature elements. It also includes comparing only the sensitive words in the request information with the preset sensitive words to judge whether the request is an exception request.

[0009] Optionally, the IP portrait includes the basic information of the IP, threat situation, and traffic trend chart.

[0010] Optionally, the fourth-level cascade display area displays the IP address under the exception index and the corresponding list of this IP address. The list names include interface name, interface code, interface priority, whether there is over-privilege, whether it involves sensitivity, number of requests, proportion of request times, peak ratio of daily request volume, number of request exceptions, and exception index identifier.

[0011] Optionally, the fifth cascaded display area includes different log time distribution graphs and log detail distribution lists generated according to different combinations of query conditions, where the different query conditions include time, interface information, IP address, regional address, and requester information.

[0012] In a second aspect, the present invention provides a malicious crawler recognition and display system, including: A request information acquisition unit, configured to acquire request information related to an IP address within a certain time range, where the characteristic elements of the request information include the number of requests, the proportion of the number of requests, the number of abnormal requests, the proportion of abnormal requests, the number of failed requests, the proportion of failed requests, the maximum session duration, and the request speed of the maximum session duration; A first cascaded display area generation unit, configured to determine whether the requests related to each IP address are abnormal requests according to any characteristic element threshold in the preset request information or two or more thresholds after combining any associated characteristic elements, and generate a first cascaded display area, and display the request information related to each IP address and the result of whether it is abnormal through the first cascaded display area; A second cascaded display area generation unit, configured to filter non-abnormal requests and generate a second cascaded display area according to the IP address where the abnormal request is located and the request information under that IP address; A third cascaded display area generation unit, configured to generate a third cascaded display area under that IP address for each abnormal request, and the third cascaded display area is used to display the IP portrait of the request information under that IP address; A fourth cascaded display area generation unit, configured to acquire interface information under the abnormal IP address and generate a fourth cascaded display area; A fifth cascaded display area generation unit, configured to generate a fifth cascaded display area according to the log information of the IP address or interface under the abnormal request, and query and verify the abnormal request information.

[0013] In a third aspect, the present invention provides a malicious crawler recognition and display device, including a memory and a processor, where: The memory is used to store computer programs; The processor is configured to read the computer program in the memory and execute the steps of the malicious crawler recognition and display method provided in the first aspect above.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a readable computer program is stored, and when the program is executed by a processor, it implements the steps of the malicious crawler recognition and display method provided in the first aspect above.

[0015] Compared with the prior art, the malicious crawler recognition and display method, system, electronic device, and storage medium provided by the present invention have the following beneficial effects: By obtaining request information related to an IP address within a certain time range, the characteristic elements of the request information include the number of requests, the proportion of the number of requests, the number of abnormal requests, the proportion of abnormal requests, the number of failed requests, the proportion of failed requests, the maximum session duration, the request speed of the maximum session duration; judging whether the requests related to each IP address are abnormal requests according to any characteristic element threshold in the preset request information or two or more thresholds after combining any associated characteristic elements, and generating a first cascading display area, and displaying the request information related to each IP address and the result of whether it is abnormal through the first cascading display area; filtering non-abnormal requests, and generating a second cascading display area according to the IP address where the abnormal request is located and the request information under the IP address; for each abnormal request, generating a third cascading display area under the IP address, and the third cascading display area is used to display the IP portrait of the request information under the IP address; obtaining interface information under the abnormal IP address and generating a fourth cascading display area; generating a fifth cascading display area according to the log information of the IP address or interface under the abnormal request for query and verification of abnormal request information. The technical solution can quickly perform fine-grained index monitoring and analysis on various request information indicators, improve the accuracy of malicious crawler recognition, make the analysis and viewing more intuitive, make the request indicators more diverse, enable users to quickly insight into the business situation, improve the recognition efficiency, prevent the leakage of malicious crawlers, and maintain data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only a part of the embodiments of the present invention, rather than all of the embodiments. For those of ordinary skill in the art, without creative efforts, other drawings obtained based on these drawings all belong to the scope protected by this application.

[0017] Figure 1 is a flowchart of a malicious crawler recognition and display method provided by an embodiment of the present invention; Figure 2 is a schematic diagram of a user interface displayed under an index dashboard provided by an embodiment of the present invention; Figure 3 is a schematic diagram of an IP portrait provided by an embodiment of the present invention; Figure 4 is a schematic diagram of the display of the fourth cascading display area provided by an embodiment of the present invention; Figure 5 is a schematic diagram of the structure of a malicious crawler recognition and display system provided by an embodiment of the present invention; Figure 6It is a schematic structural diagram of a malicious crawler recognition and display electronic device provided by an embodiment of the present invention; Figure 7 It is a schematic structural diagram of a computer-readable storage medium provided by an embodiment of the present invention. Specific embodiments

[0018] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0019] To make the description of the present disclosure more detailed and complete, the following presents an illustrative description of the embodiments and specific examples of the present invention; however, this is not the only form for implementing or applying the specific examples of the present invention. The embodiments cover the features of multiple specific examples and the method steps and their sequences for constructing and operating these specific examples. However, other specific examples can also be used to achieve the same or equivalent functions and step sequences. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

[0020] Embodiment 1 To achieve the objectives of the present invention, Apache Flink is used as the cornerstone of real-time computing for data collection and calculation in this embodiment, Kafka is used as the middleware, and the data is landed in the ad-hoc query StarRocks. Through the ad-hoc query ability of StarRocks combined with the pre-computation and window calculation abilities of Flink, the corresponding index calculations are controlled within second-level responses, providing efficient queries for users' real-time observation of abnormal requests. At the same time, it also supports real-time configuration of rules to dynamically mark and tag data, facilitating subsequent analysis and corresponding calculation processing of the tag content. Apache Flink is an open-source stream processing framework developed by the Apache Software Foundation, and its core is a distributed stream data flow engine written in Java and Scala. Flink executes any stream data program in a data-parallel and pipelined manner, and the pipeline runtime of Flink can execute batch processing and stream processing programs.

[0021] As Figure 1 shown, a malicious crawler recognition and display method provided by an embodiment of the present invention includes: S101, obtaining request information related to an IP address within a certain time range, where the characteristic elements of the request information include the number of requests, the proportion of the number of requests, the number of abnormal requests, the proportion of abnormal requests, the number of failed requests, the proportion of failed requests, the maximum session duration, and the request speed of the maximum session duration; In this step, according to actual needs, the time range for obtaining data can be set, and the time can be accurate to seconds. The characteristic elements of the request information related to the IP address obtained are as shown above. Among them, the number of requests, the number of abnormal requests, the number of failed requests, the maximum session duration, and the request speed of the maximum session duration can all be presented directly as numbers. For example, the number of requests is 1000 times, that is, within this time period, the number of requests for this IP address is 1000 times. The proportion of requests, the proportion of abnormal requests, and the proportion of failed requests can be presented in the form of a number + percentage. For example, the percentage of abnormal requests is the number of abnormal requests / the number of requests × 100%.

[0022] S102. Determine whether the requests related to each IP address are abnormal requests according to any characteristic element threshold in the preset request information or two or more thresholds after combining any associated characteristic elements, and generate a first-level cascading display area, and display the request information related to each IP address and the result of whether it is abnormal through the first-level cascading display area; As an optional implementation manner, determining whether the requests related to each IP address are abnormal requests according to any characteristic element threshold in the preset request information or two or more thresholds after combining any associated characteristic elements includes setting thresholds for the number of requests, the proportion of requests, the number of abnormal requests, the proportion of abnormal requests, the number of failed requests, the proportion of failed requests, and the maximum session duration in the request information respectively. When any one of the thresholds of any characteristic element or two or more thresholds after combining any associated characteristic elements is not met, it is determined as an abnormal request.

[0023] Exemplarily, within a certain time period, if the number of requests exceeds 1000 times, the proportion of requests exceeds 0.5%, the number of failed requests exceeds 100 times, and the maximum session duration exceeds 100 minutes, and there are 5 times when the maximum session duration exceeds 100 minutes within this time period, it is set as abnormal. In addition, the user can further observe the proportion of abnormal requests and the proportion of failed requests to master the fine-grained multi-dimensional information of abnormal requests, which is convenient for mastering the overall situation of abnormalities.

[0024] For another example, within a certain time period, if the maximum session duration exceeds 100 minutes, it is set as abnormal regardless of the situation of other indicators. After setting this single indicator as abnormal, the user can observe other indicators such as the number of requests, the proportion of requests, the number of abnormal requests, the proportion of abnormal requests, the proportion of failed requests, and the number of times the maximum session duration exceeds 100 minutes to master the fine-grained multi-dimensional information of abnormal requests, which is convenient for mastering the overall situation of abnormalities.

[0025] For another example, within a certain time period, if the number of requests exceeds 1000, the proportion of requests exceeds 0.5%, the number of failed requests exceeds 100, and the maximum session duration exceeds 100 minutes, and within this time period, if both the number of failed requests exceeds 100 and the maximum session duration exceeds 100 minutes are satisfied, it is set as an anomaly, regardless of other characteristic factors. Additionally, the user can further observe other characteristic factors such as the proportion of abnormal requests and the proportion of failed requests to grasp the fine-grained multi-dimensional information of abnormal requests and facilitate understanding the overall situation of anomalies.

[0026] As an alternative implementation, the characteristic elements of the request information further include sensitive words. Correspondingly, according to a preset threshold for any characteristic element in the request information or two or more thresholds after combining any associated characteristic elements, it is determined whether the requests related to each IP address are abnormal requests. It also includes comparing only the sensitive words in the request information with the preset sensitive words to determine whether the request is an abnormal request. Exemplarily, as long as the sensitive words involve "customer data", or for example, the waybill number starts with "AS", as long as such sensitive words are identified, it is determined as abnormal.

[0027] As an alternative implementation, the content displayed in the first cascading display area includes IP address information within a certain time range and the following list under this IP address: the number of requests, the proportion of requests, the number of abnormal requests, the proportion of abnormal requests, the number of failed requests, the proportion of failed requests, the maximum session duration, the request speed information of the maximum session duration.

[0028] Exemplarily, the first cascading display area is displayed through a user interaction interface under an index dashboard as Figure 2 shown. On the left side of the user interaction interface of this index dashboard is a function selection box, the upper right part is an indication window such as time, interface label, etc., and the lower right part is the corresponding result display under the left function selection box. For example, the request information related to the IP address in this embodiment and the result of whether it is abnormal. The function selection box includes index dashboard, IP portrait, log query, alarm management, abnormal metrics, alarm configuration, alarm handling, label management, label definition, interface classification, sensitive fields, incremental fields, management background, user management, etc. For example, when selecting the index dashboard selection box, there will be a display as Figure 2For the user interface shown, after entering the time period to be monitored in the upper right part on the right side of the indicator dashboard, the IP analysis display area will appear in the lower right part. For example, if the time period for obtaining data is from 18:40:15 on June 28, 2024 to 20:40:15 on June 28, 2024, the IP analysis display area includes a serial number column, an IP address column, a request count column, a request count percentage column, a request exception count column, a request exception percentage column, a request failure count column, a request failure percentage column, a maximum session duration column, and a maximum session duration request speed column. Each row contains the relevant request information for the IP address, including the values of the request count, request count percentage, request exception count, request exception percentage, request failure count, request failure percentage, maximum session duration, and maximum session duration request speed.

[0029] Exemplarily, after identifying sensitive words, through the sensitive column identifier in the first cascading display area, while outputting an exception indication, the IP address and the request information such as the request count, request count percentage, request exception count, request exception percentage, request failure count, request failure percentage, and maximum session duration under it can be displayed through the first cascading display area, facilitating the user to master multi-dimensional information in this sensitive exception situation.

[0030] Obtain the fine-grained multi-dimensional detailed information of the access object, and display the request information under each IP address with fine-grained multi-dimensions to the user through the first cascading display area, facilitating the user to gain insights into the access situation, and also providing a more accurate and detailed data source for the subsequent judgment of malicious crawlers, providing a judgment basis for improving the accuracy of malicious crawler judgment.

[0031] S103, Filter non-exception requests, and generate a second cascading display area according to the IP address where the exception request is located and the request information under this IP address; In order to further and more detailedly master the exception request situation and save the page space of the display area, filter out the information that does not belong to the exception according to the methods of steps S101 and S102, and only display the exception request information in the second cascading display area.

[0032] An exception index chart box is set on the interface of the first cascading display area, and the second cascading display area can be accessed by clicking on this exception index chart box. The content displayed in the second cascading display area is the request information displayed in the first cascading display area in an abnormal state.

[0033] S104, For each exception request, generate a third cascading display area under this IP address, and the third cascading display area is used to display the IP portrait of the request information under this IP address; Enter the third - level cascade display area through the IP portrait in the function selection box on the left side of the indicator dashboard electronic user interface. The IP portrait is the drill - down function of a single IP in the indicator dashboard, which can display in detail the basic information of the IP, threat situation, traffic trend chart, and indicator lists in other dimensions. Users can comprehensively analyze and judge the request behavior of the IP by combining the request situation of abnormal IPs.

[0034] As an optional implementation manner, the IP portrait includes the basic information of the IP, threat situation, and traffic trend chart. As Figure 3 shown, the basic information of the IP portrait includes time information and IP address information. The threat situation includes IP traffic information, specifically including the number of abnormal requests, interface types, UA types, Referer types, user account types, maximum session duration, abnormal request ratio, interface duplication degree, UA duplication ratio, Referer duplication ratio, user account duplication ratio, etc.; the traffic trend chart is a display of the traffic trend pulsation chart of relevant indicators in the threat traffic situation on the time axis. Of course, it can also be a chart or other graphical display.

[0035] Through the IP portrait, users can more intuitively analyze and understand the situation of IP request information.

[0036] S105, Obtain the interface information under the abnormal IP address and generate a fourth - level cascade display area; As an optional implementation manner, the fourth - level cascade display area is cascaded in through the interface label (not shown in the figure) at the lower right corner of the electronic user interaction interface of the third - level cascade display area. As Figure 4 shown, the fourth - level cascade display area displays the IP address under the abnormal indicator and the corresponding list of this IP address. The list names include interface name, interface code, interface priority, whether there is unauthorized access, whether it involves sensitive information, number of requests, request ratio, peak ratio of daily request volume, number of abnormal requests, abnormal indicator identification. By obtaining the above - mentioned interface information under the abnormal IP address, the network communication channel of the abnormal IP address can be better understood, providing a more accurate information source for subsequent abnormal information processing.

[0037] S106, Obtain the log information of the IP address or interface under the abnormal request and generate a fifth - level cascade display area.

[0038] Enter the fifth - level cascade display area through the log query window in the indicator dashboard for log query, which is used for query and verification of abnormal request information.

[0039] As an optional implementation manner, the fifth - level cascade display area includes different log time distribution charts and log detail distribution lists generated according to different combinations of query conditions. The different query conditions include time, interface information, IP address, regional address, personnel information, etc.

[0040] Exemplarily, by querying the pulsation diagram of the log time distribution and the log details distribution list output by the personnel management application, where the log details distribution list includes field information such as the specific time of event occurrence, IP address, region, interface, interface name, application name (personnel management application), etc. Additionally, the fifth cascading display area further includes a field retrieval box, through which fields appearing in the list can be selected to hide field information that does not need to appear.

[0041] Through the display in the fifth cascading display area, arbitrary combined traceability analysis of log data can be realized, and corresponding fields are retrieved and appear in the list, quickly, effectively, and intuitively meeting the user's needs while saving data storage resources and improving the query speed.

[0042] It should be noted that the metric dashboard is implemented based on starrocks ad-hoc queries. By dividing the entire metric system into regular statistical categories, deduplication statistical categories, and session window statistical categories, different categories respectively adopt real-time calculation and pre-calculation of flink. Among them, for regular statistical categories, such as IP request information, real-time calculation is adopted. For deduplication statistical categories, the IP request information is deduplicated and real-time statistics of the detailed list is adopted. Considering performance, the deduplication statistical category is optimized and implemented using starrocks bitmap. For pre-aggregated tables, such as the formation of various form display areas, flink calculates and outputs the corresponding window values and session data in the pre-stage. Finally, more than 15 dimensions and more than 100 metrics can be constructed.

[0043] Implementation Case 2 Based on the above malicious crawler recognition and display method, an embodiment of the present invention provides a malicious crawler recognition and display system, as Figure 5 shown.

[0044] Regarding other details of each module in the above malicious crawler recognition and display system for implementing the above technical solutions, reference can be made to the description in the malicious crawler recognition and display method provided in the above embodiments of the invention, which will not be elaborated here.

[0045] Based on the above malicious crawler recognition and display method, as Figure 6 shown, an embodiment of the present invention further provides a structural schematic diagram of an electronic device for malicious crawler recognition and display. The device 6 includes a processor 61 and a memory 62 coupled to the processor 61. The memory 62 stores a computer program, and when the computer program is executed by the processor 61, the processor 61 is caused to execute the steps of the malicious crawler recognition and display method in the above embodiments.

[0046] For other details of how the processor 61 in the above malicious crawler recognition and display device implements the above technical solution, reference may be made to the description in the malicious crawler recognition and display method provided in the above invention embodiment, which will not be elaborated here.

[0047] Among them, the processor 61 can also be called a CPU (Central Processing Unit, central processing unit). The processor 61 may be an integrated circuit chip with signal processing capabilities; the processor 61 can also be a general-purpose processor, DSP (Digital Signal Process, digital signal processor), ASIC (Application Specific Integrated Circuit, application-specific integrated circuit), FPGA (Field Programmable Gata Array, field-programmable gate array), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Among them, the general-purpose processor can be a microprocessor, or the processor 61 can also be any conventional processor, etc.

[0048] As Figure 7 shown, the embodiment of the present invention also provides a schematic structural diagram of a computer-readable storage medium. A readable computer program 71 is stored on the storage medium 7; among them, the computer program 71 can be stored in the above storage medium in the form of a software product, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: USB flash drives, mobile hard disks, magnetic disks or optical discs, ROM (Read-Only Memory, read-only memory), RAM (Random Access Memory, random access memory), and other media that can store program codes, or terminal devices such as computers, servers, mobile phones, and tablets.

[0049] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or modules can be in electrical, mechanical, or other forms.

[0050] The module described as a separation component may or may not be physically separated. The component shown as a module may or may not be a physical module, that is, it may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0051] In addition, in each embodiment of this application, the various functional modules can be integrated in a processing module, or each module can exist physically alone, or two or more modules can be integrated in one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0052] In the above embodiment, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0053] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are generated in whole or in part. The computer can be a general-purpose computer, a dedicated computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0054] The technical solutions provided in this application have been introduced in detail above. Specific examples are used in this application to elaborate on the principles and implementation methods of this application. The descriptions of the above embodiments are only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, based on the idea of this application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.

[0055] Those skilled in the art will appreciate that the embodiments of the present application may be provided as a method, a system, or a computer program product. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0056] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0057] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that realizes the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0058] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for realizing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0059] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

1. A method for identifying and displaying malicious crawlers, characterized in that, Including: Obtain request information related to an IP address within a certain time range. The characteristic elements of the request information include the number of requests, the proportion of the number of requests, the number of abnormal requests, the proportion of abnormal requests, the number of failed requests, the proportion of failed requests, the maximum session duration, and the request speed of the maximum session duration; Judge whether the requests related to each IP address are abnormal requests according to any characteristic element threshold in the preset request information or two or more thresholds after combining any associated characteristic elements, and generate a first-level cascade display area. Display the request information related to each IP address and the result of whether it is abnormal through the first-level cascade display area; Filter non-abnormal requests, and generate a second-level cascade display area according to the IP address where the abnormal request is located and the request information under this IP address; For each abnormal request, generate a third-level cascade display area under this IP address. The third-level cascade display area is used to display the IP portrait of the request information under this IP address; Obtain the interface information under the abnormal IP address and generate a fourth-level cascade display area; Generate a fifth-level cascade display area according to the log information of the IP address or interface under the abnormal request, which is used for query and verification of abnormal request information.

2. The malicious crawler recognition and display method according to claim 1, characterized in that: The content displayed in the first-level cascade display area includes the IP address information within a certain period of time and the following list under this IP address: the number of requests, the proportion of the number of requests, the number of abnormal requests, the proportion of abnormal requests, the number of failed requests, the proportion of failed requests, the maximum session duration, the request speed of the maximum session duration, and the abnormal index identifier.

3. The malicious crawler recognition and display method according to claim 2, characterized in that: Judging whether the requests related to each IP address are abnormal requests according to any characteristic element threshold in the preset request information or two or more thresholds after combining any associated characteristic elements includes setting thresholds for the number of requests, the proportion of the number of requests, the number of abnormal requests, the proportion of abnormal requests, the number of failed requests, the proportion of failed requests, and the maximum session duration in the request information respectively. When any one of the thresholds of any characteristic element or two or more thresholds after combining any associated characteristic elements is not met, it is judged as an abnormal request.

4. The malicious crawler recognition and display method according to claim 3, wherein: The characteristic elements of the request information also include sensitive words. Correspondingly, judging whether the requests related to each IP address are abnormal requests according to any characteristic element threshold in the preset request information or two or more thresholds after combining any associated characteristic elements includes only comparing the sensitive words in the request information with the preset sensitive words to judge whether the request is an abnormal request.

5. The malicious crawler recognition and display method according to claim 1, wherein: The IP portrait includes the basic information of the IP of the request information, the threat situation, and the traffic trend chart. The basic information of the IP portrait includes time information and IP address information. The threat situation includes IP traffic information, specifically including the number of abnormal requests, the type of interfaces, the type of UAs, the type of references, the type of user accounts, the maximum session duration, the proportion of abnormal requests, the interface repetition degree, the UA repetition ratio, the reference repetition ratio, and the user account repetition ratio; The traffic trend chart is a traffic trend chart of relevant indicators in the threat traffic situation on the time axis.

6. The malicious crawler recognition and display method according to claim 1, characterized in that: The display in the fourth cascading display area includes the IP address under the abnormal index and the corresponding list of the IP address. The list names include interface name, interface code, interface priority, whether there is unauthorized access, whether it involves sensitive information, number of requests, percentage of requests, peak ratio of daily request volume, number of abnormal requests, and abnormal index identifier.

7. The malicious crawler recognition and display method according to claim 1, wherein: The fifth cascading display area includes different log time distribution diagrams and log detail distribution lists generated according to different combinations of query conditions. The query conditions include time, interface information, IP address, regional address, and requestor information.

8. A malicious crawler recognition and display system, characterized in that It includes: A request information acquisition unit for acquiring request information related to an IP address within a certain time range. The characteristic elements of the request information include the number of requests, percentage of requests, number of abnormal requests, percentage of abnormal requests, number of failed requests, percentage of failed requests, maximum session duration, request speed of the maximum session duration request; A first cascading display area generation unit for determining whether the requests related to each IP address are abnormal requests according to any threshold of the pre-set characteristic elements in the request information or two or more thresholds after combining any related characteristic elements, and generating a first cascading display area, and displaying the request information related to each IP address and the result of whether it is abnormal through the first cascading display area; A second cascading display area generation unit for filtering non-abnormal requests and generating a second cascading display area according to the IP address where the abnormal request is located and the request information under the IP address; A third cascading display area generation unit for generating a third cascading display area under the IP address for each abnormal request. The third cascading display area is used to display the IP portrait of the request information under the IP address; A fourth cascading display area generation unit for acquiring interface information under the abnormal IP address and generating a fourth cascading display area; A fifth cascading display area generation unit for generating a fifth cascading display area according to the log information of the IP address or interface under the abnormal request, and querying and verifying the abnormal request information.

9. A malicious crawler recognition and display electronic device, characterized in that: It includes a memory and a processor, where: The memory is used to store computer programs; The processor is used to read the computer program in the memory and execute the malicious crawler identification and display method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: There is a readable computer program stored thereon. When the program is executed by the processor, it implements the malicious crawler identification and display method as described in any one of claims 1-7.

Images