A webpage data processing method and device, electronic equipment and storage medium

Abstract

The application provides a webpage data processing method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining hyperlink information and text information in a current webpage; a website address of the current webpage is stored in a target website address list, and a page corresponding to each website address in the target website address list is used to obtain malicious indicators; malicious indicators are extracted from the text information; if malicious indicators are extracted from the text information, a website address corresponding to the hyperlink information is added to the target website address list. The application realizes automatic expansion of the target website address list according to the content in the current webpage, and avoids the problem that the number of obtained malicious indicators is small and incomplete due to the acquisition of malicious indicators from only a limited range of webpages.

Description

Technical Field

[0001] This application relates to the field of secure data processing, and more particularly to a web page data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Threat intelligence is primarily used to identify and detect compromised indicators (i.e., malicious IOCs) of attack activities. To obtain target IOCs, many users acquire them from their own collected data sources or from web pages. However, due to the wide range of intelligence data sources, relying solely on self-collected data sources or web pages limits the scope of acquisition, easily resulting in a small and incomplete number of malicious IOCs obtained. Summary of the Invention

[0003] In view of this, this application provides a web page data processing method, apparatus, electronic device, and storage medium, which at least partially solves the problems existing in the prior art.

[0004] In one aspect of this application, a web page data processing method is provided, comprising:

[0005] Obtain hyperlink information and text information within the current webpage; the URL of the current page is stored in a target URL list, and the page corresponding to each URL in the target URL list is used to obtain malicious indicators;

[0006] Malicious indicators are extracted from the text information;

[0007] If malicious indicators are extracted from the text information, the URLs corresponding to the hyperlinks will be added to the target URL list.

[0008] In one exemplary embodiment of this application, the target URL list includes an visited list and an unvisited list;

[0009] Adding the URL corresponding to the hyperlink information to the target URL list includes:

[0010] Determine whether the URL corresponding to the hyperlink information exists in the visited list;

[0011] If it does not exist, the URL corresponding to the hyperlink information is added to the unvisited list.

[0012] In one exemplary embodiment of this application, the method further includes:

[0013] If no malicious indicators are obtained from the webpages containing the stored URLs within the set time period, the stored URLs will be removed from the target URL list.

[0014] The stored URL can be any URL in the target URL list.

[0015] In one exemplary embodiment of this application, the extraction of malicious indicators from the text information includes:

[0016] Candidate indicators are obtained from the text information;

[0017] The candidate indicators are input into the first judgment model and the second judgment model respectively to obtain the first judgment result and the second judgment result; both the first judgment model and the second judgment model are used to determine whether the candidate indicator is a malicious indicator, and the model types of the first judgment model and the second judgment model are different;

[0018] Whether to mark the candidate indicator as a malicious indicator is determined based on the first judgment result and / or the second judgment result.

[0019] In one exemplary embodiment of this application, the first judgment model is a heuristic classifier, and the second judgment model is a machine learning classifier;

[0020] The step of determining whether to mark the candidate indicator as a malicious indicator based on the first judgment result and / or the second judgment result includes:

[0021] Determine whether the first and second judgment results are the same;

[0022] If they are the same, then determine whether to mark the candidate indicator as a malicious indicator based on the first judgment result and / or the second judgment result;

[0023] If they are not the same, then obtain the first probability and the second probability; where the first probability is the accuracy of the first judgment model output for the first judgment result, and the second probability is the accuracy of the second judgment model output for the second judgment result.

[0024] When the difference between the first probability and the second probability is greater than a preset threshold, if the first probability is greater than the second probability, then it is determined whether to mark the candidate indicator as a malicious indicator based on the first judgment result; otherwise, it is determined whether to mark the candidate indicator as a malicious indicator based on the second judgment result.

[0025] When the difference between the first probability and the second probability is less than or equal to a preset threshold, a second judgment result is used to determine whether the candidate indicator is marked as a malicious indicator.

[0026] In one exemplary embodiment of this application, obtaining candidate indicators from the text information includes:

[0027] The text information is segmented into sentences to obtain several character segments;

[0028] Feature extraction is performed on each character segment to obtain feature information corresponding to each character segment; the feature information includes at least one of the following: text features, data source features, source distribution features, sentence features, content features, and external features;

[0029] Based on the feature information corresponding to each character segment, determine whether each character segment contains candidate indicators.

[0030] In one exemplary embodiment of this application, before feature extraction of each character segment and after sentence segmentation of the text information, the method further includes:

[0031] Determine if a specified string exists in each character segment. If it does, perform character conversion processing on the specified string in the current character segment.

[0032] In another aspect of this application, a web page data processing apparatus is provided, comprising:

[0033] The acquisition module is used to acquire hyperlink information and text information within the current webpage; the URL of the current page is stored in the target URL list, and the page corresponding to each URL in the target URL list is used to acquire malicious indicators;

[0034] The extraction module is used to extract malicious indicators from the text information;

[0035] An add module is used to add the URL corresponding to the hyperlink information to the target URL list if malicious indicators are extracted from the text information.

[0036] In another aspect of this application, an electronic device is provided, including a processor and a memory;

[0037] The processor executes the steps of any of the methods described above by invoking programs or instructions stored in the memory.

[0038] In another aspect of this application, a non-transitory computer-readable storage medium is provided, which stores a program or instructions that cause a computer to perform the steps of any of the methods described above.

[0039] This application provides a webpage data processing method that, when accessing any webpage (i.e., the current webpage) within a target URL list, retrieves hyperlink information and text information from the current webpage. If malicious indicators are extracted from the text information, it can be determined that the text information on the current webpage explains or elaborates on the malicious indicators. In this case, the URL corresponding to the hyperlink information on the current webpage is highly likely to be a webpage referenced by the current page for explaining or elaborating on the malicious indicators. Therefore, adding the URL corresponding to the hyperlink information to the target URL list ensures that when malicious indicators are subsequently retrieved from pages corresponding to URLs in the target URL list, the URL corresponding to the hyperlink information will also be accessed and malicious indicators extracted. This achieves automatic expansion of the target URL list based on the content of the current webpage, avoiding the problem of obtaining a small and incomplete number of malicious indicators due to only retrieving malicious indicators from a limited range of webpages. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A flowchart illustrating a web page data processing method provided in this application embodiment;

[0042] Figure 2 This is a structural block diagram of a web page data processing device provided in an embodiment of this application. Detailed Implementation

[0043] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0044] It should be noted that, in the absence of conflict, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0045] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0046] Definitions:

[0047] Malicious indicators, or IOCs (short for Indicators of Compromise), refer to the indicators used by attackers to infect or compromise target devices. They are generally malicious indicators used by attackers, such as hash values, domain names, URLs, IP addresses, and email addresses that appear in intelligence data.

[0048] Please refer to Figure 1 As shown, in one aspect of this application, a web page data processing method is provided, comprising the following steps:

[0049] S100, Obtain hyperlink information and text information from the current webpage. The URL of the current page is stored in a target URL list, and the page corresponding to each URL in the target URL list is used to obtain malicious indicators. The hyperlink information and text information can be obtained through a web crawler. Specifically, when the web crawler accesses the current webpage, it first saves the original HTML file containing HTML, JavaScript, and CSS code, processes the HTML file and retrieves links from it, and then identifies the retrieved links as hyperlink information. The text information is all the text information displayed on the current page.

[0050] S200, Malicious indicators are extracted from the text information.

[0051] S300, if a malicious indicator is extracted from the text information, the URL corresponding to the hyperlink information is added to the target URL list.

[0052] The method provided in this application can be applied to any electronic device, such as a computer, laptop, or server. The electronic device executing this method is configured to access each URL in the target URL list at set intervals and attempt to obtain malicious indicators. Malicious indicator extraction can be achieved by performing feature analysis on text information to extract malicious indicators.

[0053] This embodiment provides a webpage data processing method. When accessing any webpage (i.e., the current webpage) within a target URL list, the method acquires hyperlink information and text information within the current webpage. If malicious indicators are extracted from the text information, it can be determined that the text information within the current webpage explains or elaborates on the malicious indicators. In this case, the URL corresponding to the hyperlink information within the current webpage is highly likely to be a webpage referenced by the current page for explaining or elaborating on the malicious indicators. Therefore, adding the URL corresponding to the hyperlink information to the target URL list ensures that when malicious indicators are subsequently acquired through pages corresponding to URLs in the target URL list, the URL corresponding to the hyperlink information will also be accessed and malicious indicators extracted. This achieves automatic expansion of the target URL list based on the content within the current webpage, avoiding the problem of obtaining a small and incomplete number of malicious indicators due to acquiring malicious indicators only from a limited range of webpages.

[0054] In one exemplary embodiment of this application, the target URL list includes an visited list and an unvisited list. Specifically, the electronic device implementing this method transfers all URLs from the visited list to the unvisited list at set intervals, and then begins accessing each URL in the unvisited list, extracting malicious indicators. The specific method for extracting malicious indicators will be described in detail later.

[0055] Adding the URL corresponding to the hyperlink information to the target URL list includes:

[0056] Determine whether the URL corresponding to the hyperlink information exists in the visited list.

[0057] If it exists, no further operations will be performed on the hyperlink information.

[0058] If it does not exist, the URL corresponding to the hyperlink information is added to the unvisited list.

[0059] In this embodiment, after obtaining malicious indicators from the text information of the current webpage, it is determined whether the URL corresponding to the hyperlink information exists in the visited list. If it exists, it means that the URL has already been included in the list, so there is no need to add it. If it does not exist, it means that the URL has not been visited before and is not included in the list, so it will be directly added to the unvisited list to complete its inclusion. At the same time, it can be accessed and malicious indicators extracted within the current set period.

[0060] In one exemplary embodiment of this application, the method further includes:

[0061] If no malicious indicators are obtained from the webpages of the stored URLs within the set time period, the stored URLs will be deleted from the target URL list. The set time period is the time during which malicious indicators are extracted from the corresponding stored URLs.

[0062] The stored URL can be any URL in the target URL list, and the set time period can be 5 to 60 days. In this embodiment, the set time period is 30 days.

[0063] Because new URLs are continuously added to the target URL list after using this method, this embodiment, to avoid missing URLs that may provide malicious indicators, will add the URLs corresponding to the hyperlinks on the current page as soon as malicious indicators are extracted. However, in some cases, the web pages corresponding to these URLs may only occasionally publish information about malicious indicators and will not publish it again in the future. In this case, if these URLs are not cleaned up, the electronic device will access and process these web pages every time it collects malicious indicators, which will undoubtedly increase the processing load significantly. Therefore, in this embodiment, if no malicious indicators are obtained from the web pages of the stored URLs within a set time period, it means that the currently stored URLs have not provided malicious indicator information for a long time, and therefore the stored URLs will be automatically deleted from the target URL list to reduce the processing load.

[0064] In one exemplary embodiment of this application, the extraction of malicious indicator C from the text information includes:

[0065] Candidate indicators are obtained from the text information.

[0066] The candidate indicators are input into the first judgment model and the second judgment model respectively to obtain the first judgment result and the second judgment result. Both the first judgment model and the second judgment model are used to determine whether the candidate indicator is a malicious indicator, and the first judgment model and the second judgment model are of different types.

[0067] Whether to mark the candidate indicator as a malicious indicator is determined based on the first judgment result and / or the second judgment result.

[0068] Candidate indicators can be understood as suspected malicious indicators, but they have not been definitively confirmed as malicious. The specific methods for obtaining candidate indicators from the aforementioned text information will be described in detail later, and will not be repeated here.

[0069] In this embodiment, after obtaining candidate indicators, each candidate indicator (which may also include the associated information of the current candidate indicator, such as context information) is sequentially input into two pre-trained judgment models. Based on the judgment results of the two models, it is determined whether the candidate indicator is marked as a malicious indicator. Using two judgment models allows for relevant verification based on the judgment results output by the two models, thereby improving the accuracy of identifying malicious indicators.

[0070] Specifically, the first judgment model is a heuristic classifier, and the second judgment model is a machine learning classifier.

[0071] The step of determining whether to mark the candidate indicator as a malicious indicator based on the first judgment result and / or the second judgment result includes:

[0072] Determine whether the first judgment result and the second judgment result are the same; both the first judgment result and the second judgment result can be "malicious indicator" or "non-malicious indicator".

[0073] If they are the same, then it is determined whether the candidate indicator is marked as a malicious indicator based on the first judgment result and / or the second judgment result.

[0074] If they are not the same, then the first probability and the second probability are obtained. The first probability is the accuracy rate of the first judgment model output for the first judgment result, and the second probability is the accuracy rate of the second judgment model output for the second judgment result.

[0075] When the difference between the first probability and the second probability is greater than a preset threshold, if the first probability is greater than the second probability, then it is determined whether the candidate indicator is marked as a malicious indicator based on the first judgment result; otherwise, it is determined whether the candidate indicator is marked as a malicious indicator based on the second judgment result. The preset threshold can be 10%-50%, and in this embodiment, the preset threshold is 20%.

[0076] When the difference between the first probability and the second probability is less than or equal to a preset threshold, a second judgment result is used to determine whether the candidate indicator is marked as a malicious indicator.

[0077] In this embodiment, if the first judgment result and the second judgment result are the same, either judgment result can be used directly as the basis to determine whether the candidate indicator is marked as a malicious indicator. However, when the two judgment results are different, the first probability and the second probability corresponding to the two judgment results will be used respectively. If the difference between the first probability and the second probability is large (i.e., greater than a preset threshold), it means that the judgment result with the lower probability is likely to be wrong. In this case, the judgment result with the higher probability will be used directly as the basis for determining whether it is a malicious indicator.

[0078] Extensive testing has shown that the second judgment model is more accurate when the difference between the first probability and the second probability is small (i.e., greater than the preset threshold). Therefore, in this embodiment, if the difference between the first probability and the second probability is small, the second judgment result is directly used as the basis for determining whether it is a malicious indicator.

[0079] In one exemplary embodiment of this application, obtaining candidate indicators from the text information includes:

[0080] The text information is segmented into sentences to obtain several character segments (i.e., several sentences). Sentence segmentation can be performed using a pre-trained NLP model.

[0081] Determine if a specified string exists in each character segment. If it does, perform character conversion processing on the specified string in the current character segment.

[0082] Feature extraction is performed on each character segment to obtain the feature information corresponding to each character segment. The feature information includes at least one of the following: text features, data source features, source distribution features, sentence features, content features, and external features.

[0083] Based on the feature information corresponding to each character segment, determine whether each character segment contains candidate indicators.

[0084] Specifically, candidate metrics can be URLs, domains, IP addresses, email addresses, file hashes, etc. Simultaneously, the surrounding text of the "candidate metric," i.e., the metric's "context," is extracted as the candidate metric's association information. Additionally, other information related to the "candidate metric," such as DNS records associated with it, can be collected as association information to help determine whether the candidate metric itself is malicious.

[0085] Because browsers and other web and email clients often create URLs, IP addresses, email addresses, or clickable domain names, such text information is often intentionally made unclickable to prevent readers from inadvertently accessing dangerous resources; this process is known as "de-intrusion."

[0086] For example, a URL like "hXXPs[:] / / malicious.url[.]com / install.exe" is a harmless URL. However, because many textual features have been modified, this harmless information is not easy to identify candidate indicators. Therefore, in this embodiment, if a specified string (such as [:]) exists within the character field, the specified string will be converted to restore the text information and improve the recognition rate of candidate indicators.

[0087] In another aspect of this application, a web page data processing apparatus is provided, comprising:

[0088] The acquisition module is used to acquire hyperlink information and text information within the current webpage; the URL of the current page is stored in the target URL list, and the page corresponding to each URL in the target URL list is used to acquire malicious indicators;

[0089] The extraction module is used to extract malicious indicators from the text information;

[0090] An add module is used to add the URL corresponding to the hyperlink information to the target URL list if malicious indicators are extracted from the text information.

[0091] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0092] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0093] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0094] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0095] An electronic device according to this embodiment of the present application. The electronic device is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0096] Electronic devices are manifested in the form of general-purpose computing devices. Components of an electronic device may include, but are not limited to: at least one processor, at least one memory, and buses connecting different system components (including memory and processor).

[0097] The storage device stores program code that can be executed by the processor to perform the steps described in the "Exemplary Methods" section above, according to various exemplary embodiments of this application.

[0098] The storage may include readable media in the form of volatile storage, such as random access memory (RAM) and / or cache memory, and may further include read-only memory (ROM).

[0099] The storage may also include programs / utilities having a set (at least one) of program modules, including but not limited to: an operating system, one or more applications, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0100] A bus can represent one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus that uses any of the various bus architectures.

[0101] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0102] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0103] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible implementations, various aspects of this application may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of this application described in the "Exemplary Methods" section above.

[0104] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0105] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0106] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0107] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0108] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this application, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0109] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0110] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A webpage data processing method, characterized in that, include: Obtain hyperlink information and text information within the current webpage; the URL of the current page is stored in a target URL list, and the page corresponding to each URL in the target URL list is used to obtain malicious indicators; malicious indicators refer to the indicators used by attackers to infect or compromise target devices, i.e., compromise indicators; Malicious indicators are extracted from the text information; If malicious indicators are extracted from the text information, the URLs corresponding to the hyperlinks will be added to the target URL list; The extraction of malicious indicators from the text information includes: Candidate indicators are obtained from the text information; The candidate indicators are input into the first judgment model and the second judgment model respectively to obtain the first judgment result and the second judgment result; both the first judgment model and the second judgment model are used to determine whether the candidate indicator is a malicious indicator, and the model types of the first judgment model and the second judgment model are different; Whether to mark the candidate indicator as a malicious indicator is determined based on the first judgment result and / or the second judgment result; The first judgment model is a heuristic classifier, and the second judgment model is a machine learning classifier; The step of determining whether to mark the candidate indicator as a malicious indicator based on the first judgment result and / or the second judgment result includes: Determine whether the first and second judgment results are the same; If they are the same, then determine whether to mark the candidate indicator as a malicious indicator based on the first judgment result and / or the second judgment result; If they are not the same, then obtain the first probability and the second probability; where the first probability is the accuracy of the first judgment model output for the first judgment result, and the second probability is the accuracy of the second judgment model output for the second judgment result. When the difference between the first probability and the second probability is greater than a preset threshold, if the first probability is greater than the second probability, then it is determined whether to mark the candidate indicator as a malicious indicator based on the first judgment result; otherwise, it is determined whether to mark the candidate indicator as a malicious indicator based on the second judgment result. When the difference between the first probability and the second probability is less than or equal to a preset threshold, a second judgment result is used to determine whether the candidate indicator is marked as a malicious indicator.

2. The webpage data processing method according to claim 1, characterized in that, The target URL list includes a visited list and an unvisited list; Adding the URL corresponding to the hyperlink information to the target URL list includes: Determine whether the URL corresponding to the hyperlink information exists in the visited list; If it does not exist, the URL corresponding to the hyperlink information is added to the unvisited list.

3. The webpage data processing method according to claim 1, characterized in that, The method further includes: If no malicious indicators are obtained from the webpages containing the stored URLs within the set time period, the stored URLs will be removed from the target URL list. The stored URL can be any URL in the target URL list.

4. The webpage data processing method according to claim 1, characterized in that, The step of obtaining candidate indicators from the text information includes: The text information is segmented into sentences to obtain several character segments; Feature extraction is performed on each character segment to obtain feature information corresponding to each character segment; the feature information includes at least one of the following: text features, data source features, source distribution features, sentence features, content features, and external features; Based on the feature information corresponding to each character segment, determine whether each character segment contains candidate indicators.

5. The webpage data processing method according to claim 4, characterized in that, Before performing feature extraction on each character segment, and after performing sentence segmentation on the text information, the method further includes: Determine if a specified string exists in each character segment. If it does, perform character conversion processing on the specified string in the current character segment.

6. A web page data processing device, characterized in that, include: The acquisition module is used to retrieve hyperlink information and text information within the current webpage; The URL of the current page is stored in the target URL list. Each page corresponding to each URL in the target URL list is used to obtain malicious indicators. Malicious indicators refer to the indicators used by attackers to infect or compromise target devices, i.e., compromise indicators. The extraction module is used to extract malicious indicators from the text information; An add module is used to add the URL corresponding to the hyperlink information to the target URL list if malicious indicators are extracted from the text information; The extraction of malicious indicators from the text information includes: Candidate indicators are obtained from the text information; The candidate indicators are input into the first judgment model and the second judgment model respectively to obtain the first judgment result and the second judgment result; both the first judgment model and the second judgment model are used to determine whether the candidate indicator is a malicious indicator, and the model types of the first judgment model and the second judgment model are different; Whether to mark the candidate indicator as a malicious indicator is determined based on the first judgment result and / or the second judgment result; The first judgment model is a heuristic classifier, and the second judgment model is a machine learning classifier; The step of determining whether to mark the candidate indicator as a malicious indicator based on the first judgment result and / or the second judgment result includes: Determine whether the first and second judgment results are the same; If they are the same, then determine whether to mark the candidate indicator as a malicious indicator based on the first judgment result and / or the second judgment result; If they are not the same, then obtain the first probability and the second probability; where the first probability is the accuracy of the first judgment model output for the first judgment result, and the second probability is the accuracy of the second judgment model output for the second judgment result. When the difference between the first probability and the second probability is greater than a preset threshold, if the first probability is greater than the second probability, then it is determined whether to mark the candidate indicator as a malicious indicator based on the first judgment result; otherwise, it is determined whether to mark the candidate indicator as a malicious indicator based on the second judgment result. When the difference between the first probability and the second probability is less than or equal to a preset threshold, a second judgment result is used to determine whether the candidate indicator is marked as a malicious indicator.

7. An electronic device, characterized in that, Including processor and memory; The processor executes the steps of the method as described in any one of claims 1 to 5 by invoking programs or instructions stored in the memory.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores a program or instructions that cause a computer to perform the steps of the method as described in any one of claims 1 to 5.

Images