URL (Uniform Resource Locator) detection model training and URL detection method, device, equipment, medium and program

Abstract

The embodiment of the invention discloses a URL (Uniform Resource Locator) detection model training and URL detection method, device, equipment, medium and program, and the method comprises the steps: carrying out the feature extraction of URL sample data, and obtaining a URL feature set; wherein the URL feature set comprises at least one of a structural feature, an intelligence feature and a sensitive word feature; extracting a part of feature sets in the URL feature set as a target feature set according to the feature contribution degree; and training the random forest model according to the sample features in the target feature set. According to the embodiment of the invention, the unknown URL can be accurately identified, and the URL detection efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of network security technology, and in particular to a URL detection model training and URL detection method, device, equipment, medium and program. Background Art

[0002] With the continuous growth of Internet users and the great enrichment of network applications, the number of Uniform Resource Locators (URLs) is also growing. People frequently access URLs to obtain information in their daily lives, and they will inevitably be induced to access various malicious URLs. Once these malicious URLs are accidentally accessed, users may face security risks such as account theft or personal privacy information theft. Therefore, it is very important to effectively detect URLs.

[0003] At present, the detection methods for URLs are mainly divided into two categories according to the different detection objects: (1) taking the web page content as the detection object, detecting the URL by analyzing the actual content of the web page; (2) taking the URL string as the detection object, using the blacklist mechanism to detect the URL.

[0004] In the process of implementing the present invention, the inventors found that the prior art has the following defects: (1) The URL detection method based on web page content often needs to download the relevant content of the web page first, which will consume a lot of computing resources, and the detection efficiency of this method is very low; (2) The blacklist mechanism of the detection method based on URL characters can only detect malicious URLs that have been marked in the blacklist library, and it is difficult to quickly and accurately identify unknown URLs. Summary of the invention

[0005] The embodiment of the present invention provides a URL detection model training and URL detection method, device, equipment, medium and program, which can accurately identify unknown URLs and improve the efficiency of URL detection.

[0006] According to one aspect of the present invention, a URL detection model training method is provided, comprising:

[0007] Extracting features from the URL sample data to obtain a URL feature set; wherein the URL feature set includes at least one of a structural feature, an intelligence feature, and a sensitive word feature;

[0008] Extracting a part of the feature set in the URL feature set as a target feature set according to the feature contribution;

[0009] The random forest model is trained according to the sample features in the target feature set.

[0010] According to another aspect of the present invention, there is provided a URL detection method, comprising:

[0011] Get the URL data to be detected;

[0012] Using a random forest model to detect the URL data to be detected, and obtaining a detection result of the URL data to be detected;

[0013] Wherein, the random forest model is trained by the URL detection model training method described in any embodiment of the present invention.

[0014] According to another aspect of the present invention, there is provided a URL detection model training device, comprising:

[0015] A URL feature set acquisition module, used to extract features from URL sample data to obtain a URL feature set; wherein the URL feature set includes at least one of a structural feature, an intelligence feature, and a sensitive word feature;

[0016] A target feature set acquisition module, used for extracting a part of the feature set in the URL feature set as a target feature set according to the feature contribution;

[0017] The model training module is used to train the random forest model according to the sample features in the target feature set.

[0018] According to another aspect of the present invention, there is provided a URL detection device, comprising:

[0019] The module for acquiring the URL data to be detected is used to acquire the URL data to be detected;

[0020] A detection result acquisition module is used to detect the URL data to be detected by using a random forest model to obtain a detection result of the URL data to be detected;

[0021] Wherein, the random forest model is trained by the URL detection model training method described in any embodiment of the present invention.

[0022] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0023] at least one processor; and

[0024] a memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the URL detection model training method or URL detection method described in any embodiment of the present invention.

[0026] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the URL detection model training method or URL detection method described in any embodiment of the present invention when executed.

[0027] According to another aspect of the present invention, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the URL detection model training method or URL detection method described in any embodiment of the present invention.

[0028] The embodiment of the present invention obtains a URL feature set including at least one of a structural feature, an intelligence feature, and a sensitive word feature by extracting features from URL sample data. Furthermore, a partial feature set in the URL feature set can be extracted as a target feature set according to the feature contribution, so that a random forest model can be trained according to the sample features in the target feature set. After the random forest model training is completed, the random forest model can be used to detect the acquired URL data to be detected to obtain the detection result of the URL data to be detected. The above technical solution solves the problem that the existing URL detection method has low detection efficiency and cannot identify unknown URLs, can accurately identify unknown URLs, and improves the efficiency of URL detection.

[0029] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 is a flow chart of a URL detection model training method provided in Example 1 of the present invention;

[0032] Figure 2 is a flow chart of a URL detection method provided by Embodiment 2 of the present invention;

[0033] Figure 3 It is a schematic diagram of a threat intelligence sharing capability architecture provided by Embodiment 2 of the present invention;

[0034] Figure 4 is a schematic diagram of a URL detection model training device provided in Embodiment 3 of the present invention;

[0035] Figure 5 is a schematic diagram of a URL detection device provided by Embodiment 4 of the present invention;

[0036] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0038] It should be noted that the terms "target", "spare", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] Embodiment 1

[0040] Figure 1 This is a flowchart of a URL detection model training method provided by the first embodiment of the present invention. This embodiment can be applied to the case where a feature set is selected according to feature contribution to train a random forest model so that the random forest model can detect URLs. The method can be executed by a URL detection model training device, which can be implemented by software and / or hardware and can generally be integrated in an electronic device. The electronic device can be a terminal device or a server device. As long as the URL detection model training method can be executed, the embodiment of the present invention does not limit the specific device type of the electronic device. Accordingly, Figure 1 As shown, the method includes the following operations:

[0041] S110. Extract features from the URL sample data to obtain a URL feature set; wherein the URL feature set includes at least one of a structural feature, an intelligence feature, and a sensitive word feature.

[0042] Among them, the URL sample data can be a set of URL addresses and their related tags and features for training, testing and verifying the random forest model. Exemplarily, the URLs in the URL sample data can be malicious URLs, and can also include benign URLs. The embodiment of the present invention does not limit the specific URL types included in the URL sample data. The URL feature set can be a set of features extracted from the URL sample data, and these features can be used to train and evaluate the performance of the random forest model. Exemplarily, the URL feature set can include but is not limited to structural features, intelligence features, and sensitive word features. The embodiment of the present invention does not limit the specific feature types included in the URL feature set. Structural features can be a set of features extracted from the structure and composition of each URL in the URL sample data to describe the various parts of the URL and its attributes. Intelligence features can be various features with intelligence value extracted from the URL sample data. Sensitive word features can be words that appear in each URL in the URL sample data and may involve security risks, privacy issues, illegal content, or do not comply with specific rules and policies.

[0043] In the embodiment of the present invention, in order to train the URL detection model, feature extraction may be performed on the URL sample data first, thereby obtaining a URL feature set.

[0044] In an optional embodiment of the present invention, the feature extraction of URL sample data may include: extracting the structural features from the domain name body, top-level domain name and sub-domain name of the URL sample data; extracting the intelligence features from the Alexa ranking and domain name registration information query protocol Whois of the URL sample data; extracting the sensitive word features from the URL and URL path in the URL sample data.

[0045] In an embodiment of the present invention, structural features of the URL sample data can be extracted for different components such as the domain name body, top-level domain name and sub-domain name of each URL in the URL sample data, such as whether an IP (Internet Protocol) address exists in the URL, the length of the domain name body in the URL, whether the top-level domain name in the URL is ".com", ".org" or ".gov", etc., whether the domain name body in the URL includes both numbers and letters, the number of numbers in the domain name body in the URL, the percentage of the number of numbers in the domain name body in the URL to the total length, whether the domain name body in the URL is all numbers, the number of vowels in the domain name body in the URL, the number of uppercase letters in the domain name body in the URL, the number of consecutive characters in the domain name body in the URL, the number of consecutive consonants in the domain name body in the URL, the length of the sub-domain name of the URL, the percentage of the number of numbers in the sub-domain name in the URL to the total length, the number of dots in the URL, the maximum length between domain name dots in the URL, the number of parameters in the URL, whether the domain name in the URL includes special characters, and the information entropy of the domain name in the URL, etc.

[0046] Optionally, intelligence features of the URL sample data can also be extracted from the Alexa ranking and Whois of the URL sample data, such as whether the domain name in the URL is expired, the Alexa ranking of the domain name in the URL, whether the domain name in the URL has been registered with ICP (Internet Content Provider), whether the domain name server in the URL is overseas, and whether the registration time of the domain name in the URL is less than 2 years.

[0047] Optionally, sensitive word features of the URL sample data may also be extracted from each URL and URL path in the URL sample data, such as whether the path part in the URL includes sensitive words such as APP1, APP2, APP3, and safety, and whether the URL includes illegal and illegal words, etc. Among them, APP1, APP2, and APP3 represent the names of sensitive APPs (Applications).

[0048] S120, extracting part of the feature set in the URL feature set as the target feature set according to the feature contribution.

[0049] The feature contribution may be the contribution of each sample feature in the URL feature set to the prediction performance of the random forest model.

[0050] Specifically, after extracting features from URL sample data to obtain a URL feature set, the feature contribution of each sample feature in the URL feature set can be calculated, so that the URL feature set can be screened according to the feature contribution to obtain a target feature set that includes part of the feature set in the URL feature set.

[0051] In an optional embodiment of the present invention, extracting a partial feature set from the URL feature set as a target feature set based on feature contribution may include: training the random forest model based on the URL feature set and calculating a standard evaluation value of the random forest model; calculating the feature contribution of each sample feature in the URL feature set; sorting the sample features according to their feature contribution, eliminating the sample features with the smallest feature contribution, and obtaining an alternative feature set; training the random forest model based on the alternative feature set and calculating a target evaluation value of the random forest model; and when it is determined that the standard evaluation value is consistent with the target evaluation value, using the alternative feature set as the target feature set.

[0052] Among them, the standard evaluation value of the random forest model can be the mean of the evaluation indicators such as the accuracy, precision, recall, F1 value and AUC (Area Under the Curve) value of the random forest model when the random forest model is trained using the URL feature set. The alternative feature set can be the feature set obtained by eliminating the sample features with the smallest feature contribution in the URL sample feature set. The target evaluation value can be the mean of the evaluation indicators such as the accuracy, precision, recall, F1 value and AUC of the random forest model when the random forest model is trained using the alternative feature set. The target feature set can be an alternative feature set whose target evaluation value is consistent with the standard evaluation value.

[0053] In an embodiment of the present invention, in the process of extracting part of the feature set in the URL feature set as the target feature set according to the feature contribution, the random forest model can be firstly trained according to the URL feature set, and the mean of the evaluation indicators such as the accuracy, precision, recall rate, F1 value and AUC value of the random forest model is calculated as the standard evaluation value for selecting the target feature set.

[0054] Further, the feature contribution of each sample feature in the URL feature set can be calculated. Exemplarily, the feature contribution of each sample feature in the URL sample feature set can be calculated based on the Gini importance of the random forest model. After obtaining the feature contribution of each sample feature, the sample features can be sorted according to their feature contribution, and the sample features with the smallest feature contribution can be eliminated, so that the alternative feature set can be obtained. It can be understood that the number of sample features in the alternative feature set is the number of sample features in the URL feature set minus 1. Further, the random forest model can be trained according to the alternative feature set, and the mean of the evaluation indicators such as the accuracy, precision, recall, F1 value and AUC value of the random forest model can be calculated as the target evaluation value. Repeat the above steps until the number of sample features in the alternative feature set reaches a preset threshold. Exemplarily, the preset threshold can be 8. It can be understood that the preset threshold can be determined according to actual needs, and the embodiment of the present invention does not limit the specific value of the preset threshold.

[0055] Furthermore, the standard evaluation value and the target evaluation value can be compared to determine the target feature set. When it is determined that the standard evaluation value and the target evaluation value are basically consistent, the alternative feature set is used as the target feature set. It should be noted that if there are multiple alternative feature sets whose target evaluation values ​​are basically consistent with the standard evaluation values, the alternative feature set with the least sample features can be selected as the target feature set. The above method can improve the robustness and adaptability of the random forest model without significantly reducing the classification accuracy, while reducing the computational overhead of the random forest model.

[0056] S130: training the random forest model according to the sample features in the target feature set.

[0057] Specifically, after extracting part of the feature set in the URL feature set as the target feature set according to the feature contribution, the random forest model can be trained according to the sample features in the target feature set, so that the random forest model can accurately detect the URL.

[0058] In an optional embodiment of the present invention, after training the random forest model according to the sample features in the target feature set, it may also include: obtaining a hyperparameter set of the random forest model; determining the value range and objective function of each parameter in the hyperparameter set; and adjusting the hyperparameter set of the random forest model according to the objective function.

[0059] Among them, the hyperparameter set can be a parameter set that can be adjusted in the random forest model. Exemplarily, the hyperparameter set can include but is not limited to the number of decision trees, the maximum depth of the decision tree, the minimum number of samples required for leaf nodes, and the minimum number of samples required for splitting internal nodes. The embodiment of the present invention does not limit the specific parameter types included in the hyperparameter set. The value range of each parameter can be the range of each parameter in the hyperparameter set. The objective function can be a function used to measure the performance of the random forest model.

[0060] In an embodiment of the present invention, after training the random forest model according to the sample features in the target feature set, a series of parameters that need to be adjusted in the random forest model can also be determined as a hyperparameter set of the random forest model. Further, the value range of each parameter in the hyperparameter set and the objective function used to measure the performance of the random forest model can be determined, so that the hyperparameter combination that optimizes the objective function can be found through search algorithms such as grid search, random search, and Bayesian search. The objective function can improve the accuracy and generalization ability of the random forest model by combining the prediction results of multiple decision trees. It should be noted that the objective function can be set according to actual needs.

[0061] In a specific example, the number of decision trees may range from 10 to 200, the maximum depth of the decision tree may range from 5 to 100, the minimum number of samples required for a leaf node may range from 1 to 20, and the minimum number of samples required to split an internal node may range from 2 to 10.

[0062] It can be seen that the above scheme screens the URL feature set through feature contribution to obtain the target feature set, and trains the random forest model according to the target feature set, which improves the accuracy of URL detection by the random forest model and the AUC value of the random forest model. The above scheme can be applied to threat intelligence information analysis, especially the detection of unknown malicious URLs, which can greatly improve the efficiency of detection, reduce the cost of manual analysis, increase the speed and frequency of real-time updates of the intelligence library, and strengthen the control of malicious URLs in China.

[0063] The embodiment of the present invention obtains a URL feature set including at least one of a structural feature, an intelligence feature, and a sensitive word feature by extracting features from URL sample data. Furthermore, a partial feature set in the URL feature set can be extracted as a target feature set according to the feature contribution, so that a random forest model can be trained according to the sample features in the target feature set, solving the problem that the existing URL detection method has low detection efficiency and cannot identify unknown URLs, and can accurately identify unknown URLs and improve the efficiency of URL detection.

[0064] Embodiment 2

[0065] Figure 2 This is a flowchart of a URL detection method provided by the second embodiment of the present invention. This embodiment is applicable to the case where a URL detection model is used to detect a URL. The method can be executed by a URL detection device, which can be implemented by software and / or hardware and can generally be integrated in an electronic device. The electronic device can be a terminal device or a server device. As long as it can run the URL detection model to execute the URL detection method, the embodiment of the present invention does not limit the specific device type of the electronic device. Accordingly, Figure 2 As shown, the method includes the following operations:

[0066] S210: Obtain URL data to be detected.

[0067] The URL data to be detected may be URL data to be subjected to security detection.

[0068] In the embodiment of the present invention, the URL data to be detected may be obtained first, so that the URL data to be detected may be detected.

[0069] S220: Use a random forest model to detect the URL data to be detected, and obtain a detection result of the URL data to be detected.

[0070] Specifically, after obtaining the URL data to be detected, the random forest model can be used to detect the URL data to be detected, so as to determine the classification of the URL and obtain the detection result of the URL data to be detected, for example, the URL data to be detected is a malicious URL or the URL to be detected is a benign URL. Optionally, the random forest model can be trained by the URL detection model training method provided in the first embodiment.

[0071] Figure 3 is a schematic diagram of a threat intelligence sharing capability architecture provided by Embodiment 2 of the present invention. In a specific example, the above-mentioned URL detection technology can be applied to the threat intelligence sharing capability architecture, such as Figure 3As shown, the threat intelligence sharing capability relies on the self-developed threat intelligence knowledge base to provide users with highly accurate security intelligence information such as URL, TP (True Positive), domain names and vulnerabilities, and provide technical support for related security incident response and security warning. The threat intelligence sharing capability may include but is not limited to the ability to provide standard API (Application Programming Interface) threat intelligence query interface, such as URL intelligence query, IP intelligence query, domain name intelligence query, vulnerability intelligence query and IP geographic location query; covering nearly 19 types of intelligence identification such as remote control, malware, malicious network, scanning and cracking, and can provide rich intelligence information; platforms, systems and equipment for provincial companies, professional companies, government and enterprises can obtain threat intelligence information such as malicious IP, domain names and URLs by integrating threat intelligence sharing capabilities, providing technical support for security incident response and security warning.

[0072] The threat intelligence sharing capability can empower existing security devices through API interfaces to enhance the detection capabilities of existing network security devices. It can also provide multi-dimensional threat intelligence data to support security personnel in quickly locating security threats and improving the efficiency of handling security incidents. It can also enhance the ability to identify internal and external security risks through integrated threat intelligence sharing capabilities. It can also assist users in realizing network security threat discovery, monitoring and early warning, and attack tracing during the security period.

[0073] The embodiment of the present invention obtains the URL data to be detected, and further, can use the random forest model to detect the URL data to be detected, so as to obtain the detection result of the URL data to be detected, which solves the problem that the existing URL detection method has low detection efficiency and cannot identify unknown URLs, can accurately identify unknown URLs, and improve the efficiency of URL detection.

[0074] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0075] It should be noted that any arrangement and combination of the technical features in the above embodiments also falls within the protection scope of the present invention.

[0076] Embodiment 3

[0077] Figure 4 is a schematic diagram of a URL detection model training device provided in Embodiment 3 of the present invention, such as Figure 4 As shown, the device includes: a URL feature set acquisition module 310, a target feature set acquisition module 320 and a model training module 330, wherein:

[0078] The URL feature set acquisition module 310 is used to extract features from URL sample data to obtain a URL feature set; wherein the URL feature set includes at least one of a structural feature, an intelligence feature, and a sensitive word feature.

[0079] The target feature set acquisition module 320 is used to extract part of the feature sets in the URL feature set as the target feature set according to the feature contribution.

[0080] The model training module 330 is used to train the random forest model according to the sample features in the target feature set.

[0081] The embodiment of the present invention obtains a URL feature set including at least one of a structural feature, an intelligence feature, and a sensitive word feature by extracting features from URL sample data. Furthermore, a partial feature set in the URL feature set can be extracted as a target feature set according to the feature contribution, so that a random forest model can be trained according to the sample features in the target feature set, solving the problem that the existing URL detection method has low detection efficiency and cannot identify unknown URLs, and can accurately identify unknown URLs and improve the efficiency of URL detection.

[0082] Optionally, the URL feature set acquisition module 310 is specifically used to: extract the structural features from the domain name body, top-level domain name and sub-domain name of the URL sample data; extract the intelligence features from the Alexa ranking and Whois of the URL sample data; extract the sensitive word features from the URL and URL path in the URL sample data.

[0083] Optionally, the target feature set acquisition module 320 is specifically used to: train the random forest model according to the URL feature set and calculate the standard evaluation value of the random forest model; calculate the feature contribution of each sample feature in the URL feature set; sort the sample features according to their feature contribution, eliminate the sample features with the smallest feature contribution, and obtain an alternative feature set; train the random forest model according to the alternative feature set and calculate the target evaluation value of the random forest model; when it is determined that the standard evaluation value and the target evaluation value are consistent, use the alternative feature set as the target feature set.

[0084] Optionally, the above-mentioned device may also include a model optimization module, which is specifically used to: obtain a hyperparameter set of the random forest model; determine the value range and objective function of each parameter in the hyperparameter set; and adjust the hyperparameter set of the random forest model according to the objective function.

[0085] The above-mentioned URL detection model training device can execute the URL detection model training method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the URL detection model training method provided by any embodiment of the present invention.

[0086] Since the URL detection model training device introduced above is a device that can execute the URL detection model training method in the embodiment of the present invention, based on the URL detection model training method introduced in the embodiment of the present invention, the technical personnel in the field can understand the specific implementation of the XX device of the present embodiment and its various variations, so how the URL detection model training device implements the URL detection model training method in the embodiment of the present invention is not introduced in detail here. As long as the technical personnel in the field implement the device used in the URL detection model training method in the embodiment of the present invention, it belongs to the scope of protection of this application.

[0087] Embodiment 4

[0088] Figure 5 is a schematic diagram of a URL detection device provided by Embodiment 4 of the present invention, such as Figure 5 As shown, the device includes: a URL data acquisition module 410 to be detected and a detection result acquisition module 420, wherein:

[0089] The to-be-detected URL data acquisition module 410 is used to acquire the to-be-detected URL data.

[0090] A detection result acquisition module 420 is used to detect the URL data to be detected by using a random forest model to obtain a detection result of the URL data to be detected;

[0091] Wherein, the random forest model is trained by the URL detection model training method described in any embodiment of the present invention.

[0092] The embodiment of the present invention obtains the URL data to be detected, and further, can use the random forest model to detect the URL data to be detected, so as to obtain the detection result of the URL data to be detected, which solves the problem that the existing URL detection method has low detection efficiency and cannot identify unknown URLs, can accurately identify unknown URLs, and improve the efficiency of URL detection.

[0093] The above-mentioned URL detection device can execute the URL detection method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the URL detection method provided by any embodiment of the present invention.

[0094] Since the URL detection device introduced above is a device that can execute the URL detection method in the embodiment of the present invention, based on the URL detection method introduced in the embodiment of the present invention, the technicians in this field can understand the specific implementation of the URL detection device of this embodiment and its various variations, so how the URL detection device implements the URL detection method in the embodiment of the present invention is not introduced in detail here. As long as the technicians in this field implement the device adopted by the URL detection method in the embodiment of the present invention, it belongs to the scope of protection of this application.

[0095] Embodiment 5

[0096] Figure 6 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0097] like Figure 6 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0098] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0099] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 performs the various methods and processes described above, such as a URL detection model training method or a URL detection method.

[0100] In some embodiments, the URL detection model training method or the URL detection method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the URL detection model training method or the URL detection method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the URL detection model training method or the URL detection method by any other appropriate means (e.g., by means of firmware).

[0101] Optionally, the URL detection model training method may include: performing feature extraction on URL sample data to obtain a URL feature set; wherein the URL feature set includes at least one of a structural feature, an intelligence feature, and a sensitive word feature; extracting a partial feature set from the URL feature set as a target feature set based on feature contribution; and training the random forest model based on sample features in the target feature set.

[0102] Optionally, the URL detection method may include: obtaining URL data to be detected; using a random forest model to detect the URL data to be detected to obtain a detection result of the URL data to be detected; wherein the random forest model is trained by the URL detection model training method described in any embodiment of the present invention.

[0103] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0105] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0106] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0107] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0108] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0109] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0110] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A URL detection model training method, characterized in that: include: Extracting features from the uniform resource locator URL sample data to obtain a URL feature set; wherein the URL feature set includes at least one of a structural feature, an intelligence feature, and a sensitive word feature; Extracting a part of the feature set in the URL feature set as a target feature set according to the feature contribution; The random forest model is trained according to the sample features in the target feature set.

2. The method according to claim 1, characterized in that: The feature extraction of the URL sample data includes: Extracting the structural features from the domain name body, top-level domain name and sub-domain name of the URL sample data; Extracting the intelligence features from the Alexa ranking of the URL sample data and the domain name registration information query protocol Whois; The sensitive word feature is extracted from the URL and URL path in the URL sample data.

3. The method according to claim 1, characterized in that The extracting a part of the feature set in the URL feature set as the target feature set according to the feature contribution degree includes: Training the random forest model according to the URL feature set, and calculating a standard evaluation value of the random forest model; Calculate the feature contribution of each sample feature in the URL feature set; Sorting the sample features according to their contribution, removing the sample features with the smallest contribution, and obtaining a candidate feature set; Training the random forest model according to the candidate feature set, and calculating a target evaluation value of the random forest model; When it is determined that the standard evaluation value is consistent with the target evaluation value, the candidate feature set is used as the target feature set.

4. The method according to claim 1, characterized in that After training the random forest model according to the sample features in the target feature set, the method further includes: Obtaining a set of hyperparameters of the random forest model; Determine the value range and objective function of each parameter in the hyperparameter set; The hyperparameter set of the random forest model is adjusted according to the objective function.

5. A URL detection method, characterized in that: include: Get the URL data to be detected; Using a random forest model to detect the URL data to be detected, and obtaining a detection result of the URL data to be detected; Wherein, the random forest model is trained by the URL detection model training method described in any one of claims 1-4.

6. A URL detection model training device, characterized in that: include: A URL feature set acquisition module, used to extract features from URL sample data to obtain a URL feature set; wherein the URL feature set includes at least one of a structural feature, an intelligence feature, and a sensitive word feature; A target feature set acquisition module, used for extracting a part of the feature set in the URL feature set as a target feature set according to the feature contribution; The model training module is used to train the random forest model according to the sample features in the target feature set.

7. A URL detection device, characterized in that: include: The module for acquiring the URL data to be detected is used to acquire the URL data to be detected; A detection result acquisition module is used to detect the URL string data to be detected by using a random forest model to obtain a detection result of the URL data to be detected; Wherein, the random forest model is trained by the URL detection model training method described in any one of claims 1-4.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the URL detection model training method described in any one of claims 1-4 or the URL detection method described in claim 5.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the URL detection model training method described in any one of claims 1 to 4 or the URL detection method described in claim 5 when executed.

10. A computer program product comprising a computer program / instructions, wherein: When the computer program / instructions are executed by a processor, the URL detection model training method described in any one of claims 1 to 4 or the URL detection method described in claim 5 is implemented.

Images