Method and device for detecting malicious website, and computer storage medium

Abstract

The present disclosure relates to a method and device for detecting malicious websites, and a computer storage medium, and relates to the technical field of network security. The method for detecting malicious websites comprises: determining a plurality of strings according to the structure of a website to be detected; determining the character encoding corresponding to each character in each string; determining the encoding vector corresponding to each string according to the character encoding corresponding to each character in the string; and determining whether the website to be detected is a malicious website by using a first machine learning model according to the plurality of encoding vectors corresponding to the plurality of strings. According to the present disclosure, the accuracy of detecting malicious websites can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of network security technology, and in particular to methods and apparatus for detecting malicious websites, and computer-storable media. Background Technology

[0002] With the rapid development of the internet and the continuous increase in the number of netizens, the high-speed and frequent exchange of information has led to a surge in cyberattacks such as Trojan injection, phishing, and distributed attacks, seriously threatening the privacy of individual users, the ecosystem of the network environment, and the security of national information and property. Many cyberattacks are carried out by spreading malicious URLs. Therefore, detecting malicious URLs can help reduce cyberattacks.

[0003] In related technologies, the URL to be detected is vectorized and then input into a trained convolutional neural network model to detect whether the URL is malicious. Summary of the Invention

[0004] In related technologies, the URL to be detected is vectorized and then input into a trained convolutional neural network model for detection. However, convolutional neural network models cannot effectively identify the features of the URL to be detected, thus failing to accurately detect malicious URLs.

[0005] To address the aforementioned technical issues, this disclosure proposes a solution that can improve the accuracy of detecting malicious URLs.

[0006] According to a first aspect of this disclosure, a method for detecting malicious URLs is provided, comprising: determining multiple strings based on the structure of the URL to be detected; determining the character encoding corresponding to each character in each string; determining an encoding vector corresponding to the string based on the character encoding corresponding to each character in each string; and detecting whether the URL to be detected is a malicious URL using a first machine learning model based on the multiple encoding vectors corresponding to the multiple strings.

[0007] In some embodiments, determining the character encoding corresponding to each character in each string includes: performing character decomposition on multiple reference URLs to obtain multiple reference characters; performing deduplication on the multiple reference characters to obtain multiple target characters; determining the character category to which each target character belongs; sorting the multiple target characters according to the priority order of the character categories to obtain a first character dictionary, wherein the priority order is positively correlated with the probability of the character category appearing in the URL; determining a character encoding dictionary corresponding to the multiple target characters based on the first character dictionary, wherein the character encoding dictionary includes multiple character codes, and the target characters correspond one-to-one with the character codes; and determining the character encoding corresponding to each character in each string from the character encoding dictionary.

[0008] In some embodiments, determining the character encoding dictionary corresponding to the plurality of target characters includes: counting the frequency of each target character in the plurality of reference characters; sorting the plurality of target characters in the first character dictionary belonging to each character type in descending order of frequency to obtain a second character dictionary; encoding each target character in the second character dictionary to obtain the character encoding dictionary, wherein the order of the multiple character encodings in the character encoding dictionary is consistent with the order of the multiple target characters in the second character dictionary.

[0009] In some embodiments, the first machine learning model includes multiple first machine learning models with different model complexities. Detecting whether the URL to be detected is a malicious URL using the first machine learning model based on multiple encoding vectors corresponding to the multiple strings includes: determining the URL complexity of the URL to be detected based on the multiple encoding vectors corresponding to the multiple strings; selecting a first machine learning model from the multiple first machine learning models whose model complexity is suitable for the URL complexity as a target machine learning model based on the URL complexity; and using the target machine learning model based on the multiple encoding vectors to detect whether the URL to be detected is a malicious URL.

[0010] In some embodiments, selecting a first machine learning model whose model complexity is compatible with the URL complexity as the target machine learning model includes: determining the model complexity corresponding to the URL complexity as the target model complexity based on the correspondence between URL complexity and model complexity, wherein the URL complexity and model complexity are in one-to-one correspondence; and selecting a first machine learning model corresponding to the target model complexity from the plurality of first machine learning models as the target machine learning model.

[0011] In some embodiments, the URL complexity of the URL to be detected is positively correlated with the number of character codes in the plurality of encoding vectors, and the model complexity of the first machine learning model is positively correlated with at least one of the number of convolutional layers and the number of pooling layers in the first machine learning model.

[0012] In some embodiments, detecting whether the URL to be detected is a malicious URL using the target machine learning model based on the plurality of encoded vectors includes: determining a two-dimensional matrix corresponding to the URL to be detected based on the plurality of encoded vectors corresponding to the plurality of strings; formatting the two-dimensional matrix corresponding to the URL to be detected so that the length of the plurality of encoded vectors in the two-dimensional matrix is ​​the same as a preset length; and using the target machine learning model based on the formatted two-dimensional matrix to detect whether the URL to be detected is a malicious URL.

[0013] In some embodiments, determining the URL complexity of the URL to be detected includes: determining a two-dimensional matrix corresponding to the URL to be detected based on multiple encoding vectors corresponding to the multiple strings; and determining the matrix complexity of the two-dimensional matrix as the URL complexity.

[0014] In some embodiments, the method for detecting malicious URLs further includes: cross-validating the first machine learning model using a second machine learning model other than the first machine learning model.

[0015] In some embodiments, cross-validating the first machine learning model using a second machine learning model other than the first machine learning model includes: using the first machine learning model to detect whether the plurality of training URLs are malicious URLs; determining a first detection accuracy of the first machine learning model based on the detection results for the plurality of training URLs and the labels of the plurality of training URLs, wherein the labels include a first label indicating that the training URL is a malicious URL and a second label indicating that the training URL is not a malicious URL; using the second machine learning model to detect whether the plurality of training URLs are malicious URLs based on the plurality of malicious URLs; determining a second detection accuracy of the other second machine learning model based on the detection results for the plurality of training URLs and the labels of the plurality of training URLs; and adjusting the parameters of the first machine learning model so that the first detection accuracy is greater than or equal to the second detection accuracy.

[0016] In some embodiments, the first machine learning model is a convolutional neural network model, and the second machine learning model is a random tree model, a random forest model, a Bayesian model, or a decision tree model.

[0017] In some embodiments, the structure of the URL to be detected includes a protocol portion, a hostname portion, a pathname portion, a filename portion, and a parameter portion. Determining multiple strings based on the structure of the URL to be detected includes: determining the protocol portion, hostname portion, pathname portion, filename portion, parameter portion, and the URL to be detected itself as strings respectively.

[0018] In some embodiments, determining the encoding vector corresponding to the string includes: constructing a one-dimensional vector by encoding the characters corresponding to each character in each string according to the character order in each string, and determining it as the encoding vector corresponding to each string.

[0019] According to a second aspect of this disclosure, an apparatus for detecting malicious URLs is provided, comprising: a first determining module configured to determine a plurality of strings based on the structure of the URL to be detected; a second determining module configured to determine the character encoding corresponding to each character in each string; a third determining module configured to determine an encoding vector corresponding to each string based on the character encoding corresponding to each character in each string; and a detection module configured to detect whether the URL to be detected is a malicious URL using a first machine learning model based on the plurality of encoding vectors corresponding to the plurality of strings.

[0020] According to a third aspect of this disclosure, an apparatus for detecting malicious URLs is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the method for detecting malicious URLs according to any of the above embodiments based on instructions stored in the memory.

[0021] According to a fourth aspect of this disclosure, a computer-storeable medium is provided having computer program instructions stored thereon, which, when executed by a processor, implement the method for detecting malicious URLs as described in any of the above embodiments.

[0022] The above embodiments can improve the accuracy of detecting malicious URLs. Attached Figure Description

[0023] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0024] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0025] Figure 1 This is a flowchart illustrating a method for detecting malicious URLs according to some embodiments of the present disclosure;

[0026] Figure 2 This is a flowchart illustrating the determination of the character encoding corresponding to each character in each string according to some embodiments of the present disclosure;

[0027] Figure 3 This is a flowchart illustrating the determination of a character encoding dictionary corresponding to a plurality of target characters according to some embodiments of the present disclosure;

[0028] Figure 4 This is a flowchart illustrating the detection of whether a URL to be detected is a malicious URL according to some embodiments of this disclosure;

[0029] Figure 5 This is a block diagram illustrating an apparatus for detecting malicious URLs according to some embodiments of the present disclosure;

[0030] Figure 6This is a block diagram illustrating an apparatus for detecting malicious URLs according to other embodiments of the present disclosure;

[0031] Figure 7 This is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure. Detailed Implementation

[0032] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0033] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0034] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0035] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0036] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0037] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0038] Figure 1 This is a flowchart illustrating a method for detecting malicious URLs according to some embodiments of the present disclosure.

[0039] like Figure 1 As shown, the method for detecting malicious URLs includes: step S10, determining multiple strings based on the structure of the URL to be detected; step S20, determining the character encoding corresponding to each character in each string; step S30, determining the encoding vector corresponding to the string based on the character encoding corresponding to each character in each string; and step S40, using a first machine learning model based on the multiple encoding vectors corresponding to the multiple strings to detect whether the URL to be detected is a malicious URL.

[0040] In step S10, multiple strings are determined based on the structure of the URL to be detected.

[0041] In some embodiments, the structure of the URL to be detected includes a protocol portion, a hostname portion, a pathname portion, a filename portion, and a parameter portion. The protocol portion, hostname portion, pathname portion, filename portion, parameter portion, and the URL itself are each defined as a string.

[0042] For example, the URL to be detected, http: / / addanetwork.net / frees, corresponds to multiple strings such as http, addanetwork.net, free, null, null, and http: / / addanetwork.net / frees. The non-existent parts of the URL to be detected are represented by an empty string, where null represents an empty string.

[0043] In step S20, the character encoding corresponding to each character in each string is determined.

[0044] For example, it can be done through, as Figure 2 The above step S20 is implemented in the manner shown.

[0045] Figure 2 This is a flowchart illustrating the determination of the character encoding corresponding to each character in each string according to some embodiments of the present disclosure.

[0046] like Figure 2 As shown, determining the character encoding corresponding to each character in each string includes steps S21-S26.

[0047] In step S21, the multiple reference URLs are decomposed into multiple reference characters. For example, a 1-gram segmentation algorithm is used to decompose the multiple reference URLs into characters. That is, the decomposition is performed on a single character basis.

[0048] In step S22, multiple reference characters are deduplicated to obtain multiple target characters.

[0049] In step S23, the character category to which each target character belongs is determined.

[0050] In step S24, multiple target characters are sorted according to their priority order to obtain a first character dictionary. The priority order is positively correlated with the probability of a character type appearing in a URL. For example, character types include lowercase letters, uppercase letters, numbers, and special characters. In some embodiments, the priority of lowercase letters, uppercase letters, numbers, and special characters can be set to decrease in the following order.

[0051] In step S25, a character encoding dictionary corresponding to the multiple target characters is determined based on the first character dictionary. The character encoding dictionary includes multiple character codes, and there is a one-to-one correspondence between the target characters and their corresponding character codes.

[0052] For example, it can be done through, as Figure 3 The above step S25 is implemented in the manner shown.

[0053] Figure 3 This is a flowchart illustrating the determination of a character encoding dictionary corresponding to a plurality of target characters according to some embodiments of the present disclosure.

[0054] like Figure 3 As shown, determining the character encoding dictionary corresponding to multiple target characters includes steps S251-S253.

[0055] In step S251, the frequency of each target character appearing among the plurality of reference characters is counted. The higher the frequency, the higher its importance and the more it needs to be perceived.

[0056] In step S252, the target characters belonging to each character category in the first character dictionary are sorted in descending order of frequency to obtain the second character dictionary.

[0057] In step S253, each target character in the second character dictionary is encoded to obtain a character encoding dictionary. The order of the character encodings in the character encoding dictionary is consistent with the order of the target characters in the second character dictionary. For example, the character encodings corresponding to the target characters increase sequentially according to the sorting order.

[0058] For example, the character encoding values ​​corresponding to each target character range from [1, 200]. The mapping function between the target character and its character encoding can be expressed as f: u → E. u u is a set of n target characters {u1, u2, ..., u...} n}, where n is a positive integer. E u The set of character codes corresponding to the target character {e1, e2, ..., e n The domain is {a, b, c, ..., z, A, B, C, ..., Z, 0, 1, 2, ..., 9, $, %, @, ...}, and the range is [1, 200].

[0059] return Figure 2 In step S26, the character encoding corresponding to each character in each string is determined from the character encoding dictionary. For example, the position of each character in the character encoding dictionary can be determined by matching, thereby determining the corresponding character encoding.

[0060] In the above embodiments, multiple target characters are obtained by decomposing multiple reference URLs, and these target characters are sorted according to the probability of each character type appearing in the URL, thus obtaining a character encoding dictionary. Based on this character encoding dictionary, the character encoding of each character is determined, so that the character encoding of each character is associated with its importance in the URL. This improves the sensitivity and accuracy of the machine learning model in feature recognition, thereby further improving the accuracy of detection.

[0061] Furthermore, in the above embodiments, target characters of the same character type are sorted according to their frequency, and the sorted target characters are then encoded to obtain a character encoding dictionary. Based on this character encoding dictionary, the character encoding of each character is determined, thus associating the character encoding of each character with its importance in the URL. This improves the sensitivity and accuracy of the machine learning model in feature recognition, thereby further enhancing the accuracy of detection.

[0062] return Figure 1 In step S30, an encoding vector corresponding to each string is determined based on the character encoding corresponding to each character in each string. For example, the strings corresponding to the protocol part, hostname part, pathname part, filename part, parameter part, and the URL itself are p, d, pa, fi, pm, and ur, respectively. Through the processing of the character encoding mapping function f, the corresponding encoding vectors f(p), f(d), f(pa), f(fi), f(pm), and f(ur) can be obtained.

[0063] In some embodiments, the character encoding corresponding to each character in each string is constructed into a one-dimensional vector according to the character order in each string, and the resulting vector is determined as the encoding vector for each string. For example, in the case of an empty string, the encoding vector corresponding to that string is also empty, represented as [].

[0064] In step S40, based on the multiple encoding vectors corresponding to multiple strings, the first machine learning model is used to detect whether the URL to be detected is a malicious URL.

[0065] In some embodiments, the first machine learning model includes multiple first machine learning models with different model complexities. For example, the first machine learning model is a convolutional neural network model, and the model complexity of the first machine learning model is positively correlated with at least one of the number of convolutional layers and the number of pooling layers in the first machine learning model.

[0066] For example, each first machine learning model includes one input layer, one fully connected layer, and one input layer. Multiple first machine learning models include convolutional neural network model A, convolutional neural network model B, and convolutional neural network model C.

[0067] Convolutional neural network model A includes one input layer, one fully connected layer, one input layer, one convolutional layer, and one pooling layer. The initial value of the convolutional kernel W in model A is a normally distributed 2×col weight matrix, where col is the preset number of columns and its value is a positive integer. The step size is 1.

[0068] Convolutional Neural Network (CNN) Model B adds one convolutional layer and one pooling layer to CNN Model A. The convolutional kernels in CNN Model B are 2×7401 and 2×10, respectively, while the pooling layer filters are 2×1 and 1×1, respectively, with a stride of 1. The first convolutional layer outputs 8 feature layers, and the second convolutional layer outputs 16 feature layers.

[0069] Convolutional Neural Network (CNN) model C adds one convolutional layer and one pooling layer to CNN model B. The kernel sizes of CNN model C are 2×7391, 2×10, and 2×112, and the pooling layer filters are 2×1, 1×1, and 1×1, with a stride of 1. The first convolutional layer has 8 output feature layers, the second has 16, and the third has 32.

[0070] In the above embodiments, the model complexity of convolutional neural network models A, B, and C increases sequentially.

[0071] For example, it can be done through, as Figure 4 The above step S40 is implemented in the manner shown.

[0072] Figure 4 This is a flowchart illustrating the detection of whether a URL to be detected is a malicious URL according to some embodiments of the present disclosure.

[0073] like Figure 4 As shown, detecting whether a URL to be detected is a malicious URL includes steps S41-S43.

[0074] In step S41, the URL complexity of the URL to be detected is determined based on multiple encoding vectors corresponding to multiple strings. In some embodiments, the URL complexity of the URL to be detected is positively correlated with the number of character codes in the multiple encoding vectors.

[0075] For example, based on multiple encoding vectors corresponding to multiple strings, a two-dimensional matrix corresponding to the URL to be detected is determined. The matrix complexity of the two-dimensional matrix is ​​then used as the URL complexity of the URL to be detected. The number of elements in the two-dimensional matrix can be used as the URL complexity. The number of elements in the two-dimensional matrix is ​​the same as the number of character encodings.

[0076] For example, rows 1-4 of a two-dimensional matrix represent the encoded vectors for the protocol portion, hostname portion, pathname portion, filename portion, and the URL to be detected itself, respectively. For instance, a two-dimensional matrix can be represented as E = {f(p), f(d), f(pa), f(fi), f(pm), f(ur)}.

[0077] In step S42, based on the URL complexity, a first machine learning model whose model complexity is suitable for the URL complexity is selected from multiple first machine learning models as the target machine learning model. By adaptively selecting the appropriate first machine learning model based on the URL complexity of the URL to be detected, the accuracy of detecting malicious URLs can be further improved.

[0078] In some embodiments, based on the correspondence between URL complexity and model complexity, the model complexity corresponding to the URL complexity is determined as the target model complexity. From multiple first machine learning models, the first machine learning model corresponding to the target model complexity is selected as the target machine learning model. There is a one-to-one correspondence between URL complexity and model complexity.

[0079] For example, taking the URL complexity of the URL to be detected as the number of elements in the two-dimensional matrix corresponding to the URL as an example, convolutional neural network model A can detect URLs with a URL complexity less than or equal to 20, convolutional neural network model B can detect URLs with a URL complexity greater than 20 and less than or equal to 100, and convolutional neural network model C can detect URLs with a URL complexity greater than 100.

[0080] In step S43, based on multiple encoded vectors, a target machine learning model is used to detect whether the URL to be detected is a malicious URL.

[0081] In some embodiments, step S43 can be implemented in the following manner.

[0082] First, based on multiple encoding vectors corresponding to multiple strings, a two-dimensional matrix corresponding to the URL to be detected is determined. By encoding the URL to be detected into a two-dimensional matrix, different characters have different character codes, and the sequence relationship between the character codes represents the sequence and spatial relationships between the characters in the URL to be detected. For example, the data matrix can also be converted into tensor form using placeholders in the Tensorflow architecture and input into a convolutional neural network model for computation.

[0083] Then, the two-dimensional matrix corresponding to the URL to be detected is formatted so that the length of the multiple encoded vectors in the two-dimensional matrix is ​​the same as the preset length. In some embodiments, the formatting of the two-dimensional matrix is ​​achieved by padding the multiple encoded vectors in the two-dimensional matrix with padding vectors consisting of 0s on both sides.

[0084] For example, taking a one-dimensional encoding vector as an example, the padding vectors corresponding to an encoding vector include a first padding vector ep1 that fills the left side of the encoding vector and a second padding vector ep2 that fills the right side of the encoding vector. ep1 = {e pad1 , ..., e pad1} Dimension ep2={e pad2 , ..., e pad2} Dimension e pad1 =e pad2 =0. `col` is the preset maximum number of columns for a two-dimensional matrix. For example, the maximum number of columns is the maximum number of characters in all reference URLs. n i This represents the vector dimension of each encoded vector. For example, it can be expressed using the formula... This represents the formatting process, where E is a two-dimensional matrix composed of the various encoded vectors. pad A two-dimensional matrix consisting of padding vectors for each encoding vector.

[0085] By transforming the coded vectors (decomposed vectors) of a two-dimensional matrix E into vectors of uniform length through vector padding, and representing each padded coded vector in column form, we obtain the padded matrix E. fin , as a formatted two-dimensional matrix.

[0086] Filling matrix E fin The vector data of the URL to be detected can be placed at the center of the matrix, ensuring that the encoded vectors of the strings have a uniform matrix format. Formatting through vector padding can solve the problems of poor detection accuracy caused by multiple strings of the URL to be detected, varying lengths of different URLs, and the weak edge perception capability of local receptive fields.

[0087] Finally, based on the formatted two-dimensional matrix, a target machine learning model is used to detect whether the URL to be detected is a malicious URL.

[0088] For example, the computational process of convolutional neural network model A, which identifies and extracts the relevant characteristics between every two adjacent structures, is as follows: CN i,j This represents the element in the i-th row and j-th column of the extracted result. m,n This represents the weight in the m-th row and n-th column of the convolution kernel W. This represents the element in the (i+m)th row and (j+n)th column of the input information. B represents the bias term of the convolution kernel. f represents the activation function. For example, the ReLU activation function can be used.

[0089] Commonly used activation functions include ReLU (Rectified Linear Unit), Tanh (hyperbolic tangent), and Sigmoid. Compared to saturated nonlinear functions, unsaturated nonlinear functions can effectively solve the gradient explosion or vanishing gradient problem and accelerate model convergence.

[0090] Saturated nonlinear activation functions like Sigmoid and Tanh focus more on the center point (i.e., data changes around the zero point). When the input value is far from the center point, the activation function output will become the endpoint value of the activation function's range; Sigmoid will be 0 or 1, and Tanh will be -1 or 1. This is detrimental to further computation in the convolutional neural network model. In contrast, the unsaturated nonlinear function ReLU follows the formula f = max(0, CN). i,j ) activation rules.

[0091] In this invention, when the encoding of the characters contained in the URL to be detected is completed in the range of [1, 200] and 0 is used for padding, the value range of each element of the two-dimensional matrix input into the neural network model is [0, 200].

[0092] Using functions like Sigmoid or Tanh will map all data in the range [0, 200] to the range [0, 1], with most data being mapped to 1. This weakens the character and category features of the URL being detected, and may also weaken or eliminate the structural features of the URL. Therefore, it is preferable to use the ReLU function as the activation function for convolutional layers, pooling layers, and the first fully connected layer, avoiding the feature weakening caused by Sigmoid or Tanh functions.

[0093] After convolutional activation is completed in convolutional neural network model A, a set of feature matrices, i.e., feature maps F: F = [f1, f2, ..., fn]. Each feature map fk contains the feature extraction result after the k-th convolutional kernel convolves the data. The number of feature maps in convolutional neural network model A is 8.

[0094] The feature maps obtained from the convolutional layers are further fed into pooling layers (also known as downsampling layers). The main function of the pooling layer is to reduce parameter variables while preserving the main features, thereby reducing the dimensionality of the input data. This expands the receptive field while avoiding overfitting, improving the model's generalization ability, and ensuring that the convolutional neural network has better scalability for feature extraction of the target URL in practical applications. To obtain the optimal features from the feature maps, the convolutional neural network model A selects max-pooling with a filter size of m×n (2×1). The pooling operation can be expressed as pool = max(CN) i:i+m,j:j+n That is, the value of `pool` is the maximum value among the elements located between row i and row i+m and between column j and column j+n in the convolutional layer extraction result. Since the padding bits are zero in the above embodiment, max-pooling can better extract the actual features contained in the URL to be detected and reduce noisy features.

[0095] After the pooling process is complete, the pooling result is input into a fully connected layer. The fully connected layer is connected to neurons in both the upper and lower layers, allowing it to integrate the features of the target URLs that have class-discriminating functions from the pooling layer. Convolutional neural network model A contains two fully connected layers. The first layer summarizes the pooling layer output, consolidating the extracted data features. The output layer refines the features, further reducing the feature space and thus lowering the computational complexity of subsequent URL classification.

[0096] Convolutional neural network models B and C can be configured similarly to or similarly to convolutional neural network model A. For example, after building convolutional neural network models A, B, and C, the model parameters need to be initialized. To avoid the difficulty of manually selecting parameters, the model can be pre-trained using the variance of the data encoded by the features extracted by the convolutional neural network and the input data as the error, with the Adam optimizer, a learning rate of 0.0001, and a dropout parameter of 0.5.

[0097] In the above embodiments, multiple strings refined according to the structure of the URL to be detected and the encoding vector of the strings refined according to the character encoding corresponding to each character are used as input to the machine learning model to detect malicious URLs. This refines the features of the URL to be detected and can improve the accuracy of detecting malicious URLs.

[0098] In some embodiments, the method for detecting malicious URLs further includes cross-validating the first machine learning model using a second machine learning model other than the first machine learning model.

[0099] For example, cross-validation of the first machine learning model can be achieved in the following way.

[0100] First, the first machine learning model is used to detect whether multiple training URLs are malicious.

[0101] Secondly, based on the detection results and labels of multiple training URLs, the first detection accuracy of the first machine learning model is determined. The labels include a first label indicating that a training URL is a malicious URL and a second label indicating that a training URL is not a malicious URL.

[0102] Next, based on multiple malicious URLs, a second machine learning model is used to detect whether multiple training URLs are malicious. For example, the second machine learning model can be a random tree model, a random forest model, a Bayesian model, or a decision tree model (J48).

[0103] Then, based on the detection results for multiple training URLs and the labels of multiple training URLs, the second detection accuracy of other second machine learning models is determined.

[0104] Finally, the parameters of the first machine learning model are adjusted so that the first detection accuracy is greater than or equal to the second detection accuracy.

[0105] In the above embodiments, the random tree model, random forest model, Bayesian model, or decision tree model (J48) used for detecting malicious websites rely on a blacklist database constructed from multiple malicious websites, but their performance is relatively stable. By using these machine learning models for cross-validation and adjusting the detection accuracy of the first machine learning model, the detection accuracy of the first machine learning model can be improved.

[0106] Figure 5 This is a block diagram illustrating an apparatus for detecting malicious URLs according to some embodiments of the present disclosure.

[0107] like Figure 5 As shown, the device 5 for detecting malicious websites includes a first determination module 51, a second determination module 52, a third determination module 53, and a detection module 54.

[0108] The first determining module 51 is configured to determine multiple strings based on the structure of the URL to be detected, for example, executing... Figure 1 Step S10 is shown.

[0109] The second determining module 52 is configured to determine the character encoding corresponding to each character in each string, for example, by executing... Figure 1 Step S20 is shown.

[0110] The third determining module 53 is configured to determine the encoding vector corresponding to the string based on the character encoding corresponding to each character in the string, for example, by performing the following: Figure 1 Step S30 is shown.

[0111] Detection module 54 is configured to detect whether a URL to be detected is a malicious URL based on multiple encoded vectors corresponding to multiple strings, using a first machine learning model, for example, by executing... Figure 1 Step S40 is shown.

[0112] Figure 6 This is a block diagram illustrating an apparatus for detecting malicious URLs according to other embodiments of the present disclosure.

[0113] like Figure 6 As shown, the apparatus 6 for detecting malicious URLs includes a memory 61 and a processor 62 coupled to the memory 61. The memory 61 is used to store instructions for performing methods for detecting malicious URLs in embodiments corresponding to the present disclosure. The processor 62 is configured to perform methods for detecting malicious URLs in any of the embodiments of the present disclosure based on the instructions stored in the memory 61.

[0114] Figure 7 This is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.

[0115] like Figure 7 As shown, the computer system 70 can be represented in the form of a general computing device. The computer system 70 includes a memory 710, a processor 720, and a bus 700 connecting different system components.

[0116] The memory 710 may include, for example, system memory, non-volatile storage media, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs. The system memory may include volatile storage media, such as random access memory (RAM) and / or cache memory. The non-volatile storage media may store, for example, instructions for executing at least one of the methods for detecting malicious websites, according to corresponding embodiments. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, etc.

[0117] The processor 720 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete hardware components such as discrete gates or transistors. Accordingly, each module, such as the decision module and the determination module, can be implemented by the central processing unit (CPU) running instructions in memory to execute the corresponding steps, or by dedicated circuitry to execute the corresponding steps.

[0118] Bus 700 can use any of the various bus architectures. For example, bus architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, and Peripheral Component Interconnect (PCI) bus.

[0119] The computer system 70 may also include an input / output interface 730, a network interface 740, and a storage interface 750. These interfaces 730, 740, and 750, as well as the memory 710 and processor 720, can be connected via a bus 700. The input / output interface 730 provides a connection interface for input / output devices such as a monitor, mouse, and keyboard. The network interface 740 provides a connection interface for various networked devices. The storage interface 750 provides a connection interface for external storage devices such as floppy disks, USB flash drives, and SD cards.

[0120] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations thereof, can be implemented by computer-readable program instructions.

[0121] These computer-readable program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device to produce a machine, such that execution of the instructions by the processor produces means for implementing the functions specified in one or more boxes of the flowchart and / or block diagram.

[0122] These computer-readable program instructions may also be stored in a computer-readable storage medium. These instructions cause a computer to work in a particular manner to produce an article of manufacture, including instructions that implement the functions specified in one or more boxes in a flowchart and / or block diagram.

[0123] This disclosure may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0124] The method and apparatus for detecting malicious URLs, and the computer-storable medium described in the above embodiments, can improve the accuracy of detecting malicious URLs.

[0125] This concludes the detailed description of the method and apparatus for detecting malicious websites according to this disclosure, as well as the computer-storable medium. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

Claims

1. A method for detecting malicious URLs, comprising: Based on the structure of the URL to be detected, multiple strings are determined; Determine the character encoding corresponding to each character in each string; Determine the encoding vector corresponding to the string based on the character encoding corresponding to each character in the string; Based on multiple encoding vectors corresponding to the multiple strings, a first machine learning model is used to detect whether the URL to be detected is a malicious URL. The first machine learning model includes multiple first machine learning models with different model complexities. The URL complexity of the URL to be detected is determined based on the multiple encoding vectors corresponding to the multiple strings. Based on the URL complexity, a first machine learning model with a model complexity suitable for the URL complexity is selected from the multiple first machine learning models as the target machine learning model. The target machine learning model is then used to detect whether the URL to be detected is a malicious URL based on the multiple encoding vectors.

2. The method for detecting malicious URLs according to claim 1, wherein, Determining the character encoding for each character in each string includes: Each reference URL is decomposed into multiple reference characters; The multiple reference characters are deduplicated to obtain multiple target characters; Determine the character category to which each target character belongs; The target characters are sorted according to the priority order of character types to obtain a first character dictionary, wherein the priority order is positively correlated with the probability of character types appearing in the URL; Based on the first character dictionary, a character encoding dictionary corresponding to the plurality of target characters is determined. The character encoding dictionary includes a plurality of character encodings, and the target characters correspond one-to-one with the character encodings. From the character encoding dictionary, determine the character encoding corresponding to each character in each string.

3. The method for detecting malicious URLs according to claim 2, wherein, Determining the character encoding dictionary corresponding to the plurality of target characters includes: Count the frequency of each target character among the multiple reference characters; The second character dictionary is obtained by sorting the target characters belonging to each character category in the first character dictionary in descending order of frequency. Each target character in the second character dictionary is encoded to obtain the character encoding dictionary, wherein the order of the multiple character encodings in the character encoding dictionary is consistent with the order of the multiple target characters in the second character dictionary.

4. The method for detecting malicious URLs according to claim 1, wherein, Selecting a first machine learning model whose model complexity is compatible with the complexity of the URL, as the target machine learning model, includes: Based on the correspondence between URL complexity and model complexity, the model complexity corresponding to the URL complexity is determined as the target model complexity, and there is a one-to-one correspondence between URL complexity and model complexity. From the plurality of first machine learning models, select the first machine learning model that corresponds to the complexity of the target model as the target machine learning model.

5. The method for detecting malicious URLs according to claim 1, wherein, The URL complexity of the URL to be detected is positively correlated with the number of character codes in the multiple encoding vectors. The complexity of the first machine learning model is positively correlated with at least one of the number of convolutional layers and the number of pooling layers in the first machine learning model.

6. The method for detecting malicious URLs according to claim 1, wherein, Based on the multiple encoded vectors, and using the target machine learning model, detecting whether the URL to be detected is a malicious URL includes: Based on the multiple encoding vectors corresponding to the multiple strings, determine the two-dimensional matrix corresponding to the URL to be detected; The two-dimensional matrix corresponding to the URL to be detected is formatted so that the length of the multiple encoding vectors in the two-dimensional matrix is ​​the same as the preset length. Based on the formatted two-dimensional matrix, the target machine learning model is used to detect whether the URL to be detected is a malicious URL.

7. The method for detecting malicious URLs according to claim 1, wherein, Determining the URL complexity of the URL to be detected includes: Based on the multiple encoding vectors corresponding to the multiple strings, determine the two-dimensional matrix corresponding to the URL to be detected; The matrix complexity of the two-dimensional matrix is ​​determined to be the URL complexity.

8. The method for detecting malicious URLs according to claim 1, further comprising: The first machine learning model is cross-validated using a second machine learning model other than the first machine learning model.

9. The method for detecting malicious URLs according to claim 8, wherein, Cross-validation of the first machine learning model using a second machine learning model other than the first machine learning model includes: Using the first machine learning model, detect whether multiple training URLs are malicious URLs; Based on the detection results for the plurality of training URLs and the labels of the plurality of training URLs, the first detection accuracy of the first machine learning model is determined, wherein the labels include a first label indicating that the training URL belongs to a malicious URL and a second label indicating that the training URL does not belong to a malicious URL. Based on multiple malicious URLs, the second machine learning model is used to detect whether multiple training URLs are malicious URLs; Based on the detection results for the multiple training URLs and the tags of the multiple training URLs, the second detection accuracy of the second machine learning model is determined; The parameters of the first machine learning model are adjusted so that the first detection accuracy is greater than or equal to the second detection accuracy.

10. The method for detecting malicious URLs according to claim 8, wherein, The first machine learning model is a convolutional neural network model, and the second machine learning model is a random tree model, a random forest model, a Bayesian model, or a decision tree model.

11. The method for detecting malicious URLs according to claim 1, wherein, The structure of the URL to be detected includes a protocol part, a hostname part, a pathname part, a filename part, and a parameter part. Based on the structure of the URL to be detected, several strings are determined, including: The protocol portion, hostname portion, pathname portion, filename portion, parameter portion, and the URL itself are each defined as a string.

12. The method for detecting malicious URLs according to claim 1, wherein, Determining the encoding vector corresponding to the string includes: The character encoding corresponding to each character in each string is used to construct a one-dimensional vector according to the character order in each string, which is then determined as the encoding vector corresponding to each string.

13. An apparatus for detecting malicious URLs, comprising: The first determining module is configured to determine multiple strings based on the structure of the URL to be detected; The second determining module is configured to determine the character encoding corresponding to each character in each string; The third determining module is configured to determine the encoding vector corresponding to the string based on the character encoding corresponding to each character in the string; The detection module is configured to detect whether the URL to be detected is a malicious URL using a first machine learning model based on multiple encoding vectors corresponding to the multiple strings. The first machine learning model includes multiple first machine learning models with different model complexities. The module determines the URL complexity of the URL to be detected based on the multiple encoding vectors corresponding to the multiple strings. Based on the URL complexity, it selects a first machine learning model from the multiple first machine learning models whose model complexity is suitable for the URL complexity as a target machine learning model. The module then uses the target machine learning model to detect whether the URL to be detected is a malicious URL based on the multiple encoding vectors.

14. An apparatus for detecting malicious URLs, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to perform the method for detecting malicious URLs as described in any one of claims 1 to 12, based on instructions stored in the memory.

15. A computer-storeable medium having stored thereon computer program instructions that, when executed by a processor, implement the method for detecting malicious URLs as described in any one of claims 1 to 12.

Images