Attribute recognition method, device and electronic device

Abstract

The embodiment of the present application relates to the field of Internet technology, and discloses an attribute recognition method, device and electronic device, the method comprising: obtaining the uniform resource location system URL sequence data of the target object; inputting the URL sequence data into a preset identifier to obtain the recognition result of the target attribute of the target object; wherein the identifier extracts the characteristic information of the URL sequence data, and performs attribute recognition on the URL sequence data according to the characteristic information; the characteristic information comprises spatiotemporal characteristic information; the spatiotemporal characteristic information comprises at least one of: URL access time interval information, POI location information of the requested access point and location information of the target object. The embodiment of the present application provides a way to identify a crawler.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and more specifically, to an attribute recognition method, device and electronic device. Background Art

[0002] With the rapid development of Internet technology, the network has become a carrier of a large amount of information. In order to effectively extract and use network information, crawlers that crawl relevant web resources have emerged. Generally speaking, a crawler is a program that automatically extracts web pages. According to the established crawling target, it selectively accesses web pages and related links to obtain the required information.

[0003] However, with the development of crawler technology, it has brought certain negative impacts on the Internet industry. For example, crawlers continue to crawl data and leak data to bad industries. Therefore, it is necessary to identify crawlers in a timely manner to avoid losses to users and enterprises. Summary of the invention

[0004] An embodiment of the present application provides an attribute identification method to provide a way to identify crawlers.

[0005] Correspondingly, the embodiment of the present application also provides an attribute identification device, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0006] In order to solve the above problems, the present application discloses an attribute recognition method, which includes:

[0007] Get the target object's Uniform Resource Locator System URL sequence data;

[0008] Inputting the URL sequence data into a preset identifier to obtain an identification result of the target attribute of the target object; wherein the identifier extracts feature information of the URL sequence data and performs attribute identification on the URL sequence data according to the feature information;

[0009] The feature information includes spatiotemporal feature information; the spatiotemporal feature information includes at least one of URL access time interval information, point of interest (POI) location information requested for access, and location information of the target object.

[0010] The present application also discloses an attribute recognition device, which includes:

[0011] An acquisition module is used to acquire the uniform resource locator system URL sequence data of the target object;

[0012] An identification module, used to input the URL sequence data into a preset identifier to obtain an identification result of the target attribute of the target object; wherein the identifier extracts feature information of the URL sequence data and performs attribute identification on the URL sequence data according to the feature information;

[0013] The feature information includes spatiotemporal feature information; the spatiotemporal feature information includes at least one of URL access time interval information, point of interest (POI) location information requested for access, and location information of the target object.

[0014] An embodiment of the present application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the attribute identification method shown in the first aspect of the present application is implemented.

[0015] The embodiments of the present application further disclose a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, one or more methods described in the embodiments of the present application are implemented.

[0016] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0017] In an embodiment of the present application, Uniform Resource Locator System URL sequence data of a target object is obtained; the URL sequence data is input into a preset identifier to obtain an identification result of a target attribute of the target object; wherein the identifier extracts feature information of the URL sequence data, wherein the feature information includes spatiotemporal feature information, and the attributes of the target object are identified in combination with the spatiotemporal feature information to improve the identification accuracy.

[0018] Additional aspects and advantages of the embodiments of the present application will be partially given in the description below, which will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0020] Figure 1 A flowchart of an attribute identification method provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of a first example provided for an embodiment of the present application;

[0022] Figure 3 A schematic diagram of a second example provided in an embodiment of the present application;

[0023] Figure 4One of the schematic diagrams of the third example provided in the embodiment of the present application;

[0024] Figure 5 A second schematic diagram of a third example provided for an embodiment of the present application;

[0025] Figure 6 One of the schematic diagrams of the fourth example provided in the embodiment of the present application;

[0026] Figure 7 A second schematic diagram of a fourth example provided in an embodiment of the present application;

[0027] Figure 8 A third schematic diagram of a fourth example provided in an embodiment of the present application;

[0028] Fig. 9 A schematic diagram of a fifth example provided in an embodiment of the present application;

[0029] Fig.10 A schematic diagram of the structure of an attribute identification device provided in an embodiment of the present application;

[0030] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0032] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0033] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0034] The solution provided in the embodiments of the present application can be executed by any electronic device, such as a terminal device or a server, wherein the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here. For the technical problems existing in the prior art, the attribute identification method, device and electronic device provided by this application are intended to solve at least one of the technical problems of the prior art.

[0035] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0036] See also Figure 1 , an embodiment of the present application provides an attribute identification method, optionally, the method can be applied to a server, and the server can be an application (Application, APP) or a website server; for the sake of convenience of description, the server is used as the execution subject of the embodiment of the present application in the following, it can be understood that the execution subject of the embodiment of the present application is not limited to this.

[0037] like Figure 1 As shown in , the method may include the following steps:

[0038] Step 101, obtaining the Uniform Resource Locator System URL sequence data of the target object.

[0039] Among them, the target object can be a user, such as an APP client user or a website user; taking APP as a business service based on geographic location (Local-Business Service, LBS) as an example, web crawlers usually browse websites or APPs by writing software or scripts, and capture a large amount of website data, APP data and even user data. On the one hand, it occupies server bandwidth, resulting in the traffic of normal users being squeezed and the server bandwidth cost being increased; on the other hand, corporate or user information is crawled, resulting in the leakage and abuse of information resources.

[0040] In the embodiment of the present application, the server obtains the Uniform Resource Locator (URL) sequence data of the target object; the URL sequence data includes parameter information such as the access address and the access time, and the server obtains the URL sequence data of the target object to analyze the characteristic information of the URL sequence data and identify the target object with abnormal access. Optionally, the server can obtain the URL sequence data according to a preset period.

[0041] Step 102, input the URL sequence data into a preset identifier to obtain an identification result of the target attribute of the target object; wherein the identifier extracts feature information of the URL sequence data and performs attribute identification on the URL sequence data according to the feature information.

[0042] The identifier is pre-trained, for example, by training the identifier (or the recognition algorithm executed by the identifier) ​​using a machine learning method or a convolutional neural network method, so that it has a recognition capability that meets a preset accuracy requirement.

[0043] Target attributes include access anomaly attributes, operation anomaly attributes, crawler attributes (or the possibility of crawlers); taking crawler attributes as an example, the applicant found that for apps or websites with a high number of daily active users (Daily Active User, DAU), finding crawler accounts that account for a relatively small proportion among a large number of real users (the number may be in the tens of millions) is a label imbalance problem, and various processing methods (such as forced offline, sliding verification code, voice verification code, SMS uplink verification code) have their limitations. For example, simpler sliding verification codes are easily cracked by technical means, and stronger verifications such as voice verification codes are more disturbing to users. Therefore, identifying the target attributes of users through pre-trained recognizers can effectively reduce interference with normal users.

[0044] Specifically, the identifier extracts feature information of the URL sequence data, and the feature information includes spatiotemporal feature information; the spatiotemporal feature information includes: at least one of URL access time interval information, POI location information requested for access, and location information of the target object; wherein the URL access time interval information is the access time interval information of the URL sequence recorded in the URL sequence data, and the access time interval information is the time interval between the access times of adjacent URL sequences; the POI requested for access is the target POI of the URL sequence, and the location information of the target object is the location information when the target object triggers access to the target POI; it can be understood that the location information of the target object is the location information actively reported by the target object when using the APP or website, or the location information obtained by the server after the target object authorizes, or the location information that the target object needs to convey to the server when using server-related services.

[0045] For example, for apps or websites based on LBS services, their crawlers usually have certain LBS properties, and a geographical location can usually only browse business information within a few kilometers. Therefore, if a crawler system wants to crawl all business information, it needs to allocate different geographical locations within a limited time, and its URL sequence will usually browse more POI locations in a short time; therefore, the target attributes of the target object can be reflected through spatiotemporal feature information.

[0046] Still taking the crawler attribute as an example, the server extracts the feature information of the URL sequence in the URL sequence data through the identifier, performs attribute recognition on the URL sequence data according to the feature information, and determines the possibility that the target object is a crawler according to the recognition result. For example, the crawler attribute is represented by label X, and the value range of label X is [0, 1], 0 indicates that the possibility that the target object is a crawler is 0, and 1 indicates that the possibility that it is a crawler is 1.

[0047] In an embodiment of the present application, the Uniform Resource Locator System URL sequence data of the target object is obtained; the URL sequence data is input into a preset identifier to obtain an identification result of the target attribute of the target object; wherein the identifier extracts feature information of the URL sequence data, the feature information includes spatiotemporal feature information, and the attribute of the target object is identified in combination with the spatiotemporal feature information to improve the identification accuracy. The embodiment of the present application provides a method for identifying crawlers.

[0048] In the embodiment of the present application, before obtaining the uniform resource locator system URL sequence data of the target object, the method includes:

[0049] Get the sample URL sequence data of the sample object;

[0050] The identifier is trained based on the sample URL sequence data.

[0051] A preset number of sample objects may be collected to obtain sample URL sequence data of the sample objects; as an example, the sample URL sequence data is shown below:

[0052] 2020-12-08 00:08:13, * / * / **********;

[0053] 2020-12-08 00:08:24, * / * / **********;

[0054] 2020-12-08 00:08:43, * / * / **********;

[0055] 2020-12-08 00:09:01, * / * / **********;

[0056] 2020-12-08 00:09:13, * / * / **********;

[0057] Among them, * represents any character in the URL sequence data; in the above sample URL sequence data, each line is a URL sequence.

[0058] The features in the sample URL sequence data are extracted, and more abstract high-level features are formed by combining low-level features so that the trained model can complete complex learning tasks such as classification, thereby obtaining the identifier.

[0059] Optionally, in the embodiment of the present application, the training of the identifier according to the sample URL sequence data includes the following first to fifth steps:

[0060] In the first step, for the first URL sequence data in the sample URL sequence data, the behavioral feature information of the first URL sequence data, the spatiotemporal feature information of the first URL sequence data, and the context feature information of the first URL sequence data are extracted by a first identifier.

[0061] Among them, the first URL sequence data is any URL sequence in the sample URL sequence data; for the convenience of explanation, the first URL sequence data is taken as an example to introduce the processing process of the sample URL sequence data in the embodiment of the present application.

[0062] As a first example, see Figure 2In steps S11, S12, and S13, the first URL sequence data is first disassembled to extract action sequence (Action Token Sequence) feature information, temporal-spatial attribute (Temporal-spatialAttribute), and context attribute (Context Attribute).

[0063] Specifically, the behavior sequence information may include the subsystem information accessed by the user (sample object), for example, filtering the 2000 most common sequences in the sample URL sequence data as Action tokens, and then extracting the Action tokens included in the first URL sequence data; as an example, taking an online shopping APP as an example, Action tokens such as MP, PP, CART, etc., MP indicates that the user visits the store homepage (main page), PP indicates that the user visits the promotion page (promotion page), and CART indicates that the user adds the product to the shopping cart.

[0064] The spatiotemporal feature information includes: at least one of URL access time interval information, POI location information of the requested access, and location information of the target object; the spatiotemporal feature information is extracted from the spatiotemporal sequence information, and the spatiotemporal sequence information includes the user's access parameters, such as access time interval, POI location, user location, etc.; taking an online shopping APP with instant delivery as an example, the access time is such as the user's visit time to an online store, the POI location is such as the location information of the online store, and the user location is such as the user's location information at the access time.

[0065] The context information is the URL sequence data before and / or after the first URL sequence data. For example, a preset number of context information is counted to obtain statistical features to determine the attributes of the first URL sequence data. For example, through the context information, it is determined that the POI requests of the first URL sequence data are too frequent, which may be due to abnormal access by users.

[0066] Wherein, the first recognizer is the trained initial recognizer; optionally, see Figure 2 The initial identifier may include: a behavior sequence module ATS ( Figure 2 S11, S21 to S31), and a spatiotemporal module TSA ( Figure 2 S12, S22 to S32) and context feature information context module CA ( Figure 2 Among S13, S23 to S33.

[0067] As an example, the behavior sequence module may include a BERT (Bidirectional Encoder Representations from Transformers) network structure. Since there is a natural similarity between URL tokens and natural language words, the behavior sequence module may be set to a BERT (or a BERT-like) network structure.

[0068] Unlike the behavior sequence module, the spatiotemporal module processes time series; the Inception network can be used as the spatiotemporal module. The Inception network is based on a large number of convolutions, which can effectively process time intervals, longitude and latitude distances, and even represent information such as displacement speed. The integration of multiple Inception networks helps improve the performance of the recognizer. For example, three Inception networks are used, each randomly initialized, and the outputs are concatenated together at the terminal of the network. Optionally, the spatiotemporal module can execute the algorithm in the following formula 1:

[0069] Formula 1:

[0070]

[0071] Among them, TSA i Represents the spatiotemporal sequence information of the i-th user, i.e., input data;

[0072] Emb represents embedding operation, that is, feature vector extraction operation. Represents spatiotemporal feature information; The feature vector representing the spatiotemporal sequence information; concat represents the string concatenation operation; Represents the feature vector calculated by the first Inception network, Represents the feature vector calculated by the second Inception network; Represents the feature vector calculated by the third Inception network.

[0073] The context module can count the parameters in the URL sequence data to obtain statistical features and calculate the feature vector of the statistical features.

[0074] Optionally, the trained initial recognizer (first recognizer) can adopt an Adaptive Aggregation Network for Efficient Stereo Matching (AANet) model, which receives sample URL sequence data as input and simultaneously learns URL sequence information, spatiotemporal characteristics and contextual information.

[0075] The second step is to determine the first feature vector of the behavior feature information, the second feature vector of the spatiotemporal feature information, and the third feature vector of the context feature information.

[0076] See also Figure 2 In steps S21, S22, and S23, the feature vectors of the three feature information are calculated respectively by a preset feature vector calculation method; for example represents the first eigenvector, ATS i Represents the behavioral characteristic information of the i-th user; represents the second eigenvector; represents the third eigenvector, CA i Represents the context feature information of the i-th user.

[0077] The third step is to determine a fourth feature vector, and determine a first recognition result of the first URL sequence data according to the fourth feature vector; the fourth feature vector is a feature vector of a string obtained by concatenating the first feature vector, the second feature vector and the third feature vector.

[0078] First, the first feature vector, the second feature vector, and the third feature vector are concatenated to obtain a concatenated string concat1, as shown in the following formula 2:

[0079] Formula 2:

[0080]

[0081] Then calculate the fourth eigenvector logit i , as shown in the following formula 3:

[0082] Formula 3:

[0083]

[0084] The first recognition result is the recognition result of the target attribute of the first URL sequence data, and the first recognition result y of the first URL sequence data is determined according to the fourth feature vector. i ,y i It can be a label indicating whether the user has the target attribute. For example, when the target attribute is a crawler, y i=0 means the probability that the target object is a crawler is 0, y i =1 means the possibility of being a crawler is 1; y i The calculation method of can be shown as the following formula 4:

[0085] Formula 4:

[0086]

[0087] Among them, λ is the preset threshold; when the fourth eigenvector logit i Less than λ, y i =0 means that the probability that the target object has the target attribute is 0; in other cases, that is, the fourth eigenvector is greater than or equal to λ, y i =1 means that the probability that the target object has the target attribute is 1.

[0088] The fourth step is to determine the loss function value of the first recognition result, and adjust the first identifier according to the loss function value to obtain a second identifier.

[0089] Among them, the sample object has a pre-known label value (hereinafter referred to as the true label value), and the true label value is the true value of the target attribute of the sample object; the loss function value is calculated by the difference between the true value and the first recognition result, and the first identifier is reversely optimized according to the loss function value to obtain the second identifier.

[0090] As an example, the loss function can be shown as the following formula 5:

[0091] Formula 5:

[0092]

[0093] Among them, Loss represents the loss function value, x i =(ATS i , TSA i , C.A. i ), D represents the sample URL sequence dataset; c∈{0,1} represents the value range of the label, p c (x i ) represents x i The probability of belonging to c.

[0094] The fifth step is to identify the second URL sequence data in the sample URL sequence data through a second identifier, until the identifier that meets the preset loss function value requirement is obtained.

[0095] Subsequently, the second URL sequence data in the sample URL sequence data is used to train the second identifier, the loss function value is calculated, and the second identifier is optimized according to the loss function value, ..., and the cycle is executed until the identifier that meets the preset loss function value requirements is obtained.

[0096] Optionally, in the embodiment of the present application, in the first step, extracting the behavior feature information of the first URL sequence data includes:

[0097] Extracting the behavior sequence data before the first target identifier in the first URL sequence data;

[0098] Extracting behavior feature information from the behavior sequence data.

[0099] Specifically, the first target identifier is used to indicate the user's access behavior, for example, it can be "?". In the URL sequence, the characters before the first target identifier (for example, adjacent to the first target identifier) ​​represent the user's access behavior. Taking the online shopping APP as an example, the access behaviors include MP, PP, CART, etc. MP indicates that the user visits the store's main page, PP indicates that the user visits the promotion page, and CART indicates that the user adds the product to the shopping cart.

[0100] Usually, in a URL sequence, some identifiers indicate specific meanings; for example, "CLS" indicates the start of a URL sequence; "SEP" indicates a date separator, representing the beginning or end of a day; and "PAD" is used to align different sequence lengths to a consistent length.

[0101] Optionally, in the embodiment of the present application, in the first step, extracting the spatiotemporal feature information of the first URL sequence data includes:

[0102] Extracting the spatiotemporal feature information after the second target identifier in the first URL sequence data; wherein the spatiotemporal feature information includes the URL access time interval information, the POI location information requested to be accessed, and the location information of the sample object.

[0103] Specifically, the second target identifier is used to indicate the user's access behavior, for example, it can also be "?". The spatiotemporal feature information of the sample object and the spatiotemporal feature information of the target object have the same parameter type, including the URL access time interval information, the POI location information requested for access, and the location information of the sample object; optionally, time_gap can be used to represent the time interval between two URL sequences; the POI location information and the location information of the sample object can be represented by longitude and latitude, and the longitude and latitude of the POI location information are represented by log_poi and lat_poi, respectively, and the longitude and latitude of the location information of the sample object are represented by lon_user and lat_user, respectively. Normally, the longitude and longitude of the POI requested by the user are consistent with the longitude and latitude of the user (or the distance is small). If the two are far apart, it represents an abnormal signal of the user; in addition, the continuously changing requested POI and the small interval between the changes in the user's location may be abnormal behavior.

[0104] Optionally, in the embodiment of the present application, extracting the context feature information of the first URL sequence data includes:

[0105] Determine context feature information of each access parameter of the first URL sequence data; the context feature information includes at least one of the following: number of occurrences, number of occurrences without duplication, and information entropy;

[0106] Among them, the access parameters of the URL sequence are, for example, geohash, shop_id, business channel, etc. Optionally, there may be multiple context feature information in the URL sequence, such as string-like information. In the context feature information, the number of occurrences, the number of deduplications, and the information entropy have a certain correlation with the target attribute; still taking the crawler attribute as an example, for example, when the context feature information includes POI_name, the crawler will access a lot of POI_names to crawl as much merchant data as possible, so the number of occurrences is large, and the same POI_name will be deduplicated, and the number of deduplications is also large; the information entropy indicates the amount of information in the context feature information, such as the information entropy of RND (random, used for AB test bucketing of online systems). It is possible that the crawler forgets to replace the parameter, resulting in the information entropy of this indicator being 0.

[0107] As an example, if the URL sequence includes thousands of parameters, each of which uses the number of occurrences (count), the number of duplicates (distinct count), and the information entropy (entropy) to form a feature vector of several thousand dimensions, the process of calculating the feature vector is shown in the following formulas 6 to 8:

[0108] Formula 6: V0 = Flatten(Xi ca )

[0109] Formula 7: V1 ca =ρ(W1 ca V0+b1 ca )

[0110] Formula 8: Emb i ca =ρ(W2 ca V1 ca +b2 ca )

[0111] Wherein, Flatten represents the vector flattening function; ρ is the preset activation function. k ca represents the weight matrix of the kth layer, b k ca represents the bias rate of the kth layer; V0 represents the feature vector of any layer, V1 ca Represents the feature vector of the first layer, Emb i ca Represents the feature vector of the second layer.

[0112] Optionally, in the embodiment of the present application, the step of training the identifier according to the sample URL sequence data includes:

[0113] Step 1: training a third recognizer through a masked language model according to the sample URL sequence data;

[0114] Step 2: obtaining first sample data in the mask sample of the third identifier, obtaining auxiliary parameters in the first sample data, training the third identifier, and obtaining a fourth identifier;

[0115] Step three: extract the second sample data from the first sample data, train the fourth identifier, and obtain the identifier.

[0116] The third recognizer is the initial recognizer; in the process of training the initial recognizer, it can be trained based on the Masked Language Model (MLM), and then an auxiliary task is used to further train the Embedding (feature vector) of the entire sequence. Finally, the recognizer is trained on a small number of labeled samples (labeled samples are samples with known target attributes).

[0117] Specifically, as a second example, see Figure 3In step 1, a self-supervised pre-training method is used, and the number of samples is N1, which is a number in the tens of millions. For example, the sample URL sequence data includes 90 million samples; based on the sample URL sequence data, the third identifier is trained; for example, 15% of the sequence is randomly masked as a mask sample (MASK), and the target attribute of the first sample data is predicted by the third identifier. In the mask sample, 80% of the masked tokens are further selected to be replaced with special characters [MASK], 10% are replaced with random characters, and 10% retain the original characters.

[0118] In step two, auxiliary task training is used to help the task to be trained. For example, in the crawler identification scenario, whether the user has placed an order within n days after being identified as a crawler is highly correlated with the crawler attribute label. Therefore, by training the third identifier as an auxiliary task, the auxiliary task is pre-set, for example, the number of samples is N2 (N2<N1), N2 is a number in the tens of millions, for example, including 10 million samples, the auxiliary task is to predict through the third identifier whether the user in the first sample data will place an order in the next n days, and optimize the third identifier through the known results of the labeled samples, eliminate samples that do not have crawler attributes, and use tens of millions of samples to train the third identifier to obtain the fourth identifier.

[0119] In step 3, the main task is trained to predict a user's target attribute. In the prediction of the main task, the second sample data in the first sample data is extracted, the training samples are reduced in one step, and the samples without crawler attributes are eliminated. The fourth recognizer is trained using tens of thousands of samples (the number of samples is N3, N3 is a number of tens of thousands), and the recognizer is obtained.

[0120] As an example, still taking the target attribute as the crawler attribute, the server obtains the URL data of the user accessing the gateway as the sample URL sequence data; for example, the original data has 10 billion access records, and after aggregation by account, 10 million URL sequence samples are formed. Using d days of data, the first k tokens are extracted every day, and the special character [SEP] is used to split the daily sequence, plus the initial [CLS] character, the sequence length is T = d*100+d+1.

[0121] In the pre-training phase of step 1, both normal users and crawler accounts will participate in this phase of pre-training, because this phase aims to predict a small number of masked tokens through a large amount of sequence data. In the auxiliary task training, some normal accounts will be filtered out and will not participate in the auxiliary task training.

[0122] Optionally, an ablation experiment is performed on three modules as a third example. Figure 4 The simulation results of the ablation experiment for auxiliary task training are shown in Figure 2. Figure 5Represents the simulation results of the ablation experiment of the main task training; P90R represents precision and recall; AUC represents the area under the curve, which is the area enclosed by the ROC curve and the coordinate axis. The larger the AUC, the better the classification effect of the model; KS (Kolmogorov-Smirnov) represents a statistic used to measure the model's discrimination of positive and negative samples. The larger its value, the stronger the model's ability to distinguish between positive and negative samples; F1 score is a measurement indicator for classification problems, which is the harmonic mean of precision and recall, with a maximum of 1 and a minimum of 0.

[0123] Any combination of two modules usually performs better than either module alone, and all three modules work best; for example, see Figure 4 After the behavior sequence module ATS and the context module CA are combined (ATS+CA), most evaluation indicators have a certain increase; after the three modules are combined (ATS+TAS+CA), the values ​​of each evaluation indicator are the highest. Figure 5 , this effect is more obvious, and I will not go into details.

[0124] To further illustrate the effect of three-stage progressive training, as a fourth example, see Figures 6 to 8 ,The pre-training task in step 1 and the auxiliary task in step 2 are also trained to different intensities to examine their impact on the performance of other tasks. Figures 6 to 8 In the figure, the horizontal axis represents the task training intensity (such as accuracy), and the vertical axis represents the impact on the performance of other tasks.

[0125] Figure 6 The relationship between the self-supervised pre-training task and the auxiliary task in step 1 is shown. Only when the accuracy of the pre-training task reaches a relatively high value (greater than 60%), the auxiliary task can be improved.

[0126] Figure 7 Show the performance relationship between the auxiliary tasks and the main task in step 3, according to Figure 7 It can be seen that if there is no pre-training process in step one, the training intensity of the auxiliary task will have little effect on the performance of the main task in step three.

[0127] Figure 8 The performance relationship between self-supervised pre-training and the main task in step 3 is shown. Only when the pre-training task reaches a higher accuracy (greater than 60%) does the gap between introducing auxiliary tasks begin to widen.

[0128] based on Figures 6 to 8The sensitivity study of the three training stages shows that step one (self-supervised pre-training) and step two (auxiliary tasks) help improve the performance of step three (main task); the performance of the main task is more sensitive to the accuracy of self-supervised pre-training. Significant improvements in the main task depend on high accuracy of self-supervised pre-training and auxiliary task training (for example, accuracy exceeding 60%).

[0129] Further, as a fifth example, in combination Fig. 9 We also conducted comparative experiments on the performance improvements of various algorithms, including GBDT (Gradient Boosting Decision Tree), LSTM (Long Short-Term Memory), InceptionTime algorithm, ASE (Attributed Sequence Embedding) algorithm, based on Fig. 9 It can be seen that AANet surpasses other algorithms in all kinds of indicators, especially P90R.

[0130] In an embodiment of the present application, the Uniform Resource Locator System URL sequence data of the target object is obtained; the URL sequence data is input into a preset identifier to obtain an identification result of the target attribute of the target object; wherein the identifier extracts feature information of the URL sequence data, the feature information includes spatiotemporal feature information, and the attribute of the target object is identified in combination with the spatiotemporal feature information to improve the identification accuracy. The embodiment of the present application provides a method for identifying crawlers.

[0131] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application also provides an attribute recognition device, such as Fig.10 As shown, the device comprises:

[0132] The acquisition module 1001 is used to acquire the uniform resource locator system URL sequence data of the target object.

[0133] Among them, the target object can be a user, such as an APP client user or a website user; taking APP as a business service based on geographic location (Local-Business Service, LBS) as an example, web crawlers usually browse websites or APPs by writing software or scripts, and capture a large amount of website data, APP data and even user data. On the one hand, it occupies server bandwidth, resulting in the traffic of normal users being squeezed and the server bandwidth cost being increased; on the other hand, corporate or user information is crawled, resulting in the leakage and abuse of information resources.

[0134] In the embodiment of the present application, the server obtains the Uniform Resource Locator (URL) sequence data of the target object; the URL sequence data includes parameter information such as the access address and the access time, and the server obtains the URL sequence data of the target object to analyze the characteristic information of the URL sequence data and identify the target object with abnormal access. Optionally, the server can obtain the URL sequence data according to a preset period.

[0135] The identification module 1002 is used to input the URL sequence data into a preset identifier to obtain an identification result of the target attribute of the target object; wherein the identifier extracts feature information of the URL sequence data and performs attribute identification on the URL sequence data according to the feature information.

[0136] The feature information includes spatiotemporal feature information; the spatiotemporal feature information includes at least one of URL access time interval information, point of interest (POI) location information requested for access, and location information of the target object.

[0137] The identifier is pre-trained, for example, by training the identifier (or the recognition algorithm executed by the identifier) ​​using a machine learning method or a convolutional neural network method, so that it has a recognition capability that meets a preset accuracy requirement.

[0138] Target attributes include access anomaly attributes, operation anomaly attributes, crawler attributes (or the possibility of crawlers); taking crawler attributes as an example, the applicant found that for apps or websites with a high number of daily active users (Daily Active User, DAU), finding crawler accounts that account for a relatively small proportion among a large number of real users (the number may be in the tens of millions) is a label imbalance problem, and various processing methods (such as forced offline, sliding verification code, voice verification code, SMS uplink verification code) have their limitations. For example, simpler sliding verification codes are easily cracked by technical means, and stronger verifications such as voice verification codes are more disturbing to users. Therefore, identifying the target attributes of users through pre-trained recognizers can effectively reduce interference with normal users.

[0139] Specifically, the identifier extracts feature information of the URL sequence data, and the feature information includes spatiotemporal feature information; the spatiotemporal feature information includes: at least one of URL access time interval information, POI location information requested for access, and location information of the target object; wherein the URL access time interval information is the access time interval information of the URL sequence recorded in the URL sequence data, and the access time interval information is the time interval between the access times of adjacent URL sequences; the POI requested for access is the target POI of the URL sequence, and the location information of the target object is the location information when the target object triggers access to the target POI; it can be understood that the location information of the target object is the location information actively reported by the target object when using the APP or website, or the location information obtained by the server after the target object authorizes, or the location information that the target object needs to convey to the server when using server-related services.

[0140] For example, for apps or websites based on LBS services, their crawlers usually have certain LBS properties, and a geographical location can usually only browse business information within a few kilometers. Therefore, if a crawler system wants to crawl all business information, it needs to allocate different geographical locations within a limited time, and its URL sequence will usually browse more POI locations in a short time; therefore, the target attributes of the target object can be reflected through spatiotemporal feature information.

[0141] Still taking the crawler attribute as an example, the server extracts the feature information of the URL sequence in the URL sequence data through the identifier, performs attribute recognition on the URL sequence data according to the feature information, and determines the possibility that the target object is a crawler according to the recognition result. For example, the crawler attribute is represented by label X, and the value range of label X is [0, 1], 0 indicates that the possibility that the target object is a crawler is 0, and 1 indicates that the possibility that it is a crawler is 1.

[0142] In an optional embodiment, the device comprises:

[0143] A sample acquisition module is used to obtain sample URL sequence data of a sample object;

[0144] The training module is used to train the identifier according to the sample URL sequence data.

[0145] In an optional embodiment, the training module includes:

[0146] An extraction submodule, configured to extract, for the first URL sequence data in the sample URL sequence data, behavioral feature information of the first URL sequence data, spatiotemporal feature information of the first URL sequence data, and context feature information of the first URL sequence data through a first identifier;

[0147] A first determination submodule, used to determine a first feature vector of the behavior feature information, a second feature vector of the spatiotemporal feature information, and a third feature vector of the context feature information;

[0148] A second determination submodule is used to determine a fourth feature vector, and determine a first recognition result of the first URL sequence data according to the fourth feature vector; the fourth feature vector is a feature vector of a string obtained by concatenating the first feature vector, the second feature vector and the third feature vector;

[0149] A third determination submodule is used to determine a loss function value of the first recognition result, and adjust the first identifier according to the loss function value to obtain a second identifier;

[0150] The identification submodule is used to identify the second URL sequence data in the sample URL sequence data through a second identifier, so as to obtain the identifier that meets the preset loss function value requirement.

[0151] In an optional embodiment, the extraction submodule is used to:

[0152] Extracting the behavior sequence data before the first target identifier in the first URL sequence data;

[0153] Extracting behavior feature information from the behavior sequence data.

[0154] In an optional embodiment, the extraction submodule is used to:

[0155] Extracting the spatiotemporal feature information after the second target identifier in the first URL sequence data; wherein the spatiotemporal feature information includes the URL access time interval information, the POI location information requested to be accessed, and the location information of the sample object.

[0156] In an optional embodiment, the extraction submodule is used to:

[0157] Determine context feature information of each access parameter of the first URL sequence data; the context feature information includes at least one of the following: number of occurrences, number of occurrences without duplication, and information entropy;

[0158] In an optional embodiment, the identification submodule is used to:

[0159] According to the sample URL sequence data, training a third recognizer through a masked language model;

[0160] Obtaining first sample data in the mask sample of the third identifier, obtaining auxiliary parameters in the first sample data, training the third identifier, and obtaining a fourth identifier;

[0161] The second sample data in the first sample data is extracted, and the fourth identifier is trained to obtain the identifier.

[0162] The attribute recognition device provided in the embodiment of the present application can achieve Figures 1 to 9 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0163] The attribute recognition device provided by the present application comprises an acquisition module 1001 which acquires the Uniform Resource Locator System URL sequence data of the target object; an identification module 1002 which inputs the URL sequence data into a preset identifier to obtain an identification result of the target attribute of the target object; wherein the identifier extracts feature information of the URL sequence data, the feature information includes spatiotemporal feature information, and the attribute recognition of the target object is performed in combination with the spatiotemporal feature information to improve the recognition accuracy.

[0164] The attribute identification device of the embodiment of the present application can execute the attribute identification method provided by the embodiment of the present application, and the implementation principle is similar. The actions performed by each module and unit in the attribute identification device in each embodiment of the present application correspond to the steps in the attribute identification method in each embodiment of the present application. For the detailed functional description of each module of the attribute identification device, please refer to the description in the corresponding attribute identification method shown in the previous text, which will not be repeated here.

[0165] Based on the same principle as the method shown in the embodiment of the present application, the embodiment of the present application also provides an electronic device, which may include but is not limited to: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the attribute recognition method shown in any optional embodiment of the present application by calling the computer program. Compared with the prior art, the attribute recognition method provided by the present application obtains the uniform resource location system URL sequence data of the target object; inputs the URL sequence data into a preset identifier to obtain the recognition result of the target attribute of the target object; wherein the identifier extracts the feature information of the URL sequence data, the feature information includes spatiotemporal feature information, and combines the spatiotemporal feature information to perform attribute recognition on the target object to improve the recognition accuracy.

[0166] In an optional embodiment, an electronic device is also provided, such as Fig.11 As shown, Fig.11The electronic device 11000 shown may be a server, including: a processor 11001 and a memory 11003. The processor 11001 and the memory 11003 are connected, such as through a bus 11002. Optionally, the electronic device 11000 may also include a transceiver 11004. It should be noted that in actual applications, the transceiver 11004 is not limited to one, and the structure of the electronic device 11000 does not constitute a limitation on the embodiments of the present application.

[0167] Processor 11001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 11001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0168] The bus 11002 may include a path to transmit information between the above components. The bus 11002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 11002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0169] Memory 11003 can be ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or it can be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0170] The memory 11003 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 11001. The processor 11001 is used to execute the application code stored in the memory 11003 to implement the contents shown in the above method embodiment.

[0171] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.11 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0172] The server provided in this application may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud Internet, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal may be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0173] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0174] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0175] It should be noted that the computer-readable storage medium mentioned above in the present application can also be a computer-readable signal medium or a combination of a computer-readable storage medium and a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0176] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0177] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0178] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the attribute identification method provided in the above-mentioned various optional implementations.

[0179] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0180] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0181] The modules involved in the embodiments described in the present application may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, module A may also be described as "module A for performing operation B".

[0182] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method for attribute recognition, characterized in that: include: Get the target object's Uniform Resource Locator System URL sequence data; Inputting the URL sequence data into a preset identifier to obtain an identification result of the target attribute of the target object; wherein the identifier extracts feature information of the URL sequence data and performs attribute identification on the URL sequence data according to the feature information; The feature information includes spatiotemporal feature information; the spatiotemporal feature information includes at least one of: URL access time interval information, point of interest (POI) location information requested for access, and location information of the target object; Before obtaining the uniform resource locator system URL sequence data of the target object, the method includes: Get the sample URL sequence data of the sample object; According to the sample URL sequence data, the identifier is trained; The step of training the identifier according to the sample URL sequence data comprises: The identifier is trained based on the behavioral feature information, spatiotemporal feature information and extracted context feature information of the sample URL sequence data.

2. The attribute identification method according to claim 1, characterized in that: The step of training the identifier based on the behavior feature information, spatiotemporal feature information and context feature information of the sample URL sequence data includes: For the first URL sequence data in the sample URL sequence data, extracting the behavioral feature information of the first URL sequence data, extracting the spatiotemporal feature information of the first URL sequence data, and extracting the context feature information of the first URL sequence data through a first identifier; Determine a first feature vector of the behavior feature information, a second feature vector of the spatiotemporal feature information, and a third feature vector of the context feature information; Determine a fourth feature vector, and determine a first recognition result of the first URL sequence data according to the fourth feature vector; the fourth feature vector is a feature vector of a string obtained by concatenating the first feature vector, the second feature vector, and the third feature vector; Determine a loss function value of the first recognition result, and adjust the first recognizer according to the loss function value to obtain a second recognizer; The second URL sequence data in the sample URL sequence data is identified by a second identifier until the identifier that meets the preset loss function value requirement is obtained.

3. The attribute identification method according to claim 2, characterized in that: The extracting the behavior feature information of the first URL sequence data includes: Extracting the behavior sequence data before the first target identifier in the first URL sequence data; Extracting behavior feature information from the behavior sequence data.

4. The attribute identification method according to claim 2, characterized in that: The step of extracting the spatiotemporal feature information of the first URL sequence data includes: Extracting the spatiotemporal feature information after the second target identifier in the first URL sequence data; wherein the spatiotemporal feature information includes the URL access time interval information, the POI location information requested to be accessed, and the location information of the sample object.

5. The attribute identification method according to claim 4, characterized in that: The extracting the context feature information of the first URL sequence data includes: Determine context feature information of each access parameter of the first URL sequence data; the context feature information includes at least one of the following: number of occurrences, number of occurrences without duplication, and information entropy.

6. The attribute identification method according to claim 1, characterized in that: The step of training the identifier according to the sample URL sequence data comprises: According to the sample URL sequence data, training a third recognizer through a masked language model; Obtaining first sample data in the mask sample of the third identifier, obtaining auxiliary parameters in the first sample data, training the third identifier, and obtaining a fourth identifier; The second sample data in the first sample data is extracted, and the fourth identifier is trained to obtain the identifier.

7. An attribute recognition device, characterized in that: include: An acquisition module is used to acquire the uniform resource locator system URL sequence data of the target object; An identification module, used to input the URL sequence data into a preset identifier to obtain an identification result of the target attribute of the target object; wherein the identifier extracts feature information of the URL sequence data and performs attribute identification on the URL sequence data according to the feature information; The feature information includes spatiotemporal feature information; the spatiotemporal feature information includes at least one of: URL access time interval information, point of interest (POI) location information requested for access, and location information of the target object; A sample acquisition module is used to obtain sample URL sequence data of a sample object; A training module, used for training the identifier according to the sample URL sequence data; The step of training the identifier according to the sample URL sequence data comprises: The identifier is trained based on the behavioral feature information, spatiotemporal feature information and extracted context feature information of the sample URL sequence data.

8. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the method according to any one of claims 1 to 6 is implemented when the processor executes the program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images