Webpage asset enterprise affiliation identification method based on BERT language model

Abstract

The invention relates to the technical field of internet asset management, and discloses a webpage asset enterprise affiliation identification method based on a BERT language model, which comprises the following steps: S1, acquiring an HTML source code of webpage assets through a crawler technology, and analyzing and extracting a title and body content of a webpage; s2, performing text processing on the title and the body content to obtain a to-be-recognized text; s3, inputting the to-be-recognized text into a trained BERT language model, and outputting an enterprise affiliation recognition result of a webpage through semantic understanding and feature interaction calculation; and S4, carrying out manual verification on the enterprise affiliation identification result, and updating the abnormal text as a new sample to the BERT language model so as to complete the iterative optimization of the model. According to the method and the device, the BERT language model is constructed by training and learning the webpage data, so that the automatic identification of the enterprise attribution of the webpage assets is realized, and the identification efficiency and accuracy are greatly improved.

Description

Technical Field

[0001] This application relates to the field of internet asset management technology, specifically a method for identifying the ownership of web page assets based on the BERT language model. Background Technology

[0002] With the rapid development of internet technology, cyberspace has become a crucial national strategic resource. As an important component of cyberspace, the security and management of cyber assets are of paramount importance for safeguarding national cyber sovereignty and ensuring network security. However, the sheer number and diverse types of cyber assets, coupled with their constant evolution, pose significant challenges to their management and ownership identification. Traditional methods for identifying cyber assets primarily include keyword matching, regular expression matching, and text correlation analysis. However, these methods are inefficient and prone to errors when dealing with complex and ever-changing network environments and large volumes of assets.

[0003] Currently, the identification of corporate ownership of web pages in asset identification mainly involves obtaining the HTML source code of the target web page through web crawling technology or API interfaces, then parsing the HTML source code to extract the title and body content of the web page, and then using keyword and regular expression matching, combined with keyword and regular expression libraries, to obtain the corporate ownership of the web page assets.

[0004] Currently, the identification of corporate ownership of web assets in asset recognition mainly relies on keyword and regular expression matching, but these methods have the following drawbacks:

[0005] 1. Although keyword and regular expression matching methods are simple and easy to implement, they may be affected by interfering information in the web page content, leading to inaccurate recognition results.

[0006] 2. Keywords and regular expressions can only match predefined patterns and may not be able to recognize unforeseen patterns.

[0007] 3. As web page design and content change, keyword libraries and regular expressions need to be constantly updated and maintained to adapt to new patterns and avoid becoming obsolete. This requires continuous human investment and resource consumption.

[0008] 4. Due to the diversity and complexity of web page content, keywords and regular expressions may produce false positives (mistaking non-target content for target content) or false negatives (failing to identify the actual target content). For example, similar words or phrases may have different meanings in different contexts, and static matching has difficulty distinguishing such contextual differences.

[0009] Therefore, there is an urgent need for a web asset ownership identification technology that is both efficient and accurate. Summary of the Invention

[0010] The purpose of this application is to provide a method for identifying the ownership of web page assets based on the BERT language model, so as to solve the technical problems mentioned in the background art, realize the automatic identification of the ownership of web page assets, and improve the efficiency and accuracy of identification.

[0011] To achieve the above objectives, this application discloses the following technical solution: a method for identifying enterprise ownership of web page assets based on the BERT language model, the method comprising the following steps:

[0012] S1: Obtain the HTML source code of web page assets through web crawling technology, and parse and extract the title and body content of the web page;

[0013] S2: Perform text processing on the title and the body content to obtain the text to be recognized;

[0014] S3: Input the text to be identified into the trained BERT language model, and output the enterprise affiliation identification result of the webpage through semantic understanding and feature interaction calculation;

[0015] S4: Manually verify the enterprise affiliation identification results, and update the BERT language model with the identified abnormal text as new samples to complete the model iterative optimization.

[0016] Preferably, the text processing in S2 specifically includes:

[0017] S21: Extract text that conforms to Chinese language rules from the body content;

[0018] S22: If the length of the extracted text exceeds 100,000 Chinese characters, the first 50,000 Chinese characters and the last 50,000 Chinese characters are extracted and concatenated to form the text to be identified.

[0019] Preferably, the training steps of the BERT language model in S3 include:

[0020] S31: Obtain keywords from the enterprise database, manually construct positive template data and negative template data, and assemble them to form a sample dataset, which includes content and the corresponding enterprise tags;

[0021] S32: Add a positive and negative sample label column to the sample dataset, and divide the sample dataset into a training set, a test set, and a validation set;

[0022] S33: The model is trained based on the training set, the test set, and the validation set as inputs to the BERT Chinese pre-trained model to generate a BERT language model for enterprise affiliation identification, wherein the training includes model training, model evaluation, model testing, and model tuning.

[0023] Preferably, the input to the BERT Chinese pre-trained model is used to train the model, specifically including:

[0024] S331: Vectorize the input text content into a text tensor, and vectorize the candidate companies into candidate company vectors;

[0025] S332: The interaction between the operator tensor and the text tensor is calculated through an attention mechanism to obtain the feature tensor;

[0026] S333: Aggregate the candidate enterprise vectors to generate an aggregated candidate enterprise vector;

[0027] S334: Calculate the interaction between the candidate enterprise aggregation vector and the feature tensor through an attention mechanism to obtain the text interaction result;

[0028] S335: Perform dot product calculation and feature summation on the candidate enterprise aggregation vector and the text interaction result to obtain the candidate score;

[0029] S336: Calculate the loss function based on the candidate scores, optimize the model parameters batch by batch, and maximize the true positive (TP) and true negative (TN) indices of the predicted attribution enterprise and the labeled enterprise, while minimizing the false positive (FP) and false negative (FN) indices.

[0030] Preferably, the manual verification in S4 specifically includes:

[0031] S41: Mark the erroneous text in the recognition results and use it as the text of the recognition anomaly.

[0032] Preferably, the model iterative optimization in S4 further includes:

[0033] S42: Collect the labeled normal text and misidentified text, supplement the sample dataset of the BERT language model, and periodically trigger model retraining.

[0034] Preferably, the calculation process of the attention mechanism specifically includes:

[0035] The Query-Key / Value mechanism is used to perform feature interaction between the text tensor and the candidate enterprise vector, generating the feature tensor and the text interaction result.

[0036] Preferably, the Query-Key / Value mechanism specifically includes:

[0037] S3301: Denote the tensor as T text The dimensions are [batch, context_len, dim]; the candidate enterprise vector is denoted as V. candThe dimensions are [cand_num, cand_len, dim];

[0038] S3302: Extract the Query vector Q from the text tensor. The expression for vector Q is:

[0039]

[0040] in, T is a weight vector dynamically generated based on the semantic importance of webpage content. text [i] represents the element at the i-th position of the text tensor;

[0041] S3303: Extract the Key vector K from the text tensor. The expression for vector K is:

[0042]

[0043] Where, α j T is an adjustment factor determined based on the syntactic structure and semantic coherence of the webpage text, with a value ranging from 0 to 1. text [j] represents the element at position j of the text tensor;

[0044] S3304: Extract the Value vector V from the candidate enterprise vector. The expression for vector V is:

[0045]

[0046] Among them, V cand [k] represents the element at the k-th position of the candidate enterprise vector;

[0047] S3304: Calculate the feature tensor, wherein the formula for calculating the feature tensor is:

[0048]

[0049] Where σ is the activation function; K T Let K be the transpose of vector K, β be the balance coefficient, and its value ranges from 0 to 1; K[m] represents the element of vector K in the m-th dimension;

[0050] S3305: Calculate the text interaction result, wherein the formula for calculating the text interaction result is:

[0051]

[0052] Where tanh is the hyperbolic tangent function, F[n] represents the element of the feature tensor at the nth position, and V cand [n] represents the element at the nth position of the candidate enterprise vector, and θ is a bias parameter that is dynamically adjusted based on the model training error.

[0053] Preferably, S4 further includes: during each model iteration optimization, adjusting the weight vector, adjustment coefficient, balance coefficient, and bias parameter based on the recognition accuracy (Acc) and recall (Rec) of different webpage samples during training; wherein:

[0054] The adjustment of the weight vector is specifically as follows:

[0055]

[0056] ω is a reference coefficient. Δw i The amount of fine-tuning is determined based on the update frequency of words or phrases in the webpage content; This is the weight vector for this training session;

[0057] The adjustment of the adjustment coefficient is specifically as follows:

[0058] α j(new) =α j(old) ·(1-ω·Δα j ), Δα j The adjustment amount is determined based on the degree of change in syntactic structure and semantic coherence; α j(old) This is the adjustment factor for this training session;

[0059] The adjustment of the balance coefficient is specifically as follows:

[0060] β (new) =β (old) +ω·(β target -β (old) )

[0061] β target A target balance coefficient pre-set to achieve optimal recognition results for different types of web page data; β (old) This is the balance coefficient during this training session;

[0062] The adjustment of the bias parameter is specifically as follows:

[0063] θ (new) =θ (old) +ω·Δθ)

[0064] Δθ is an adjustment value determined based on the changing trend of the model training error.

[0065] Preferably, the BERT language model takes a text tensor of dimension [batch, context_len] as input and outputs candidate scores of dimension [batch, cand_num]. After normalization using the Softmax function, the company corresponding to the highest score is determined.

[0066] Beneficial effects: The webpage asset enterprise ownership identification method based on the BERT language model in this application constructs a BERT language model by training and learning webpage data, thereby realizing the automatic identification of webpage asset enterprise ownership. At the same time, it can update the abnormal identification text into the BERT language model, which greatly improves the efficiency and accuracy of identification and reduces the subsequent manual verification process. It can be widely applied to network security products such as Internet asset exposure surface management and attack surface management. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 A flowchart illustrating the webpage asset enterprise ownership identification method based on the BERT language model provided in this application embodiment.

[0069] Figure 2 This is a schematic diagram illustrating the construction process of the BERT language model provided in an embodiment of this application. Detailed Implementation

[0070] The technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0071] In this document, the term "comprising" is intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0072] Currently, webpage-based enterprise attribution identification in asset recognition primarily relies on web scraping or API interfaces to obtain the target webpage's HTML source code. This HTML source code is then parsed to extract the webpage's title and body content. Keyword and regular expression matching, combined with keyword and regular expression libraries, is then used to determine the webpage's enterprise attribution. While this method is simple and easy to implement, it can be affected by interfering information within the webpage content, leading to inaccurate results. Furthermore, keywords and regular expressions can only match predefined patterns and may fail to recognize unforeseen patterns. As webpage design and content change, keyword and regular expression libraries require continuous updates and maintenance to adapt to new patterns and avoid becoming obsolete. This necessitates ongoing human and resource investment. Due to the diversity and complexity of webpage content, keywords and regular expressions may produce false positives (mistaking non-target content for target content) or false negatives (failing to identify actual target content). For example, similar words or phrases may have different meanings in different contexts, and static matching struggles to distinguish these contextual differences.

[0073] Therefore, this embodiment provides a method for identifying enterprise ownership of web page assets based on the BERT language model, as shown in the figure. The method includes the following steps:

[0074] S1: Obtain the URL of the webpage to be identified, obtain the HTML source code of the webpage assets through web crawling technology, and parse and extract the title and body content of the webpage;

[0075] S2: Perform text processing on the title and the body content to obtain the text to be recognized;

[0076] S3: Input the text to be identified into the trained BERT language model, and output the enterprise affiliation identification result of the webpage through semantic understanding and feature interaction calculation;

[0077] S4: Manually verify the enterprise affiliation identification results, and update the BERT language model with the identified abnormal text as new samples to complete the model iterative optimization.

[0078] This embodiment trains and learns from web page data to construct a BERT language model, enabling automatic identification of web page asset ownership. It can be widely applied to internet exposure surface asset management platforms, internet space mapping platforms, and attack surface management platforms.

[0079] In one embodiment, the text processing specifically includes:

[0080] S21: Extract text that conforms to Chinese language rules from the body content;

[0081] S22: If the text is too large, that is, if the length of the extracted text exceeds 100,000 Chinese characters, then the first 50,000 Chinese characters and the last 50,000 Chinese characters are truncated and concatenated to form the text to be identified.

[0082] In one implementation, such as Figure 2 As shown, the training steps of the BERT language model include:

[0083] S31: Obtain keywords from the enterprise database, manually construct positive template data and negative template data, and assemble them to form a sample dataset, which includes content and the corresponding enterprise tags;

[0084] S32: Add a positive and negative sample label column to the sample dataset, and divide the sample dataset into a training set, a test set, and a validation set;

[0085] S33: The model is trained based on the training set, the test set, and the validation set as inputs to the BERT Chinese pre-trained model to generate a BERT language model for enterprise affiliation identification, wherein the training includes model training, model evaluation, model testing, and model tuning.

[0086] The BERT language model takes a text tensor of dimension [batch, context_len] as input and outputs candidate scores of dimension [batch, cand_num]. After normalization by the Softmax function, the company corresponding to the highest score is determined.

[0087] Specifically, the input to the BERT Chinese pre-trained model is used to train the model, including:

[0088] S331: Vectorize the input text content into a text tensor, and vectorize the candidate companies into candidate company vectors;

[0089] S332: The interaction between the operator tensor and the text tensor is calculated through an attention mechanism to obtain the feature tensor;

[0090] S333: Aggregate the candidate enterprise vectors to generate an aggregated candidate enterprise vector;

[0091] S334: Calculate the interaction between the candidate enterprise aggregation vector and the feature tensor through an attention mechanism to obtain the text interaction result;

[0092] S335: Perform dot product calculation and feature summation on the candidate enterprise aggregation vector and the text interaction result to obtain the candidate score;

[0093] S336: Calculate the loss function based on the candidate scores to obtain the current batch loss. Input the candidate scores into the model batch by batch to calculate the candidate scores. Take the company with the highest candidate score as the company predicted by the model and compare it with the actual company marked. Statistically calculate TP (True Instance), TN (True Negative Instance), FP (False Positive Instance), and FN (False Negative Instance) to maximize the True Instance TP and True Negative Instance TN indices of the predicted company and the marked company, and minimize the False Positive Instance FP and False Negative Instance FN indices.

[0094] In one implementation, the manual verification specifically includes:

[0095] S41: Mark the erroneous text in the recognition results and use it as the text of the recognition anomaly.

[0096] Furthermore, the aforementioned model iterative optimization also includes:

[0097] S42: Collect the labeled normal text and misidentified text, supplement the sample dataset of the BERT language model, and periodically trigger model retraining.

[0098] In one implementation, the computation process of the attention mechanism specifically includes:

[0099] The Query-Key / Value mechanism is used to perform feature interaction between the text tensor and the candidate enterprise vector, generating the feature tensor and the text interaction result.

[0100] It is feasible that the aforementioned Query-Key / Value mechanism specifically includes:

[0101] S3301: Denote the tensor as T text The dimensions are [batch, context_len, dim]; the candidate enterprise vector is denoted as V. cand The dimensions are [cand_num, cand_len, dim];

[0102] S3302: Extract the Query vector Q from the text tensor. The expression for vector Q is:

[0103]

[0104] in, This is a weight vector dynamically generated based on the semantic importance of webpage content. Its generation is based on analyzing the contribution of different words or phrases on the webpage to the task of identifying enterprise affiliation. text [i] represents the element at the i-th position of the text tensor;

[0105] S3303: Extract the Key vector K from the text tensor. The expression for vector K is:

[0106]

[0107] Where, α j T is an adjustment coefficient determined based on the syntactic structure and semantic coherence of the webpage text. Its value ranges from 0 to 1, and it is obtained through syntactic analysis and semantic correlation calculation of the webpage text. text [j] represents the element at position j of the text tensor;

[0108] S3304: Extract the Value vector V from the candidate enterprise vector. The expression for vector V is:

[0109]

[0110] Among them, V cand [k] represents the element at the k-th position of the candidate enterprise vector;

[0111] S3304: Calculate the feature tensor, wherein the formula for calculating the feature tensor is:

[0112]

[0113] Where σ is the activation function, in the form of: γ is a parameter that is adaptively adjusted based on gradient changes during model training, and its value ranges from -1 to 1; K T K is the transpose of vector K, β is the balance coefficient, which ranges from 0 to 1 and is determined by statistical analysis of a large amount of web page data, combined with the characteristics of the enterprise affiliation identification task; K[m] represents the element of vector K in the m-th dimension.

[0114] S3305: Calculate the text interaction result, wherein the formula for calculating the text interaction result is:

[0115]

[0116] Where tanh is the hyperbolic tangent function, F[n] represents the element of the feature tensor at the nth position, and V cand [n] represents the element at the nth position of the candidate enterprise vector, and θ is a bias parameter that is dynamically adjusted based on the model training error.

[0117] It should be noted that the dimensions of the sample tensor and candidate company vectors are described below:

[0118] 1. batch

[0119] Definition: Batch size represents the number of samples processed in a single training or computation process. In deep learning model training, multiple samples are typically combined into a batch for computation, rather than being processed one sample at a time. This leverages the advantages of parallel computing, improving computational efficiency and optimizing model training performance. In this method, the text tensor T... text The first dimension, batch, means processing a batch of web page text data at the same time.

[0120] How it's obtained: It's determined by the hyperparameter settings during model training. In practical applications, the appropriate batch size needs to be selected based on hardware resources (such as GPU memory), model complexity, and data size. Setting the batch size too large may lead to insufficient memory; setting it too small may affect the stability and efficiency of model training. Generally, the optimal batch size is determined through empirical testing and performance evaluation. For example, in initial testing, different batch sizes (such as 16, 32, 64, etc.) can be tried to observe the convergence speed and accuracy of model training, and the batch size that best optimizes model performance can be selected.

[0121] 2. context_len

[0122] Definition: The length of a text sequence in the text tensor T. text In this context, it represents the length of each webpage text after it has been processed into a vector sequence. That is, after vectorization and processing, each webpage text is represented as a sequence with `context_len` elements, where each element can be a word vector, phrase vector, or other form of text feature vector. This length is used to limit the scope of the context considered by the model when processing the text.

[0123] Methods for determining context_len: There are generally two methods. One is the fixed-length method, which selects a suitable fixed value as context_len based on statistical analysis of the length of a large number of web page texts. For example, if analysis shows that most web page texts, after reasonable truncation or padding, are between 500 and 1000 characters long, then 800 can be chosen as context_len. For texts shorter than context_len, padding (e.g., using zero vectors) is usually performed; for texts longer than context_len, truncation is performed. The other method is the dynamic-length method. In some advanced models or processing methods, the model's processing method can be dynamically adjusted according to the actual length of the text, but in subsequent calculations, some standardization operations are usually performed to adapt to the model's computational requirements.

[0124] 3. dim

[0125] Definition: Represents the feature dimension of a vector, whether it is a text tensor T text Or candidate firm vector V cand The last dimension, dim, represents the number of features contained in each vector element. For example, when dim = 128, it means that each word vector (or other text feature vector, candidate company vector element) is 128-dimensional, and each dimension carries specific feature information about the text or company. These features together describe the attributes of the text or company.

[0126] Acquisition Method: `dim` is an important hyperparameter in model design, and its value depends on the complexity of the task, the characteristics of the data, and the performance requirements of the model. In simple text classification tasks, a smaller `dim` value (e.g., 64) might be chosen to reduce computation and model complexity; while in complex natural language processing tasks, such as identifying the ownership of assets on this webpage, a larger value (e.g., 256 or 512) might be chosen to more comprehensively and accurately represent the features of the text and the company. The optimal value is usually determined by experimentally comparing the model's performance (e.g., accuracy, recall) under different `dim` values.

[0127] 4. cand_num

[0128] Definition: Represents the number of candidate firms in the candidate firm vector V. cand In the dimension [cand_num, cand_len, dim], cand_num determines how many candidate companies participate in the model's calculation and judgment. In the task of identifying the ownership of web assets, these candidate companies are the set of companies that may have an ownership relationship with web assets.

[0129] Acquisition Method: The value of `cand_num` is determined by the specific business scenario and data conditions. In practical applications, it can be determined based on factors such as the size of the existing enterprise database and the distribution of enterprises in the target webpage's industry. For example, if it's for identifying webpage assets in a specific industry, and it's known that there are 100 major companies in that industry, then `cand_num` can be set to 100 or adjusted according to the actual situation. Alternatively, you can gradually increase `cand_num` and observe the model's recognition performance. When the model's performance improvement is not significant, a reasonable `cand_num` value can be determined.

[0130] 5. cand_len

[0131] Definition: The sequence length of each candidate firm vector, in the candidate firm vector V. cand In this context, it represents the length of the vector sequence obtained after vectorizing each candidate company. `cand_len` is used to limit the length of the vector for each candidate company, enabling the model to uniformly process the vector representations of different candidate companies.

[0132] Acquisition Method: The method for determining `cand_len` is similar to that for `context_len`. A fixed-length approach can be used, uniformly processing the information of all candidate companies (e.g., truncating or padding) to fix the vector length to `cand_len`. For example, when vectorizing the name, description, and other information of each candidate company, if statistical analysis shows that the vectorized length of most company information is between 10 and 50, 30 can be chosen as `cand_len`. Alternatively, a dynamic-length approach can be used, but corresponding adjustments will be needed in subsequent calculations to adapt to the model's computational requirements.

[0133] Furthermore, S4 also includes: during each model iteration optimization, adjusting the weight vector, adjustment coefficients, balance coefficients, and bias parameters based on the recognition accuracy (Acc) and recall (Rec) of different webpage samples during training; wherein:

[0134] The adjustment of the weight vector is specifically as follows:

[0135]

[0136] ω is a reference coefficient. Δw i Δw is a fine-tuning amount determined based on the update frequency of words or phrases in the webpage content. For frequently updated words or phrases, Δw i A larger value indicates a smaller value; conversely, a smaller value indicates a smaller value. This is the weight vector for this training session;

[0137] The adjustment of the adjustment coefficient is specifically as follows:

[0138] a j(new) =α j(old) ·(1-ω·Δα j ), Δα j The adjustment amount is determined based on the degree of change in grammatical structure and semantic coherence; a larger degree of change results in an adjustment Δα. j The value is relatively large; α j(old) This is the adjustment factor for this training session;

[0139] The specific adjustments to the adaptive adjustment parameters are as follows:

[0140] γ (new) =γ (old) +ω·Δγ

[0141] Δγ is an adjustment value determined based on the direction and magnitude of the gradient change. When the gradient descent direction is stable and the magnitude is large, Δγ takes a positive value; otherwise, it takes a negative value. (old) These are the parameters that are adaptively adjusted during this training process;

[0142] The adjustment of the balance coefficient is specifically as follows:

[0143] β (new) =β (old) +ω·(β target -β (old) )

[0144] β target A target balance coefficient pre-set to achieve optimal recognition results for different types of web page data; β (old) This is the balance coefficient during this training session;

[0145] The adjustment of the bias parameter is specifically as follows:

[0146] θ (new) =θ (old) +ω·Δθ)

[0147] Δθ is an adjustment value determined based on the changing trend of the model training error. When the error is decreasing, Δθ is smaller, and vice versa.

[0148] This dynamic update and optimization further improves the model's accuracy in identifying the ownership of web page assets.

[0149] In summary, the webpage asset enterprise ownership identification method based on the BERT language model in this embodiment demonstrates significant technical advantages in terms of accuracy, adaptability, and efficiency. It effectively overcomes the shortcomings of traditional methods and improves the overall level of webpage asset enterprise ownership identification. Specifically:

[0150] 1. Significantly Improved Recognition Accuracy: Traditional methods rely on keyword and regular expression matching, which are easily affected by interference and struggle to handle complex semantics and unforeseen patterns, leading to inaccurate recognition results. This method leverages the powerful semantic understanding capabilities of the BERT language model to deeply analyze the semantics of web page text and accurately capture key information related to enterprise ownership. Through a Query-Key / Value mechanism and innovative feature interaction formulas, such as extracting specific vectors from text tensors and candidate enterprise vectors, and innovative formulas for calculating the interaction results between feature tensors and text, the model more accurately matches web page content with enterprise ownership, reducing false positives and false negatives, improving recognition accuracy and recall, and ensuring the reliability of online asset ownership identification.

[0151] 2. Significantly Enhanced Adaptability: Traditional methods require frequent updates to keyword libraries and regular expressions as webpage design and content constantly evolve, consuming significant manpower and resources. This method employs an incremental learning mechanism, manually verifying the identification results and using anomalous text as new samples to update the BERT language model. This allows the model to continuously learn new webpage content patterns and enterprise affiliation characteristics. During model training, the parameters in the formula are dynamically adjusted based on factors such as the accuracy and recall of webpage samples, as well as gradient changes, syntactic structure, and semantic coherence. α j The use of γ, β, and θ enables the model to automatically adapt to changes in webpage content, maintain good recognition performance, and reduce maintenance costs.

[0152] 3. Highly Efficient for Processing Complex Web Page Data: Traditional methods have limitations in processing speed and efficiency when faced with massive and complex web page data. This method improves data processing efficiency by vectorizing text content and candidate companies and using an attention mechanism for feature interaction calculation. The reasonable use of training, testing, and validation sets during model construction, along with batch optimization of model parameters, accelerates model training and convergence, enabling rapid processing of large amounts of web page data. This meets the real-time requirements of internet exposure surface asset management platforms, internet space mapping platforms, and attack surface management platforms, providing timely and accurate asset ownership information for network security management.

[0153] In the embodiments provided in this application, it should be understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, code, or any suitable combination thereof. For hardware implementation, the processor may be implemented in one or more of the following: application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic units designed to implement the functions described herein, or combinations thereof. For software implementation, some or all of the processes of the embodiments may be performed by a computer program instructing the associated hardware. During implementation, the program may be stored in a computer-readable storage medium or transmitted as one or more instructions or code on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. Storage media may be any available medium accessible to a computer. Computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code having the form of instructions or data structures and accessible to a computer.

[0154] Finally, it should be noted that the above description is only a preferred embodiment of this application and is not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for identifying enterprise ownership of web page assets based on the BERT language model, characterized in that, The method includes the following steps: S1: Obtain the HTML source code of web page assets through web crawling technology, and parse and extract the title and body content of the web page; S2: Perform text processing on the title and the body content to obtain the text to be recognized; S3: Input the text to be identified into the trained BERT language model, and output the enterprise affiliation identification result of the webpage through semantic understanding and feature interaction calculation; S4: Manually verify the enterprise affiliation identification results, and update the BERT language model with the identified abnormal text as new samples to complete the model iterative optimization.

2. The method for identifying enterprise ownership of web page assets based on the BERT language model according to claim 1, characterized in that, The text processing in S2 specifically includes: S21: Extract text that conforms to Chinese language rules from the body content; S22: If the length of the extracted text exceeds 100,000 Chinese characters, the first 50,000 Chinese characters and the last 50,000 Chinese characters are extracted and concatenated to form the text to be identified.

3. The method for identifying enterprise ownership of web page assets based on the BERT language model according to claim 1, characterized in that, The training steps for the BERT language model in S3 include: S31: Obtain keywords from the enterprise database, manually construct positive template data and negative template data, and assemble them to form a sample dataset, which includes content and the corresponding enterprise tags; S32: Add a positive and negative sample label column to the sample dataset, and divide the sample dataset into a training set, a test set, and a validation set; S33: The model is trained based on the training set, the test set, and the validation set as inputs to the BERT Chinese pre-trained model to generate a BERT language model for enterprise affiliation identification, wherein the training includes model training, model evaluation, model testing, and model tuning.

4. The method for identifying enterprise ownership of web page assets based on the BERT language model according to claim 3, characterized in that, The input to the BERT Chinese pre-trained model is used to train the model, specifically including: S331: Vectorize the input text content into a text tensor, and vectorize the candidate companies into candidate company vectors; S332: The interaction between the operator tensor and the text tensor is calculated through an attention mechanism to obtain the feature tensor; S333: Aggregate the candidate enterprise vectors to generate an aggregated candidate enterprise vector; S334: Calculate the interaction between the candidate enterprise aggregation vector and the feature tensor through an attention mechanism to obtain the text interaction result; S335: Perform dot product calculation and feature summation on the candidate enterprise aggregation vector and the text interaction result to obtain the candidate score; S336: Calculate the loss function based on the candidate scores, optimize the model parameters batch by batch, maximize the true positive (TP) and true negative (TN) indices of the predicted attribution enterprise and the labeled enterprise, and minimize the false positive (FP) and false negative (FN) indices.

5. The method for identifying enterprise ownership of web page assets based on the BERT language model according to claim 1, characterized in that, The manual verification in S4 specifically includes: S41: Mark the erroneous text in the recognition results and use it as the text of the recognition anomaly.

6. The method for identifying enterprise ownership of web page assets based on the BERT language model according to any one of claims 1-5, characterized in that, The model iterative optimization in S4 also includes: S42: Collect the labeled normal text and misidentified text, supplement the sample dataset of the BERT language model, and periodically trigger model retraining.

7. The method for identifying enterprise ownership of web page assets based on the BERT language model according to claim 4, characterized in that, The computational process of the attention mechanism specifically includes: The Query-Key / Value mechanism is used to perform feature interaction between the text tensor and the candidate enterprise vector, generating the feature tensor and the text interaction result.

8. The method for identifying enterprise ownership of web page assets based on the BERT language model according to claim 7, characterized in that, The aforementioned Query-Key / Value mechanism specifically includes: S3301: Denote the tensor as T text The dimensions are [batch, context_len, dim]; the candidate enterprise vector is denoted as V. cand The dimensions are [cand_num, cand_len, dim]; S3302: Extract the Query vector Q from the text tensor. The expression for vector Q is: in, T is a weight vector dynamically generated based on the semantic importance of webpage content. text [i] represents the element at the i-th position of the text tensor; S3303: Extract the Key vector K from the text tensor. The expression for vector K is: Where, α j T is an adjustment factor determined based on the syntactic structure and semantic coherence of the webpage text, with a value ranging from 0 to 1. text [j] represents the element at position j of the text tensor; S3304: Extract the Value vector V from the candidate enterprise vector. The expression for vector V is: Among them, V cand [k] represents the element at the k-th position of the candidate enterprise vector; S3304: Calculate the feature tensor, wherein the formula for calculating the feature tensor is: Where σ is the activation function; K T Let K be the transpose of vector K, β be the balance coefficient, and its value ranges from 0 to 1; K[m] represents the element of vector K in the m-th dimension; S3305: Calculate the text interaction result, wherein the formula for calculating the text interaction result is: Where tanh is the hyperbolic tangent function, F[n] represents the element of the feature tensor at the nth position, and V cand [n] represents the element at the nth position of the candidate enterprise vector, and θ is a bias parameter that is dynamically adjusted based on the model training error.

9. The method for identifying enterprise ownership of web page assets based on the BERT language model according to claim 8, characterized in that, S4 further includes: during each model iteration optimization, adjusting the weight vector, adjustment coefficients, balance coefficients, and bias parameters based on the recognition accuracy (Acc) and recall (Rec) of different webpage samples during training; wherein: The adjustment of the weight vector is specifically as follows: ω is a reference coefficient. Δw i The amount of fine-tuning is determined based on the update frequency of words or phrases in the webpage content; This is the weight vector for this training session; The adjustment of the adjustment coefficient is specifically as follows: a j(new) =α j(old) ·(1-ω·Δα j ), Δα j The adjustment amount is determined based on the degree of change in syntactic structure and semantic coherence; α j(old) This is the adjustment factor for this training session; The adjustment of the balance coefficient is specifically as follows: b (new) =b (old) +ω·(β target -b (old) ) β target A target balance coefficient pre-set to achieve optimal recognition results for different types of web page data; β (old) This is the balance coefficient during this training session; The adjustment of the bias parameter is specifically as follows: i (new) =θ (old) +ω·Δθ) Δθ is an adjustment value determined based on the changing trend of the model training error.

10. The method for identifying enterprise ownership of web page assets based on the BERT language model according to claim , characterized in that, The BERT language model takes a text tensor of dimension [batch, context_len] as input and outputs candidate scores of dimension [batch, cand_num]. After normalization by the Softmax function, the company corresponding to the highest score is determined.

Images