Website map generation method, system and equipment based on artificial intelligence and medium

Abstract

The invention provides a website map generation method, system and device based on artificial intelligence and a medium, and the method specifically comprises the following steps: calculating similarity scores among page contents of different webpages by adopting a text similarity algorithm based on a text feature vector, determining a plurality of approximate duplicate webpage groups by comparing the similarity score with a preset similarity score threshold; determining the approximate duplicate webpage with the highest importance in each approximate duplicate webpage group by evaluating the importance of each approximate duplicate webpage in each approximate duplicate webpage group; and inputting a plurality of webpages outside the approximate duplicate webpage group and the approximate duplicate webpage with the highest importance in each approximate duplicate webpage group into a website map, and performing security risk assessment on each webpage in the website map. According to the method, the problems of website content repetition, webpage value evaluation, website map optimization and the like are solved, and the search engine recording efficiency and the user access experience are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of website maps, and in particular to a website map generation method, system, device and medium based on artificial intelligence. Background Art

[0002] In today's Internet ecosystem, the prosperity of e-commerce websites (e-commerce platforms) has made sitemaps an indispensable tool for optimizing website structure and improving search engine rankings (SEO). Traditional sitemap generation methods, such as using XML sitemap generators or online services, greatly reduce the burden of manually writing sitemaps through automation, provide search engines with a clear overview of website structure and content, and effectively promote the indexing and ranking of website content.

[0003] However, in the specific field of e-commerce, the generation and optimization of sitemaps face unique challenges:

[0004] 1. Frequent changes in product information: Product information on e-commerce websites, such as product listing, delisting, price adjustment, etc., changes frequently and rapidly. This dynamic nature requires that the sitemap must be able to be updated in real time to ensure that search engines can crawl the latest page information. Traditional static or semi-dynamic sitemap generation methods are difficult to keep up with this rapid pace of change, which can easily lead to outdated page information indexed by search engines, thereby affecting the search ranking and exposure of products.

[0005] 2. Identification of massive pages and key pages: Large e-commerce websites have a huge number of pages, which may reach millions or even tens of millions. At such a scale, how to effectively identify pages that are critical to SEO (such as hot-selling product pages, high-conversion rate pages, etc.) and assign reasonable link weights in the site map has become an urgent problem to be solved. Traditional analysis algorithms are often inefficient when processing such large-scale data and cannot meet the needs of real-time generation and optimization of site maps.

[0006] 3. Incremental update and accuracy issues: Although the incremental update method can alleviate the pressure of full update to a certain extent, it is difficult to accurately capture the subtle changes of key pages on e-commerce websites by relying solely on incremental updates. For example, fine-tuning product descriptions and replacing pictures, although seemingly subtle, may have a significant impact on users' purchasing decisions and search engine evaluations. Summary of the invention

[0007] The object of the present invention is to provide a method, system, device and medium for generating a website map based on artificial intelligence to solve at least one of the above-mentioned problems in the prior art.

[0008] In a first aspect, the present invention provides a method for generating a website map based on artificial intelligence, the method specifically comprising:

[0009] Use web crawler technology to obtain the URL addresses and corresponding page contents of several web pages to form a web page content corpus;

[0010] Using natural language processing technology to extract and process the web page content corpus to obtain a text feature vector;

[0011] Based on the text feature vector, a text similarity algorithm is used to calculate similarity scores between page contents of different web pages, and a plurality of nearly duplicate web page groups are determined by comparing the similarity scores with a preset similarity score threshold, or, based on the text feature vector, a K-means clustering algorithm is used to perform cluster analysis on a plurality of web pages to determine a plurality of nearly duplicate web page groups;

[0012] Determine the most important near-duplicate web page in each near-duplicate web page group by evaluating the importance of each near-duplicate web page in each near-duplicate web page group;

[0013] Several web pages outside the group of nearly duplicate web pages and the most important nearly duplicate web page in each group of nearly duplicate web pages are entered into a website map, and a security risk assessment is performed on each web page in the website map.

[0014] In a second aspect, the present invention provides a website map generation system based on artificial intelligence, the system specifically comprising:

[0015] The first generation module is used to obtain the URL addresses and corresponding page contents of several web pages using web crawler technology to form a web page content corpus;

[0016] A second generation module is used to extract and process the web page content corpus using natural language processing technology to obtain a text feature vector;

[0017] A third generation module is used to calculate the similarity scores between the page contents of different web pages based on the text feature vector using a text similarity algorithm, and determine a number of nearly duplicate web page groups by comparing the similarity scores with a preset similarity score threshold, or to perform cluster analysis on a plurality of web pages based on the text feature vector using a K-means clustering algorithm to determine a number of nearly duplicate web page groups;

[0018] A fourth generating module, configured to determine the most important near-duplicate web page in each near-duplicate web page group by evaluating the importance of each near-duplicate web page in each near-duplicate web page group;

[0019] The fifth generating module is used to enter several web pages outside the near-duplicate web page group and the most important near-duplicate web page in each near-duplicate web page group into the website map, and perform security risk assessment on each web page in the website map.

[0020] In a third aspect, the present invention provides a computer device comprising: a memory and a processor and a computer program stored in the memory, and when the computer program is executed on the processor, it implements the artificial intelligence-based website map generation method as described in any one of the above methods.

[0021] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for generating a website map based on artificial intelligence as described in any one of the above methods is implemented.

[0022] Compared with the prior art, the present invention has at least one of the following technical effects:

[0023] 1. The present invention can effectively improve the quality and timeliness of website maps, optimize search engine inclusion and ranking of websites, and increase website visibility and traffic.

[0024] 2. The present invention solves the problems of website content duplication, web page value evaluation and website map optimization, and effectively improves the search engine indexing efficiency and user access experience.

[0025] 3. The present invention realizes intelligent deduplication and optimization of web page content through multi-dimensional analysis, provides an efficient web page management solution for website administrators, and helps to improve the overall quality of the website and search engine ranking.

[0026] 4. The present invention can not only automatically generate a website map that accurately reflects the latest status of the website, but also effectively identify and process nearly duplicate content, and provide personalized SEO optimization suggestions for the website, thereby significantly improving the website's search engine ranking and user experience, and meeting the high requirements for website optimization in the current Internet environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 It is a flowchart of a method for generating a website map based on artificial intelligence provided by an embodiment of the present invention;

[0029] Figure 2It is a structural diagram of a website map generation system based on artificial intelligence provided by an embodiment of the present invention;

[0030] Figure 3 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0032] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0033] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0034] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0035] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0036] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0037] In the embodiment of the present application, the execution subject of the process includes a terminal device, which includes but is not limited to: a server, a computer, a smart phone, a tablet computer, and other devices capable of executing the method disclosed in the present application. Figure 1 A schematic diagram of a process of generating a website map based on artificial intelligence according to an embodiment of the present invention is shown, and the details are as follows:

[0038] S101, using web crawler technology to obtain URL addresses and corresponding page contents of several web pages to form a web page content corpus.

[0039] In the present embodiment, in order to more accurately reflect the changes in website content, improve the crawling efficiency of search engines to website content, a corpus containing the URL addresses of each page of the website and the corresponding page content is constructed using web crawler technology. Specifically, first, the target website scope of the crawl is clearly defined, including each channel under the main site and subdomain, classification page and product details page, etc., and crawling rules are set, such as crawling depth, frequency, anti-crawler strategy response, etc., to ensure the validity and legitimacy of the crawling process. Then, select suitable programming language (such as Python) and crawler framework (such as Scrapy, BeautifulSoup, etc.), write crawler scripts, including request header settings, URL queue management, page parsing logic, etc., set crawler startup parameters, such as concurrent number, proxy IP pool, etc., to cope with performance bottlenecks and access restrictions during large-scale crawling.

[0040] Start the crawler program and start crawling web pages according to the set rules and goals. During the crawling process, the crawler will continuously take out URLs from the queue of URLs to be crawled, send HTTP requests to the server, and receive and parse the returned HTML page content. For each crawled page, the crawler will extract its URL address and page content (including text, pictures, links, etc.), and store this information in the local database or file system. Organize the crawled web page content in a certain format (such as JSON, CSV, etc.) to form a web page content corpus. The corpus contains each page's unique identifier (such as URL), page title, body content, publishing time, link relationship and other information. Preprocess the corpus, such as removing HTML tags, stop word filtering, text segmentation, etc., to facilitate subsequent natural language processing and data analysis.

[0041] In this embodiment, by using a timed or triggered crawling task, it is possible to ensure that the web page content corpus can reflect changes in website content in real time, providing the search engine with the latest and most accurate website information. The automated crawling process greatly reduces the burden of manually collecting website content and improves work efficiency and accuracy.

[0042] In some embodiments, in the above step S101, the use of web crawler technology to obtain the URL addresses and corresponding page contents of several web pages specifically includes:

[0043] Determine a web page URL set to be crawled according to business requirements, wherein the web page URL set includes URL addresses of several web pages;

[0044] Based on the web page URL set, a web crawler technology is used to obtain HTML source code contents of several web pages;

[0045] By analyzing the HTML source code content, the initial page content of each web page is extracted;

[0046] Dividing the initial page content into blocks, calculating the text density and label density of each block, and determining the blocks whose text density and label density are both higher than respective preset density thresholds as text blocks;

[0047] Merge consecutive text blocks to form the target page content.

[0048] In this embodiment, according to business needs, a web page URL address set to be captured is obtained, and for each URL link in the set, web crawler technology is used to capture and parse the web page content. By analyzing the DOM structure and HTML tags of the web page, key information in the web page, such as title, text, pictures, etc., is extracted, and the information is stored in a structured manner. During the capture process, the priority and frequency of the capture are dynamically adjusted according to the update frequency and importance of the web page to ensure the efficiency and quality of the capture. The captured web page content is deduplicated and filtered to remove irrelevant or repeated information to obtain the initial page content. The initial page content to be processed is obtained, and the page is divided into multiple independent content blocks according to the page layout and tag structure. For each content block, the ratio of the length of the text content contained therein to the area occupied is calculated to obtain the text density; the ratio of the number of HTML tags contained therein to the area occupied is calculated to obtain the tag density. The calculated text density and tag density of each block are compared with the preset text density threshold and tag density threshold. If the text density and tag density of a block exceed the corresponding threshold, the block is determined to be a candidate text block. Screen all candidate text blocks, remove non-text noise blocks with too small area and too short text content, and obtain a preliminary text block set. Analyze the text block set. If there are multiple adjacent text blocks, merge them into a larger text block to avoid the fragmentation of the text content. Perform text extraction on the merged text blocks, remove embedded tags and redundant blank characters, and obtain the text content in plain text. All extracted text block contents are spliced ​​in the order of the original page to form the final target page text content.

[0049] Exemplary, first determine the webpage URL set to be crawled according to business needs, for example, to crawl the details page of all products on a certain e-commerce platform, you can analyze the URL rules of the product list page, generate the URL addresses of all product details pages, and form a set containing thousands of URLs. Then use Python's Scrapy framework to crawl the webpage, access the pages one by one according to the addresses in the URL set, and obtain the HTML source code content of each webpage. Then use XPath or CSS selectors to parse the HTML source code to extract the initial content of the page, such as product name, price, specification parameters and other information. Then divide the initial content into different blocks according to HTML tags, and calculate its text density and label density for each block of content. The calculation formula of text density is: text length / (text length+label length), and the calculation formula of label density is: number of labels / total number of characters in the block. Blocks with text density greater than 7 and label density less than 1 are determined as text blocks. Finally, the text blocks that appear continuously are merged, and redundant text is removed to form the final target page content. The whole process is realized through data analysis and algorithms, and finally outputs structured and highly available product information data.

[0050] S102, extracting and processing the web page content corpus using natural language processing technology to obtain a text feature vector.

[0051] In this embodiment, after the corpus containing the webpage URL address and the corresponding page content is constructed, in order to conduct a deeper analysis and optimization, the text content in the corpus is extracted and processed using natural language processing technology (NLP) to obtain feature vectors that can represent the text content. These feature vectors will be used for subsequent tasks such as text classification, clustering, and search engine optimization (SEO).

[0052] Specifically, first, load the web content corpus, clean the text content in each record, including removing irrelevant information such as HTML tags, punctuation, numbers, stop words, etc., and perform text segmentation, that is, dividing the continuous word sequence into independent words or phrases. This step usually relies on specific segmentation tools or algorithms, such as jieba segmentation (for Chinese content). Then, count all unique terms that appear in the text and assign a unique index to each term. Generate a vector for each text, each element in the vector corresponds to the index of a term, and the value is usually the number of times the term appears in the text (term frequency) or the value after TF-IDF weighting. Use a pre-trained word embedding model (such as Word2Vec, GloVe, BERT, etc.) to convert each word in the text into a vector of fixed dimension. Then, the vectors of all words in the text can be converted into feature vectors of the entire text by averaging, summing, or more complex aggregation methods (such as using LSTM, BERT's [CLS] tag output, etc.). Generate a corresponding feature vector for each text record in the corpus based on the selected feature extraction method. The quality of the generated feature vector is evaluated to check whether it can effectively represent the text content. According to the evaluation results, the parameters of preprocessing and feature extraction are adjusted to optimize the quality of the feature vector.

[0053] In this embodiment, the text feature vector extracted by natural language processing technology can more accurately represent the semantic information of the text content, and can capture the subtle differences and complex relationships between texts compared to traditional methods such as bag-of-words models. The automated natural language processing process can significantly improve the processing efficiency of text content, reduce the need for manual intervention, and ensure the accuracy and consistency of the processing results.

[0054] In some embodiments, in the above step S102, the extracting and processing the webpage content corpus using natural language processing technology to obtain a text feature vector specifically includes:

[0055] Using a word segmentation tool to perform word segmentation processing on the target page content, and converting the target page content into a word sequence;

[0056] Based on a preset part-of-speech tagging model, a part-of-speech tagging tool is used to perform part-of-speech tagging on the word sequence to obtain part-of-speech information of each word in the word sequence;

[0057] The word sequence after part-of-speech tagging is input into the pre-trained bidirectional LSTM network model for encoding, and the attention mechanism is used to perform weighted summation on the hidden state of each time step to obtain the preliminary text feature vector;

[0058] According to the preset web page classification system, the preliminary text feature vector is multi-classified using a Softmax classifier to obtain a target text feature vector.

[0059] In this embodiment, according to the preset part-of-speech tagging rules, the input word sequence is tagged with parts of speech to obtain the word sequence after part-of-speech tagging; the word sequence after part-of-speech tagging is input into the pre-trained bidirectional LSTM network model, and the word sequence is encoded by the LSTM network in the forward and backward directions to obtain the hidden state sequence at each time step; the attention mechanism is used to perform weighted summation on the hidden state sequence of each time step output by the bidirectional LSTM network to obtain a preliminary text feature vector that integrates context information; according to the pre-constructed web page classification system, the preliminary text feature vector is input into the Softmax classifier for multi-classification processing, and the feature vector is calculated by The probability of belonging to each category is measured, and the category with the largest probability is obtained as the final classification result; the category label corresponding to the classification result is determined, and if the category label is the target category, the preliminary text feature vector is determined as the target text feature vector; if the category label is not the target category, the current feature vector is discarded and the next word sequence is returned to be reprocessed; the similarity is calculated based on the target text feature vector and the feature vectors of each category in the preset web page classification system, and the Top-N categories most similar to the target text feature vector are obtained as candidate classification results; the candidate classification results and their corresponding confidence levels are output, and the candidate classification results are sorted according to the confidence levels to obtain the final web page classification results.

[0060] Exemplarily, the annotated word sequence is first input into the pre-trained bidirectional LSTM network model for encoding. The hidden layer dimension of the model is set to 128, the maximum length of the sequence is set to 100, and the attention mechanism is used to perform weighted summation of the hidden state of each time step. The attention distribution probability matrix is ​​multiplied and added with the hidden state matrix to obtain a preliminary text feature vector with a dimension of 128. Finally, according to the preset web page classification system, including 12 categories such as news, sports, entertainment, technology, and finance, the Softmax classifier is used to perform multi-classification on the above preliminary text feature vector. By setting the temperature parameter to 8, the normalized probability of each category is calculated 10 times through iteration, and the category with the maximum probability is taken as the final classification result. At the same time, the classification result is used as the weight to multiply and add with the preliminary text feature vector to obtain the target text feature vector that integrates the classification information, and the feature dimension is 128. The whole process realizes the automatic extraction and semantic understanding of text features through deep learning algorithms, and finally obtains accurate and comprehensive web page text representation.

[0061] S103, based on the text feature vector, using a text similarity algorithm to calculate similarity scores between page contents of different web pages, and determining a number of approximately duplicate web page groups by comparing the similarity scores with a preset similarity score threshold, or, based on the text feature vector, using a K-means clustering algorithm to perform cluster analysis on multiple web pages to determine a number of approximately duplicate web page groups.

[0062] In this embodiment, when calculating the similarity of web page content based on the text similarity algorithm and determining the group of approximately duplicate web pages, first select a suitable text similarity algorithm, such as cosine similarity, Jaccard similarity, Euclidean distance (note that it needs to be converted into a similarity score) or a similarity model based on deep learning (such as Siamese network), and then traverse the web page pairs in the corpus, and use the selected similarity algorithm to calculate the similarity score between the text feature vectors of each pair of web pages. According to actual needs, a similarity score threshold is set. The threshold is used to determine whether the content of two web pages is similar enough to be considered as approximately duplicated. For web page pairs whose similarity scores exceed the preset threshold, they are grouped into a group of approximately duplicate web pages. Pay attention to handling possible transitive relationships, that is, if A is similar to B and B is similar to C, then A, B, and C should be grouped together. Finally, all determined approximately duplicate web page groups are output.

[0063] When performing cluster analysis on web pages and determining groups of nearly duplicate web pages based on the K-means clustering algorithm, first, select a suitable number of cluster centers K based on prior knowledge or by trying different K values. K represents the number of nearly duplicate web page groups expected to be found. Then use the K-means algorithm to perform cluster analysis on text feature vectors. The algorithm iteratively assigns web pages to one of the K cluster centers until the convergence condition is met. The set of web pages around each cluster center can be regarded as a group of nearly duplicate web pages. The web pages within a cluster have high similarity in content. Evaluate the quality of the clustering results, such as by calculating indicators such as the average similarity or silhouette coefficient within the cluster. Adjust the K value or clustering algorithm parameters based on the evaluation results to obtain better clustering effects.

[0064] In this embodiment, by calculating the similarity between text feature vectors, web pages with highly similar content can be accurately identified, helping website managers to clean up duplicate content and improve user experience and SEO effects. The K-means clustering algorithm can automatically discover the potential structure in web page content and group similar web pages together without the need for a preset threshold.

[0065] In some embodiments, in the above step S103, the similarity scores between the page contents of different web pages are calculated based on the text feature vector using a text similarity algorithm, and a plurality of nearly duplicate web page groups are determined by comparing the similarity scores with a preset similarity score threshold, specifically including:

[0066] Obtain text feature vectors of any two web pages, and use the cosine similarity algorithm to calculate the similarity score between the text feature vectors of every two web pages;

[0067] Comparing the similarity score with a preset similarity score threshold, and if the similarity score is greater than or equal to the preset similarity score threshold, determining that the two web pages are nearly duplicate web pages;

[0068] The DBSCAN clustering algorithm is used to cluster every two web pages in several groups that are determined to be nearly duplicate web pages, so as to obtain several nearly duplicate web page groups, and the web pages in each nearly duplicate web page group are in a nearly duplicate relationship with each other.

[0069] In this embodiment, the webpage text content of at least two target webpages is obtained; the webpage text content is preprocessed to remove noise data and extract keyword features; a webpage text feature vector is constructed based on the extracted keyword features; a cosine similarity algorithm is used to calculate the similarity score between the two webpage text feature vectors; the calculated similarity score is compared with a preset similarity threshold; if the similarity score is greater than or equal to the preset threshold, the two webpages are determined to be approximately duplicate webpages; if the similarity score is less than the preset threshold, the two webpages are determined not to be approximately duplicate webpages. A similarity matrix is ​​constructed based on the similarity between each two webpages. The parameters of the DBSCAN algorithm are set, including the neighborhood radius and the minimum number of included points, for determining the core object. Each webpage data point in the similarity matrix is ​​traversed, and if it is a core object, the cluster to which it belongs is determined, and the points that can be reached by density are added to the cluster. For noise points that are not assigned to a cluster, they are assigned to the nearest cluster based on their connectivity with points within the cluster. The above steps are iterated until all data points are assigned to clusters or marked as noise. Output clustering results. Each cluster is a group of nearly duplicate web pages, and the web pages in the group are nearly duplicated with each other. Through web page clustering, duplicate web pages can be effectively discovered and classified, providing a basis for subsequent web page deduplication.

[0070] Exemplarily, first obtain the text feature vectors of any two web pages, for example, the feature vector of web page A is [2,1,5,3,6], and the feature vector of web page B is [3,2,4,5,7], and then use the cosine similarity algorithm to calculate the similarity score between the two feature vectors. Specifically, the corresponding elements of the two feature vectors are multiplied and summed, and then divided by the modulus of the two vectors. The result is the similarity score, which ranges from 0 to 1. Assuming that the similarity score of web pages A and B is 85, it is compared with the preset similarity threshold of 8. Since 85 is greater than 8, web pages A and B are determined to be nearly duplicate web pages. Then, the DBSCAN clustering algorithm is used to cluster the web pages determined to be nearly duplicated in several groups. The algorithm is based on density and can identify clusters of any shape. First, the similarity score of each web page with other web pages is calculated, and the neighborhood radius parameter ε is set to 2, and the minimum number of included points minPts is set to 3, that is, the premise of taking a web page as the core object is that its ε-neighborhood contains at least minPts web pages. Then, starting from any unvisited web page P, find all density-reachable web pages, that is, all core objects in the ε-neighborhood of P, recorded as set Q. For each web page in Q, recursively find its density-reachable web pages, and continue to expand Q until no new density-reachable web pages are added. At this time, Q is a near-duplicate web page group. Repeat the above process for all web pages, and eventually you can get several near-duplicate web page groups, where the web pages in the group are near-duplicates with each other. This density-based clustering method can effectively identify near-duplicate web pages in the web page library, providing a basis for subsequent web page deduplication.

[0071] In some embodiments, in the above step S103, clustering analysis is performed on multiple web pages using a K-means clustering algorithm based on the text feature vector to determine a number of nearly duplicate web page groups, specifically including:

[0072] Based on the text feature vector, a K-means clustering algorithm is used to cluster multiple web pages, and an optimal number of clusters is determined by the elbow rule;

[0073] The number of web pages in each cluster is counted to determine whether the number of web pages in each cluster exceeds a preset threshold value of the number of web pages. If so, the cluster is determined as a group of approximately duplicate web pages.

[0074] In this embodiment, the K-means clustering algorithm is used to cluster web pages according to the text feature vector, and the optimal number of cluster clusters is determined by calculating the clustering loss under different K values ​​using the elbow rule. For each generated cluster, the number of web pages in the cluster is counted to determine whether the number of web pages exceeds the preset threshold value of the number of web pages. If the number of web pages in a cluster exceeds the preset threshold, the cluster is determined as a near-duplicate web page group, indicating that the web page content in the group is highly similar. For all determined near-duplicate web page groups, the keyword features of each group are extracted, the similarity between the web page groups is calculated by cosine similarity, and a directed graph of web page duplication is constructed to facilitate further deduplication and screening of high-quality web pages.

[0075] For example, assume that there are 1000 web pages, and the dimension of the feature vector of each web page is 200. Then use the K-means clustering algorithm to cluster these 1000 feature vectors, and determine the optimal number of clusters by the elbow rule. Specifically, set the number of clusters K from 1 to 10 in sequence, and calculate the sum of square errors SSE for each clustering. When K=1, the maximum SSE is 120; when K=2, SSE is 85; when K=3, SSE is 60; when K=4, SSE is 45; when K=5, SSE is 43; when K=6, SSE is 42. It can be seen that when K is greater than 4, the decrease in SSE becomes smaller, so it is judged that the optimal number of clusters is 4. Then count the number of web pages in each cluster, assuming that the four clusters contain 300, 280, 250, and 170 web pages respectively. The preset threshold of the number of web pages is set to 200. Since the number of web pages in the first three clusters exceeds the threshold, these three clusters are determined to be near-duplicate web page groups, and the fourth cluster is excluded. This clustering-based method can quickly identify near-duplicate web page groups in the web page library, providing a basis for subsequent web page deduplication.

[0076] S104, determining the most important near-duplicate web page in each near-duplicate web page group by evaluating the importance of each near-duplicate web page in each near-duplicate web page group.

[0077] In this embodiment, after determining several groups of nearly duplicate web pages, in order to optimize the SEO, content management and user experience of the website, it is necessary to select a "representative" or "most important" web page from each group. Specifically, the nearly duplicate web pages in each nearly duplicate web page group can be evaluated according to importance indicators such as content quality, user interaction data, link structure, update time and originality, and weights can be assigned to different importance indicators according to business needs and goals. For example, for scenarios that emphasize content quality, a higher weight can be given to content quality. Finally, a weighted sum algorithm is used to calculate the comprehensive score of each web page based on the indicator values ​​and corresponding weights of each web page. In each nearly duplicate web page group, the most important web page is determined based on the comprehensive score. The selected most important web page is marked or optimized, such as improving its ranking in search results, increasing internal link pointing, etc. For other duplicate web pages, redirection, content merging or deletion can be considered to avoid content redundancy and SEO problems.

[0078] In this embodiment, by selecting the most important web page in each group of nearly duplicate web pages as a representative, users can find high-quality and valuable content more quickly, improving user experience. By processing duplicate web pages, content redundancy and noise on the website are reduced, making the website structure clearer and more concise.

[0079] In some embodiments, in the above step S104, determining the most important nearly duplicate web page in each nearly duplicate web page group by evaluating the importance of each nearly duplicate web page in each nearly duplicate web page group specifically includes:

[0080] According to each group of nearly duplicate web pages, a directed graph of web page links is constructed respectively, wherein the nodes in the directed graph of web page links represent web pages, and the directed edges represent the link relationship between web pages;

[0081] Based on the web page link directed graph, the PageRank algorithm is used to calculate the importance weight score of each web page node;

[0082] According to the importance weight score, determining the near-duplicate web page with the highest importance weight score in each near-duplicate web page group as the representative web page of the group;

[0083] If there are multiple near-duplicate web pages with the highest importance weight scores in any near-duplicate web page group, the multiple near-duplicate web pages with the highest importance weight scores are evaluated according to the multi-dimensional web page quality evaluation index to determine the representative web page of the group.

[0084] In this embodiment, based on the group of nearly duplicate web pages, the URL links of all web pages in the group are obtained to construct a directed graph of web page links. For the directed graph of web page links, the web page URLs are extracted as nodes, and the link relationships between web pages are taken as directed edges to form a set of nodes and edges. The PageRank algorithm is used to iteratively calculate the importance weight of each node based on the nodes and edges of the directed graph. By setting a threshold for the number of iterations, it is determined whether the current number of iterations reaches the threshold. If it reaches the threshold, the iteration is stopped, otherwise the iterative calculation continues. The final PageRank value of each node is obtained as the importance weight score of the web page represented by the node. The importance weight ranking result of the web page nodes is obtained by sorting from high to low according to the PageRank value. The web page node with the highest score in the sorting result is determined as the representative web page of the nearly duplicate web page group. For each web page group, obtain the importance weight score of each web page in the group; compare the importance weight scores of the web pages in each web page group, obtain the web page with the highest score, and determine it as the candidate representative web page of the group; determine whether the candidate representative web page is unique, and if the candidate representative web page is unique, determine it as the representative web page of the group; if the candidate representative web page is not unique, that is, there are multiple web pages with the highest scores, obtain the preset multi-dimensional web page quality evaluation index; according to the multi-dimensional web page quality evaluation index, such as the update frequency of the web page, the user visits and the content quality, score the multiple candidate representative web pages with the highest scores, and obtain the quality evaluation score of each candidate web page; compare the quality evaluation scores of the candidate web pages, and obtain the candidate web page with the highest quality evaluation score; determine the candidate web page with the highest quality evaluation score as the representative web page of the nearly duplicate web page group.

[0085] S105, entering a number of web pages outside the group of nearly duplicate web pages and the most important nearly duplicate web page in each group of nearly duplicate web pages into a website map, and performing a security risk assessment on each web page in the website map.

[0086] In this embodiment, several web pages outside the group of nearly duplicate web pages (i.e., non-duplicate, unique web pages) and the most important nearly duplicate web pages in each group of nearly duplicate web pages are entered into the website map. According to the logical structure and hierarchical relationship of the website, the order of web pages and link relationships in the website map are reasonably arranged to ensure that search engines and users can browse and access smoothly. Then, appropriate security risk assessment indicators are selected, such as the risk of sensitive information leakage, service interruption risk, malicious attack risk, compliance risk, etc. Relevant data is collected from channels such as website background, log files, and security scanning tools for subsequent risk assessment. According to the collected data and evaluation indicators, a security risk assessment is performed on each web page, and the web pages are risk graded according to the evaluation results. In response to the potential safety hazards and risk points found in the assessment, corresponding response measures and emergency plans are formulated, such as data encryption, access control, vulnerability repair, etc.

[0087] In this embodiment, by constructing an accurate and comprehensive website map, the structure and hierarchical relationship of the website are optimized, the indexing efficiency of the search engine and the browsing experience of users are improved. By conducting a security risk assessment on each web page in the website map, potential security risks and risk points are discovered and responded to in a timely manner, thereby enhancing the security and stability of the website.

[0088] In some embodiments, in the above step S105, the security risk assessment of each web page in the website map specifically includes:

[0089] Calculate and obtain a risk score for each web page in the website map based on risk assessment indicators, wherein the risk assessment indicators include content sensitivity, access rights, and exposure scope;

[0090] Using a support vector machine algorithm to classify the risk of each web page in the website map to determine the risk type of each web page in the website map;

[0091] Generate corresponding security optimization suggestions for each web page in the website map according to the risk score and the risk type.

[0092] In this embodiment, all web page URLs in the website map are obtained, and features such as web page content, access rights, and exposure range are extracted. According to the preset content sensitivity vocabulary, the content sensitivity score of each web page is determined. According to the access rights setting of the web page, the access rights risk score is determined. The lower the authority, the higher the score. The exposure range score is calculated by the number of external links and access traffic of the web page. The greater the exposure, the higher the score. The scores of the three dimensions of content sensitivity, access rights, and exposure range are weighted and summed to obtain the comprehensive risk score of each web page. A clustering algorithm is used to group web pages according to the comprehensive risk score to identify high-risk web pages. Risk visualization is performed on the website map, and the risk level of each web page is displayed in the form of a heat map to facilitate the identification and disposal of risks. For each web page in the web page set, web page content features are extracted, including keywords, links, codes, etc., to construct a feature vector. According to the labeled web page risk type training set, the risk classification model is trained using the support vector machine algorithm. The web page feature vector is input into the trained support vector machine classification model to predict the risk type of each web page. According to the risk score and risk type, the corresponding optimization suggestions are matched from the preset security optimization suggestion library; if the risk score of the current web page exceeds the preset threshold, the web page is marked as a high-risk web page, and its URL address and optimization suggestions are recorded; if all web pages are traversed, the URL address and optimization suggestions of the high-risk web pages are used to generate a security optimization report; according to the security optimization report, the high-risk web pages in the website are reinforced and optimized to reduce the overall security risk of the website.

[0093] Reference Figure 2 An embodiment of the present invention provides a website map generation system based on artificial intelligence, and the system specifically includes:

[0094] The first generation module is used to obtain the URL addresses and corresponding page contents of several web pages using web crawler technology to form a web page content corpus;

[0095] A second generation module is used to extract and process the web page content corpus using natural language processing technology to obtain a text feature vector;

[0096] A third generation module is used to calculate the similarity scores between the page contents of different web pages based on the text feature vector using a text similarity algorithm, and determine a number of nearly duplicate web page groups by comparing the similarity scores with a preset similarity score threshold, or to perform cluster analysis on a plurality of web pages based on the text feature vector using a K-means clustering algorithm to determine a number of nearly duplicate web page groups;

[0097] A fourth generating module, configured to determine the most important near-duplicate web page in each near-duplicate web page group by evaluating the importance of each near-duplicate web page in each near-duplicate web page group;

[0098] The fifth generating module is used to enter several web pages outside the near-duplicate web page group and the most important near-duplicate web page in each near-duplicate web page group into the website map, and perform security risk assessment on each web page in the website map.

[0099] It is understandable that if Figure 1 The contents of the embodiment of the website map generation method based on artificial intelligence shown in the figure are applicable to the embodiment of the website map generation system based on artificial intelligence. The functions specifically implemented by the embodiment of the website map generation system based on artificial intelligence are similar to those in the embodiment of the figure. Figure 1 The embodiment of the method for generating a website map based on artificial intelligence is the same as that shown in the embodiment, and the beneficial effects achieved are the same as those of the embodiment shown in the embodiment. Figure 1 The beneficial effects achieved by the embodiment of the website map generation method based on artificial intelligence are also the same.

[0100] It should be noted that the information interaction, execution process and other contents between the above-mentioned systems are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0101] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0102] Reference Figure 3 The embodiment of the present invention further provides a computer device 3, comprising: a memory 302 and a processor 301 and a computer program 303 stored in the memory 302. When the computer program 303 is executed on the processor 301, the website map generation method based on artificial intelligence as described in any one of the above methods is implemented.

[0103] The computer device 3 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will appreciate that Figure 3 It is only an example of computer device 3 and does not constitute a limitation on computer device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.

[0104] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0105] In some embodiments, the memory 302 may be an internal storage unit of the computer device 3, such as a hard disk or memory of the computer device 3. In other embodiments, the memory 302 may also be an external storage device of the computer device 3, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 3. Further, the memory 302 may also include both an internal storage unit of the computer device 3 and an external storage device. The memory 302 is used to store an operating system, an application program, a boot loader (Boot Loader), data, and other programs, such as the program code of the computer program, etc. The memory 302 may also be used to temporarily store data that has been output or is to be output.

[0106] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for generating a website map based on artificial intelligence as described in any one of the above methods is implemented.

[0107] In this embodiment, if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0108] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0109] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0110] In the embodiments disclosed in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0111] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Claims

1. A method for generating a website map based on artificial intelligence, characterized in that: The method specifically comprises: Use web crawler technology to obtain the URL addresses and corresponding page contents of several web pages to form a web page content corpus; Using natural language processing technology to extract and process the web page content corpus to obtain a text feature vector; Based on the text feature vector, a text similarity algorithm is used to calculate similarity scores between page contents of different web pages, and a plurality of nearly duplicate web page groups are determined by comparing the similarity scores with a preset similarity score threshold, or, based on the text feature vector, a K-means clustering algorithm is used to perform cluster analysis on a plurality of web pages to determine a plurality of nearly duplicate web page groups; Determine the most important near-duplicate web page in each near-duplicate web page group by evaluating the importance of each near-duplicate web page in each near-duplicate web page group; Several web pages outside the group of nearly duplicate web pages and the most important nearly duplicate web page in each group of nearly duplicate web pages are entered into a website map, and a security risk assessment is performed on each web page in the website map.

2. The method according to claim 1, characterized in that The method of using web crawler technology to obtain the URL addresses and corresponding page contents of several web pages specifically includes: Determine a web page URL set to be crawled according to business requirements, wherein the web page URL set includes URL addresses of several web pages; Based on the web page URL set, a web crawler technology is used to obtain HTML source code contents of several web pages; By analyzing the HTML source code content, the initial page content of each web page is extracted; Dividing the initial page content into blocks, calculating the text density and label density of each block, and determining the blocks whose text density and label density are both higher than respective preset density thresholds as text blocks; Merge consecutive text blocks to form the target page content.

3. The method according to claim 2, characterized in that The adopting of natural language processing technology to extract and process the webpage content corpus to obtain a text feature vector specifically includes: Using a word segmentation tool to perform word segmentation processing on the target page content, and converting the target page content into a word sequence; Based on a preset part-of-speech tagging model, a part-of-speech tagging tool is used to perform part-of-speech tagging on the word sequence to obtain part-of-speech information of each word in the word sequence; The word sequence after part-of-speech tagging is input into the pre-trained bidirectional LSTM network model for encoding, and the attention mechanism is used to perform weighted summation on the hidden state of each time step to obtain the preliminary text feature vector; According to the preset web page classification system, the preliminary text feature vector is multi-classified using a Softmax classifier to obtain a target text feature vector.

4. The method according to claim 1, characterized in that: The method of calculating similarity scores between page contents of different web pages based on the text feature vector using a text similarity algorithm and determining a plurality of nearly duplicate web page groups by comparing the similarity scores with a preset similarity score threshold specifically includes: Obtain text feature vectors of any two web pages, and use the cosine similarity algorithm to calculate the similarity score between the text feature vectors of every two web pages; Comparing the similarity score with a preset similarity score threshold, and if the similarity score is greater than or equal to the preset similarity score threshold, determining that the two web pages are nearly duplicate web pages; The DBSCAN clustering algorithm is used to cluster every two web pages in several groups that are determined to be nearly duplicate web pages, so as to obtain several nearly duplicate web page groups, and the web pages in each nearly duplicate web page group are in a nearly duplicate relationship with each other.

5. The method according to claim 1, characterized in that The method of clustering multiple web pages based on the text feature vector using a K-means clustering algorithm to determine a number of nearly duplicate web page groups specifically includes: Based on the text feature vector, a K-means clustering algorithm is used to cluster multiple web pages, and an optimal number of clusters is determined by the elbow rule; The number of web pages in each cluster is counted to determine whether the number of web pages in each cluster exceeds a preset threshold value of the number of web pages. If so, the cluster is determined as a group of approximately duplicate web pages.

6. The method according to claim 1, characterized in that The step of evaluating the importance of each near-duplicate web page in each near-duplicate web page group to determine the most important near-duplicate web page in each near-duplicate web page group specifically includes: According to each group of nearly duplicate web pages, a directed graph of web page links is constructed respectively, wherein the nodes in the directed graph of web page links represent web pages, and the directed edges represent the link relationship between web pages; Based on the web page link directed graph, the PageRank algorithm is used to calculate the importance weight score of each web page node; According to the importance weight score, determining the near-duplicate web page with the highest importance weight score in each near-duplicate web page group as the representative web page of the group; If there are multiple near-duplicate web pages with the highest importance weight scores in any near-duplicate web page group, the multiple near-duplicate web pages with the highest importance weight scores are evaluated according to the multi-dimensional web page quality evaluation index to determine the representative web page of the group.

7. The method according to claim 1, characterized in that The security risk assessment of each web page in the website map specifically includes: Calculate and obtain a risk score for each web page in the website map based on risk assessment indicators, wherein the risk assessment indicators include content sensitivity, access rights, and exposure scope; Using a support vector machine algorithm to classify the risk of each web page in the website map, and determine the risk type of each web page in the website map; Generate corresponding security optimization suggestions for each web page in the website map according to the risk score and the risk type.

8. The website map generation system based on artificial intelligence is characterized by: The system specifically comprises: The first generation module is used to obtain the URL addresses and corresponding page contents of several web pages using web crawler technology to form a web page content corpus; A second generation module is used to extract and process the web page content corpus using natural language processing technology to obtain a text feature vector; A third generation module is used to calculate the similarity scores between the page contents of different web pages based on the text feature vector using a text similarity algorithm, and determine a number of nearly duplicate web page groups by comparing the similarity scores with a preset similarity score threshold, or to perform cluster analysis on a plurality of web pages based on the text feature vector using a K-means clustering algorithm to determine a number of nearly duplicate web page groups; A fourth generating module, configured to determine the most important near-duplicate web page in each near-duplicate web page group by evaluating the importance of each near-duplicate web page in each near-duplicate web page group; The fifth generating module is used to enter several web pages outside the near-duplicate web page group and the most important near-duplicate web page in each near-duplicate web page group into the website map, and perform security risk assessment on each web page in the website map.

9. A computer device, characterized in that: include: A memory and a processor and a computer program stored in the memory, when the computer program is executed on the processor, implements the website map generation method based on artificial intelligence as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for generating a website map based on artificial intelligence as claimed in any one of claims 1 to 7 is implemented.

Images