Artificial intelligence-based website map generation method, system, device and medium

Abstract

The present invention provides a method, system, device, and medium for generating a website map based on artificial intelligence. The method specifically includes: calculating similarity scores between the page contents of different web pages using a text similarity algorithm based on text feature vectors, determining several groups of near-duplicate web pages by comparing the similarity scores with a preset similarity score threshold; determining the most important near-duplicate web page in each near-duplicate web page group by evaluating the importance of each near-duplicate web page in each near-duplicate web page group; entering several web pages outside the near-duplicate web page group and the most important near-duplicate web page in each near-duplicate web page group into a website map, and performing a security risk assessment on each web page in the website map. The present invention solves problems such as website content duplication, web page value assessment, and website map optimization, effectively improving search engine indexing efficiency and user access experience.

Images