A standardized dark chain sample construction and label annotation system and method thereof
By constructing standardized dark link samples and using a tagging system, the problem of low efficiency in traditional website security monitoring has been solved, achieving high efficiency and accuracy in website security monitoring and timely detection of security threats.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 陈昭奕
- Filing Date
- 2026-04-07
- Publication Date
- 2026-06-26
AI Technical Summary
Traditional website security technologies are unable to efficiently and accurately monitor website security, leading to frequent security incidents such as viruses, Trojans, and hacker intrusions, which seriously endanger national security and the interests of the people.
A standardized dark link sample construction and tagging system is adopted, including a task scheduling module, a memory management module, a DNS caching module, a web page asynchronous download module, and a content recognition and detection module. Through DNS caching, asynchronous download, and fast analysis algorithms, it achieves high efficiency and accuracy in website monitoring.
It speeds up DNS resolution, saves memory, improves HTML analysis speed and encoding recognition accuracy, and enables efficient and accurate website monitoring, allowing for timely detection of security threats.
Smart Images
Figure CN122293404A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer network security technology, specifically to a standardized dark chain sample construction and labeling system and method. Background Technology
[0002] With the rapid development of my country's national economy and social informatization, the internet has become an indispensable part of people's work and life. More and more government agencies, banks, enterprises, and other organizations are establishing their own websites to adapt to social development, build a positive image, expand their social influence, and improve work efficiency. However, because websites operate in the relatively open environment of the internet, the complexity and diversity of various web application systems lead to numerous system vulnerabilities. Viruses, Trojans, and malicious code run rampant online, and security incidents involving hacker intrusions and website tampering occur frequently. In some cases, website tampering has even escalated into political incidents, seriously endangering national security and the interests of the people.
[0003] Traditional website security protection technologies can no longer meet current security needs, and the ability to detect problems in advance is becoming increasingly important. Therefore, there is an urgent need for a method that can efficiently, accurately, and reasonably monitor website security. Summary of the Invention
[0004] To address the aforementioned technical problems, a standardized dark chain sample construction and labeling system and method are provided. This technical solution solves the problems mentioned in the background section.
[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A standardized dark link sample construction and labeling system includes a task scheduling module, a memory management module, a DNS caching module, a web page asynchronous download module, and a content recognition and detection module, characterized in that: The task scheduling module is used to schedule status detection and content detection tasks so that website monitoring can be completed within a specific period. The memory management module pre-allocates and serializes the memory required by all URL scanning tasks to facilitate rapid memory allocation and unified management. The DNS caching module caches the DNS address of the requested URL to improve its resolution speed; The asynchronous webpage download module enables fast downloading of webpages using various transmission methods; The content recognition and detection module includes webpage content encoding recognition, Chinese and English word segmentation, hidden link recognition, misspelling recognition, and sensitive word recognition; Includes the following steps: Step 1: Retrieve the website attributes of the task to be tested from the task table, including the URL address and scheduling frequency; Step 2: Based on the URL, use DNS caching technology to retrieve the network address corresponding to the URL; Step 3: Construct an HTTP request, establish a connection to the service, send the request, and obtain its status; Step 4: If an abnormality is detected, an alarm will be triggered; Step 5: Construct an HTTP request to the server and download the webpage; Step Six: Use a webpage analysis algorithm to analyze the webpage content and filter out unnecessary tags and content; Step 7: Use content encoding recognition algorithms to encode and recognize the webpage content, and then transcode the webpage content; Step 8: Use Chinese and English word segmentation and recognition algorithms to perform Chinese word segmentation on the transcoded content; Step 9: For the webpage analysis results, use hidden link identification technology to analyze the hyperlinks in the webpage. If a hidden link is found, an alert will be issued. Step 10: For the webpage content, use typo detection technology to analyze the content. If typos are found, issue an alert.
[0006] Preferably, the steps of the webpage fast analysis algorithm are as follows: Step 201: Check the current character. If the character is a space or a newline character, skip these characters. Step 202: If it is an escape character, skip it; Step 203: If it is not a '<' symbol, save it as text; otherwise, proceed to step four. Step 204: If it is a comment, skip all comments; otherwise, go to step 205. Step 205: If it is any of the tags a, link, base, frame, iframe, title, meta, img, or style, then analyze the attributes of the tag. If it is an a tag, then use the Sunday algorithm to extract the content between the a tags. Step 206: Skip the other tags directly; Step 207: If the end of the file has been reached, exit; otherwise, return to step 201.
[0007] Preferably, the specific steps of the content encoding and recognition algorithm are as follows: Step 301: Obtain the encoding from the HTTP response header; Step 302: Obtain the encoding of the HTML page; Step 303: If either of the codes obtained in steps 301 and 302 is empty, then take the non-empty code set; If the two are different, proceed to step five; Step 304: Based on the non-empty encoding set, transcode the HTML content that has been obtained. If the transcoding is successful, end the process; otherwise, proceed to step 305. Step 305: Clear the encoding set and call the self-written Python encoding judgment module interface to judge the content of the webpage, obtain the encoding of the content, and go to step 304. If it fails, the encoding recognition is unsuccessful.
[0008] Preferably, the specific steps of the Chinese and English word segmentation and recognition algorithm are as follows: Step 401: Starting from the rightmost matched position of the string to be segmented, determine the length of the first character – based on UTF-8 encoding; Step 402: If the character is an English character or an ASCII character, it indicates that the string will be an English word; otherwise, proceed to step 404. Step 403: If there is already a recognized English character, and that character is a space or a non-ASCII character, then the current English word is considered to be the end; otherwise, continue matching. Step 404: Starting from this byte, backtrack by the length of one character, as a Chinese character. At the same time, match the previously matched length in the dictionary. If the match is successful, continue backtracking. If there are unrecognized characters, add the unrecognized string to the word queue. Otherwise, consider it a mismatch and proceed to step 405. Step 406: If the match fails, and if there are English characters before it, then the English characters are considered to have ended; at the same time, the character is added to the unrecognized character queue; and the recognized words are added to the word segmentation queue. Step 407: Change the word segmentation position. If the beginning has been reached, the word segmentation ends; otherwise, proceed to step 401.
[0009] Preferably, the domain name is hashed and checked in the hash chain. If it exists, it is determined whether the DNS resolution failed or the DNS address has expired. If so, the domain name is resolved and the resolved address is added to the hash chain; otherwise, the Internet address of the domain name is directly retrieved.
[0010] Compared with existing technologies, this invention provides a standardized dark chain sample construction and labeling system and method, which has the following beneficial effects: 1. This invention employs a program that, upon startup, retrieves the range of sites to be monitored from the task scheduling node based on the local IP address and computing power; simultaneously, it obtains the detection frequency for each site. After program startup, a timeout period is set according to the detection frequency of each site. If the current time exceeds the timeout period, task scheduling begins. To accelerate website DNS resolution, we implemented our own DNS resolution system. The main implementation method is to first perform a hash calculation on the domain name and check if it exists in the hash chain. If it exists, we check whether DNS resolution has failed or the DNS address has expired. If so, we resolve the domain name and add the resolved address to the hash chain; otherwise, we directly retrieve the domain name's Internet address, thus accelerating DNS resolution.
[0011] 2. This invention employs asynchronous download technology, fully utilizing computer resources to maintain a monitoring capacity of 5000 sites per server per day, even within the operating system's maximum open file limit. It features rapid HTML analysis technology. Traditional HTML analysis builds a DOM tree by analyzing all tags, wasting memory and computational resources. Our implementation uses rapid tag matching to filter out unnecessary tags, saving memory and accelerating HTML analysis. It also boasts accurate HTML page encoding analysis capabilities. Traditional HTML page encoding analysis relies solely on meta tags or server-returned encoding, which is inaccurate. Our approach uses meta tags, server-returned encoding values, and a local encoding detection program, significantly improving page encoding recognition accuracy. Finally, it utilizes powerful Chinese word segmentation technology. Traditional Chinese word segmentation only analyzes known Chinese words, failing to adequately analyze unknown words such as dates and English words. Our implementation correctly distinguishes known Chinese words while grouping unrecognized words, such as dates, together, effectively solving this problem. Attached Figure Description
[0012] Figure 1 This is a flowchart illustrating the website monitoring methodology. Figure 2 This is a schematic diagram of the server configuration; Figure 3 This is a schematic diagram of the overall process of the task detection and scheduling module; Figure 4 A schematic diagram of the overall functional flow of the internal storage management module. Detailed Implementation
[0013] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.
[0014] Example 1 Please refer to Figures 1-4 As shown, a standardized dark link sample construction and labeling system includes a task scheduling module, a memory management module, a DNS caching module, a web page asynchronous download module, and a content recognition and detection module, characterized in that: The task scheduling module is used to schedule status detection and content detection tasks so that website monitoring can be completed within a specific period. The memory management module pre-allocates and serializes the memory required by all URL scanning tasks to facilitate rapid memory allocation and unified management. The DNS caching module caches the DNS addresses of the requested URLs to improve their resolution speed. The asynchronous webpage download module enables fast downloading of webpages using various transmission methods. The content recognition and detection module includes webpage content encoding recognition, Chinese and English word segmentation, hidden link recognition, misspelling recognition, and sensitive word recognition; Includes the following steps: Step 1: Retrieve the website attributes of the task to be tested from the task table, including the URL address and scheduling frequency; Step 2: Based on the URL, use DNS caching technology to retrieve the network address corresponding to the URL; Step 3: Construct an HTTP request, establish a connection to the service, send the request, and obtain its status; Step 4: If an abnormality is detected, an alarm will be triggered; Step 5: Construct an HTTP request to the server and download the webpage; Step Six: Use a webpage analysis algorithm to analyze the webpage content and filter out unnecessary tags and content; Step 7: Use content encoding recognition algorithms to encode and recognize the webpage content, and then transcode the webpage content; Step 8: Use Chinese and English word segmentation and recognition algorithms to perform Chinese word segmentation on the transcoded content; Step 9: For the webpage analysis results, use hidden link identification technology to analyze the hyperlinks in the webpage. If a hidden link is found, an alert will be issued. Step 10: For the webpage content, use typo detection technology to analyze the content. If typos are found, issue an alert.
[0015] The steps of a fast webpage analysis algorithm are as follows: Step 201: Check the current character. If the character is a space or a newline character, skip these characters. Step 202: If it is an escape character, skip it; Step 203: If it is not a '<' symbol, save it as text; otherwise, proceed to step four. Step 204: If it is a comment, skip all comments; otherwise, go to step 205. Step 205: If it is any of the tags a, link, base, frame, iframe, title, meta, img, or style, then analyze the attributes of the tag. If it is an a tag, then use the Sunday algorithm to extract the content between the a tags. Step 206: Skip the other tags directly; Step 207: If the end of the file has been reached, exit; otherwise, return to step 201.
[0016] The specific steps of the content encoding recognition algorithm are as follows: Step 301: Obtain the encoding from the HTTP response header; Step 302: Obtain the encoding of the HTML page; Step 303: If either of the codes obtained in steps 301 and 302 is empty, then take the non-empty code set; If the two are different, proceed to step five; Step 304: Based on the non-empty encoding set, transcode the HTML content that has been obtained. If the transcoding is successful, end the process; otherwise, proceed to step 305. Step 305: Clear the encoding set and call the self-written Python encoding judgment module interface to judge the content of the webpage, obtain the encoding of the content, and go to step 304. If it fails, the encoding recognition is unsuccessful.
[0017] The specific steps of the Chinese and English word segmentation and recognition algorithm are as follows: Step 401: Starting from the rightmost matched position of the string to be segmented, determine the length of the first character – based on UTF-8 encoding; Step 402: If the character is an English character or an ASCII character, it indicates that the string will be an English word; otherwise, proceed to step 404. Step 403: If there is already a recognized English character, and that character is a space or a non-ASCII character, then the current English word is considered to be the end; otherwise, continue matching. Step 404: Starting from this byte, backtrack by the length of one character, as a Chinese character. At the same time, match the previously matched length in the dictionary. If the match is successful, continue backtracking. If there are unrecognized characters, add the unrecognized string to the word queue. Otherwise, consider it a mismatch and proceed to step 405. Step 406: If the match fails, and if there are English characters before it, then the English characters are considered to have ended; at the same time, the character is added to the unrecognized character queue; and the recognized words are added to the word segmentation queue. Step 407: Change the word segmentation position. If the beginning has been reached, the word segmentation ends; otherwise, proceed to step 401.
[0018] The domain name is hashed and checked against the hash chain. If it exists, the DNS resolution is checked to see if the DNS address has expired. If so, the domain name is resolved and the resolved address is added to the hash chain. Otherwise, the domain name's Internet address is retrieved directly.
[0019] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.
Claims
1. A standardized dark link sample construction and labeling system, characterized in that, It includes a task scheduling module, a memory management module, a DNS caching module, a web page asynchronous download module, and a content recognition and detection module, characterized in that: The task scheduling module is used to schedule status detection and content detection tasks so that website monitoring can be completed within a specific period. The memory management module pre-allocates and serializes the memory required by all URL scanning tasks to facilitate rapid memory allocation and unified management. The DNS caching module caches the DNS address of the requested URL to improve its resolution speed; The asynchronous webpage download module enables fast downloading of webpages using various transmission methods; The content recognition and detection module includes webpage content encoding recognition, Chinese and English word segmentation, hidden link recognition, misspelling recognition, and sensitive word recognition; Includes the following steps: Step 1: Retrieve the website attributes of the task to be tested from the task table, including the URL address and scheduling frequency; Step 2: Based on the URL, use DNS caching technology to retrieve the network address corresponding to the URL; Step 3: Construct an HTTP request, establish a connection to the service, send the request, and obtain its status; Step 4: If an abnormality is detected, an alarm will be triggered; Step 5: Construct an HTTP request to the server and download the webpage; Step Six: Use a webpage analysis algorithm to analyze the webpage content and filter out unnecessary tags and content; Step 7: Use content encoding recognition algorithms to encode and recognize the webpage content, and then transcode the webpage content; Step 8: Use Chinese and English word segmentation and recognition algorithms to perform Chinese word segmentation on the transcoded content; Step 9: For the webpage analysis results, use hidden link identification technology to analyze the hyperlinks in the webpage. If a hidden link is found, an alert will be issued. Step 10: For the webpage content, use typo detection technology to analyze the content. If typos are found, issue an alert.
2. The standardized dark chain sample construction and labeling system and method according to claim 1, characterized in that: The steps of the fast webpage analysis algorithm are as follows: Step 201: Check the current character. If the character is a space or a newline character, skip these characters. Step 202: If it is an escape character, skip it; Step 203: If it is not a '<' symbol, save it as text; otherwise, proceed to step four. Step 204: If it is a comment, skip all comments; otherwise, go to step 205. Step 205: If it is any of the tags a, link, base, frame, iframe, title, meta, img, or style, then analyze the attributes of the tag. If it is an a tag, then use the Sunday algorithm to extract the content between the a tags. Step 206: Skip the other tags directly; Step 207: If the end of the file has been reached, exit; otherwise, return to step 201.
3. The standardized dark chain sample construction and labeling system and method according to claim 1, characterized in that: The specific steps of the content encoding and recognition algorithm are as follows: Step 301: Obtain the encoding from the HTTP response header; Step 302: Obtain the encoding of the HTML page; Step 303: If either of the codes obtained in steps 301 and 302 is empty, then take the non-empty code set; If the two are different, proceed to step five; Step 304: Based on the non-empty encoding set, transcode the HTML content that has been obtained. If the transcoding is successful, end the process; otherwise, proceed to step 305. Step 305: Clear the encoding set and call the self-written Python encoding judgment module interface to judge the content of the webpage, obtain the encoding of the content, and go to step 304. If it fails, the encoding recognition is unsuccessful.
4. The standardized dark chain sample construction and labeling system and method according to claim 1, characterized in that: The specific steps of the Chinese and English word segmentation and recognition algorithm are as follows: Step 401: Starting from the rightmost matched position of the string to be segmented, determine the length of the first character – based on UTF-8 encoding; Step 402: If the character is an English character or an ASCII character, it indicates that the string will be an English word; Otherwise, proceed to step 404; Step 403: If there is already a recognized English character, and that character is a space or a non-ASCII character, then the current English word is considered to be the end; otherwise, continue matching. Step 404: Starting from this byte, backtrack by the length of one character, as a Chinese character. At the same time, match the previously matched length in the dictionary. If the match is successful, continue backtracking. If there are unrecognized characters, add the unrecognized string to the word queue. Otherwise, consider it a mismatch and proceed to step 405. Step 406: If the match fails, and if there are English characters before it, then the English characters are considered to have ended; at the same time, the character is added to the unrecognized character queue; and the recognized words are added to the word segmentation queue. Step 407: Change the word segmentation position. If the beginning has been reached, the word segmentation ends; otherwise, proceed to step 401.
5. The standardized dark chain sample construction and labeling system and method according to claim 1, characterized in that: The domain name is hashed and checked against the hash chain. If it exists, the DNS resolution is checked to see if the DNS address has expired. If so, the domain name is resolved and the resolved address is added to the hash chain. Otherwise, directly retrieve the Internet address of the domain name.