A web page text content matching analysis system based on big data
The webpage text content matching and analysis system based on big data solves the problems of complicated rule formulation, identification errors and disordered order in webpage text extraction, and achieves efficient and accurate text content extraction and storage. It has a wide range of applications and is suitable for various webpage formats.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ASIA PACIFIC CONSTR TECH INFORMATION RES INST CO LTD
- Filing Date
- 2023-09-21
- Publication Date
- 2026-05-29
AI Technical Summary
Existing technologies for extracting web page text suffer from problems such as complicated rule formulation, limited applicability, frequent recognition errors, and disordered order, and the extraction efficiency of web page text content is low.
A webpage text content matching and analysis system based on big data is adopted, including a webpage capture module, an image processing program, a text matching and analysis unit, and a cloud storage platform. Through image processing, character recognition, and grammatical error analysis, coordinate rules are established to extract key text content. Combined with deep learning algorithms, grammatical errors are processed to achieve accurate extraction and storage of text content.
It achieves efficient and accurate extraction of text content, removes irrelevant content, purifies the page, improves recognition accuracy, shortens processing time, expands the scope of application, and provides a memory function for easy subsequent retrieval and download.
Smart Images
Figure CN117197819B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of big data processing technology, specifically to a webpage text content matching and analysis system based on big data. Background Technology
[0002] In the existing network, most web pages contain a large amount of useless information besides text, such as advertisements, images, and non-article content. Therefore, existing technologies employ web page extraction schemes to extract the text from web pages, such as titles and article content, and then display the extracted content to users in a customized reading webpage, providing clean web page content free from useless information.
[0003] In existing technology, after a user loads a webpage, the URL is matched against set rules. If the URL matches a rule, the content of the webpage is split according to the rules. This allows the user to see a filtered webpage, making reading more convenient and focused.
[0004] However, the above methods have many problems because titles, advertisements, pop-ups, recommended content, etc., have different forms for different web pages. It is difficult to formulate rules that can be adapted to many types of web pages, so the recognition will have many problems. In addition, every time the web page text is extracted, regardless of whether it has been matched before, it needs to be completely matched again, which is time-consuming.
[0005] Chinese patent CN103150389B discloses a method and apparatus for processing webpage text content matching settings, which can solve the problem that the matching process requires a complete comparison of the webpage text content with the webpage text content matching settings, which consumes a lot of time and resources, resulting in low webpage extraction efficiency and excessively long waiting time for webpage display.
[0006] This method saves the matched webpages and extracted text content to shorten the time and improve efficiency for the next matching. However, like existing methods, this method requires extracting the text content of the webpage. Some existing webpages extract text line by line from top to bottom, while some webpages, such as academic papers, may be formatted with two columns. In this case, extracting text line by line from top to bottom will cause the order of the lines to be intertwined, resulting in overall disorder and making the extracted text content unusable. In addition, some webpages cannot be copied and can only extract text using methods such as text recognition. However, text recognition is prone to problems such as similar character and text recognition errors, inability to correctly and completely recognize and display foreign languages, and even interference from other lines, leading to more errors in text extraction. Summary of the Invention
[0007] To address the shortcomings of existing technologies, this invention provides a webpage text content matching and analysis system based on big data. This system solves the problems of existing webpage text extraction methods that rely on rules to remove irrelevant content, which are cumbersome to formulate and have limited applicability. It also addresses the issues of order disorder and recognition errors that easily occur in existing webpage text content extraction methods.
[0008] To achieve the above objectives, the present invention is implemented through the following technical solution: a webpage text content matching and analysis system based on big data, including a webpage capture module and a text matching and analysis unit, and also including an image processing program, a text matching degree processing program and a cloud storage platform;
[0009] The webpage capture module is used to monitor page browsing data and capture webpage images;
[0010] The image processing program is used to remove the inherent frames of the browser page from multiple captured images of a webpage, delete duplicate images, and then stitch them together.
[0011] The text matching degree processing program is used to identify the position of the search keywords on the parent page in the text content of the current page, and to obtain the matching degree data between the keywords and the text content;
[0012] The text matching and analysis unit is used to identify images to obtain text, and to extract the text content where the keywords are located by modeling and creating coordinates. At the same time, it extracts text content with the same rules according to coordinate rules to filter out irrelevant content, and performs recognition and analysis on the text content. It uses a basic grammar error algorithm framework with a training model to insert a deep learning algorithm for grammar error recognition, and combines big data to process the grammar error to eliminate the grammar error.
[0013] The cloud storage platform allows images and webpage links processed by the image processing program to be uploaded and categorized. It also includes a search and redirect module that redirects users to the searched links or redirects images to the text matching and analysis unit for further processing.
[0014] Preferably, the image processing program includes a webpage frame extraction module, an image stitching module, and an image fusion module;
[0015] The webpage frame extraction module extracts the inherent frame area of the browser webpage and outputs the frame area data to the image stitching module.
[0016] The image stitching module first crops the image to remove the inherent frame, then aligns multiple images so that their feature points correspond, and uses an image stitching algorithm to stitch the aligned images together to finally obtain a large image containing all the image content, which is then output to the image fusion module.
[0017] The image fusion module will segment the stitched large image, extract the regions of each original image, perform image fusion on each region, and merge the pixels of the same region in multiple images to finally obtain a composite image with duplicate parts removed.
[0018] Preferably, the text matching and analysis unit includes a character recognition and screening program, a text grammar analysis and processing program, and a text display and download module;
[0019] The text recognition and screening program identifies and highlights all text, adds background color for modeling, marks the background area of the text containing keywords, establishes a coordinate range for the background area of the text containing keywords, with the origin at the top left corner of the image, the X-axis increasing to the right, and the Y-axis increasing downwards. Each character in the background color model is set as a point to obtain coordinates (X, Y). The minimum coordinate A(X, Y) in the model containing keywords is obtained and set. a Y a The maximum coordinate is B(X). b Y b ), set the minimum coordinate C(X) in the model without keywords. c Y c The maximum coordinate is D(X). d Y d Set rules to identify the required text content, and remove the text content corresponding to other models that do not conform to the set rules.
[0020] Preferably, the rules are set as follows:
[0021] X a ≤X≤X b ;
[0022] or
[0023] X d -X c =X b -X a ;
[0024] The text content corresponding to the model that meets the above rules is the text content to be extracted.
[0025] Preferably, the text grammar analysis and processing program includes a letter recognition and translation module, a grammar error identification and marking module, and a marked text character replacement module. The grammar error identification and marking module identifies grammar errors and marks contradictory words. If a grammar error exists, it is output to the letter recognition and translation module and the marked text character replacement module for processing. If no grammar error exists, it is directly output to the text matching and analysis unit.
[0026] The letter recognition and translation module translates existing letters. If a letter cannot be translated, it is removed. If the translation is successful, it is output to the grammar error recognition and marking module for further grammar error judgment.
[0027] The marked text character replacement module recognizes the marked non-alphabetic characters, combines big data to perform similar replacements on Chinese characters and symbols, and outputs the replacements to the grammar error recognition and marking module for identification. If there are no grammar errors, the text is retained; if there are grammar errors, the text is deleted.
[0028] Preferably, the grammar error recognition and marking module identifies lines consisting entirely of letters or numbers as formulas and does not mark them as grammar errors.
[0029] Preferably, the text display and download module is used to list the processed text in segments and create a pop-up marker on the right side of the webpage. Clicking the marker will bring up the recognized text, which can be downloaded and copied.
[0030] Preferably, the cloud storage platform includes an automatic upload module, a tagging and classification module, an automatic cleaning module, and a retrieval and redirection module. The automatic upload module uploads secondary and lower-level webpage links and image recording data to the tagging and classification module for classification and tagging.
[0031] The tagging and classification module independently classifies the data saved from each keyword search and tags it into first-level and second-level categories according to conditions.
[0032] The automatic cleanup module automatically deletes data when the cloud storage platform's memory is insufficient, following a priority order of secondary to primary and chronological order.
[0033] The search and redirect module provides a window for searching content stored on the cloud storage platform and clicking on links to redirect to the relevant page.
[0034] Preferably, if the link summary interface that pops up after the browser searches for keywords is set as the first-level webpage, then the second-level webpage refers to the webpage that pops up after clicking on the first-level webpage, the webpage opened on the basis of the second-level webpage is the third-level webpage, and so on.
[0035] Preferably, the difference between the first-level marked file and the second-level marked file lies in the manually marked content and the search files below the browser's favorite pages.
[0036] Beneficial effects
[0037] This invention provides a webpage text content matching and analysis system based on big data. Compared with existing technologies, it has the following advantages:
[0038] 1. This big data-based webpage text content matching and analysis system can capture and process images of browsed webpages, and use text recognition and processing algorithms to identify the important text content and remove the unimportant content, thus obtaining simple and direct text content. It can also process the identified text content for grammatical errors, automatically producing more accurate text content. It also has a storage and memory function, which facilitates subsequent retrieval and re-identification of the retrieved content.
[0039] 2. This big data-based webpage text content matching and analysis system uses a method of screenshotting and stitching together the browsed page, removing the browser page frame first, and then removing duplicate parts from multiple screenshots. On the one hand, this reduces the processing scope, and on the other hand, merging it into a single image facilitates subsequent text recognition of the entire browsed page, making subsequent viewing more convenient.
[0040] 3. This big data-based webpage text content matching and analysis system uses coordinates to frame text. Based on simple coordinate comparison rules, it can extract text content that matches normal search criteria and remove useless text content such as advertisements, pop-ups, and recommended content that does not meet the rules. This achieves the extraction of the required text and cleans up the page for easier viewing and subsequent downloading. The extraction rules are simple and convenient, with a wide range of applications. Furthermore, it identifies text by dividing it into regions according to paragraph distribution, avoiding the problem of line order disorder caused by parallel paragraph text recognition when performing whole-page recognition in existing text recognition methods.
[0041] 4. This big data-based webpage text content matching and analysis system, through the setting of a text error analysis and processing program, can analyze text content for grammatical errors. It can also attempt to replace grammatically incorrect content by matching similar text characters to eliminate errors caused by text recognition errors or page line recognition errors. For English, it can also identify semantics through translation and other methods to organize the document into a normal one, effectively improving the accuracy of text recognition. Furthermore, it incorporates deep learning algorithms, which can record some commonly used professional terms in daily input and retrieval to avoid misidentification as grammatical errors. It can also roughly determine the algorithm formula, making it quite intelligent.
[0042] 4. This big data-based webpage text content matching and analysis system, by setting up a cloud storage platform, can save the previously processed search page images and URLs for easy re-searching. This adds a memory function, avoiding the problem of difficulty in quickly finding webpages that have been viewed in memory based solely on browser history. Moreover, the stored content consists of processed images, allowing for quick skipping of previous image processing steps when processing the page again, reducing text extraction efficiency. The cloud storage platform also has functions for automatically classifying based on importance and automatically cleaning up memory, making it easy to use. Attached Figure Description
[0043] Figure 1 This is a system principle block diagram of the present invention;
[0044] Figure 2 A schematic diagram of coordinate axes is established for this invention. Detailed Implementation
[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0046] This invention provides five technical solutions:
[0047] Figure 1 The first implementation is shown: a web page text content matching and analysis system based on big data, including a web page capture module and a text matching and analysis unit, as well as an image processing program, a text matching degree processing program and a cloud storage platform;
[0048] The webpage capture module is used to monitor page browsing data and capture webpage images;
[0049] Image processing program used to stitch together multiple captured images from a webpage after removing the browser's inherent frames and eliminating duplicate images.
[0050] The text matching score processing program is used to identify the position of the search keywords from the parent page in the text content of the current page, and to obtain the matching score data between the keywords and the text content;
[0051] The text matching and analysis unit is used to identify images and obtain text. It uses a modeling and coordinate creation method to extract the text content where the keywords are located. At the same time, it extracts text content with the same coordinate rules to filter out irrelevant content. It also performs recognition and analysis on the text content, uses a basic grammar error algorithm framework with a training model and inserts a deep learning algorithm to identify grammar errors, and combines big data to process the grammar errors to eliminate them.
[0052] The cloud storage platform allows users to upload and categorize images and webpage links processed by image processing programs. It also includes a search and redirect module, which redirects users to the searched links or to the text matching and analysis unit for further processing.
[0053] This system can capture and process images of browsed web pages, and use text recognition and processing algorithms to identify the important text content and remove the unimportant content, thus obtaining simple and direct text content. It can also perform grammatical error processing on the identified text content, automatically producing more accurate text content. It also has a storage and memory function, which facilitates subsequent retrieval and re-identification of the retrieved content.
[0054] Figure 1 The second implementation is shown, and the main difference from the first implementation is that the image processing program includes a webpage frame extraction module, an image stitching module, and an image fusion module;
[0055] The webpage frame extraction module extracts the inherent frame area of the browser webpage and outputs the frame area data to the image stitching module.
[0056] The image stitching module first crops the image to remove the inherent frame, then aligns multiple images so that their feature points correspond, and uses an image stitching algorithm (such as SIFT algorithm) to stitch the aligned images together to finally obtain a large image containing all the image content, which is then output to the image fusion module.
[0057] The image fusion module will segment the stitched large image, extract the regions of each original image, perform image fusion on each region, and merge the pixels of the same region in multiple images (the fusion method can use weighted average, maximum value, minimum value, etc.) to finally obtain a composite image with duplicate parts removed.
[0058] By using a method of stitching together screenshots of the browsed pages and removing the browser page frames in advance, and further removing duplicate parts from multiple page screenshots, the processing scope can be reduced on the one hand, and the fusion into a single image facilitates text recognition of the entire browsed page later, making subsequent viewing more convenient.
[0059] Figure 1-2 The third implementation is shown, and the main difference from the second implementation is that the text matching and analysis unit includes a character recognition screening program, a text grammatical error analysis and processing program, and a text display and download module;
[0060] The text recognition and screening program identifies and highlights all text, adds background color for modeling, marks the background area containing keywords, establishes a coordinate range for the background area containing keywords, with the origin at the top left corner of the image, the X-axis increasing to the right, and the Y-axis increasing downwards. Each character in the background color model is assigned a point, resulting in coordinates (X, Y). The minimum coordinate A(X, Y) of the model containing keywords is obtained and set. a Y a The maximum coordinate is B(X).b Y b ), set the minimum coordinate C(X) in the model without keywords. c Y c The maximum coordinate is D(X). d Y d Set rules to identify the required text content, and remove the text content corresponding to other models that do not conform to the set rules.
[0061] The rules are set as follows:
[0062] X a ≤X≤X b ;
[0063] or
[0064] X d -X c =X b -X a ;
[0065] The text content corresponding to the model that meets the above rules is the text content to be extracted.
[0066] By using coordinates to frame the text and following simple coordinate comparison rules, text content that matches normal search criteria can be extracted, while useless text content such as advertisements, pop-ups, and recommended content that does not meet the rules can be removed. This achieves the extraction of the required text and cleans up the page for easier viewing and subsequent downloading. The extraction rules are simple, convenient, and widely applicable. Furthermore, region recognition based on paragraph distribution avoids the problem of line order disorder caused by parallel paragraph text recognition when performing full-page recognition in existing text recognition methods.
[0067] Figure 1 The fourth implementation method is shown. The main difference between the fourth and third implementation methods is that the text grammar analysis and processing program includes a letter recognition and translation module, a grammar grammar identification and marking module, and a marked text character replacement module. The grammar grammar identification and marking module identifies grammar ...
[0068] The letter recognition and translation module translates existing letters. If a letter cannot be translated, it is removed. If the translation is successful, it is output to the grammar error recognition and marking module for further grammar error judgment.
[0069] The text character replacement module at the marked location identifies the marked non-alphabetic characters, combines big data to perform similar replacements on Chinese characters and symbols, and outputs the replacements to the grammar error recognition and marking module for identification. If there are no grammar errors, the text is retained; if there are grammar errors, the text is deleted.
[0070] The grammar error detection and marking module identifies lines consisting entirely of letters or numbers as formulas and does not mark them as grammar errors.
[0071] The text display and download module is used to list the processed text in segments and create a pop-up marker on the right side of the webpage. Clicking the marker will bring up the recognized text, which can then be downloaded and copied.
[0072] By setting up a text error analysis and processing program, it is possible to analyze the grammatical errors in the text content. It can also attempt to replace grammatically incorrect content by matching similar characters, thereby eliminating grammatical errors caused by text recognition errors or page line recognition errors. For English, it can also identify semantics through translation and other methods to organize the document into a normal one, effectively improving the accuracy of text recognition. Furthermore, it incorporates a deep learning algorithm, which can record some commonly used professional terms in daily input and retrieval to avoid misidentification as grammatical errors. It can also roughly determine the algorithm formula, making it quite intelligent.
[0073] Figure 1 The fifth implementation is shown. The main difference between the fifth and fourth implementations is that the cloud storage platform includes an automatic upload module, a tagging and classification module, an automatic cleaning module, and a retrieval and redirection module. The automatic upload module uploads secondary and lower-level webpage links and image recording data to the tagging and classification module for classification and tagging.
[0074] The tagging and classification module independently categorizes the data saved from each keyword search and tags it into primary and secondary levels according to conditions;
[0075] The automatic cleanup module automatically deletes files when the cloud storage platform runs out of memory, prioritizing secondary files and then primary files, as well as in chronological order.
[0076] The search and redirect module provides a window for searching content stored on the cloud storage platform and redirecting users to relevant content by clicking links.
[0077] If the browser's search keyword-based link summary interface is set as the first-level webpage, then the second-level webpage refers to the webpage that pops up after clicking on the first-level webpage, the webpage that opens based on the second-level webpage is the third-level webpage, and so on.
[0078] The difference between level 1 and level 2 marked files lies in the manually marked content and the search files below the browser's favorite pages.
[0079] By setting up a cloud storage platform, previously processed search page images and URLs can be saved for easy re-searching. This adds a memory function, avoiding the problem of difficulty in quickly finding web pages that you remember viewing based solely on your browser history. Moreover, the stored content consists of processed images, allowing you to quickly skip previous image processing steps when working on the same page again, reducing text extraction efficiency. The cloud storage platform also has functions for automatically classifying pages based on their importance and automatically cleaning up memory, making it easy to use.
[0080] Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.
[0081] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0082] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A webpage text content matching and analysis system based on big data, comprising a webpage capture module and a text matching and analysis unit, characterized in that: It also includes image processing programs, text matching processing programs, and cloud storage platforms; The webpage capture module is used to monitor page browsing data and capture webpage images; The image processing program is used to remove the inherent frames of the browser page from multiple captured images of a webpage, delete duplicate images, and then stitch them together. The text matching degree processing program is used to identify the position of the search keywords on the parent page in the text content of the current page, and to obtain the matching degree data between the keywords and the text content; The text matching and analysis unit is used to identify images to obtain text, and to extract the text content where the keywords are located by modeling and creating coordinates. At the same time, it extracts text content with the same rules according to coordinate rules to filter out irrelevant content, and performs recognition and analysis on the text content. It uses a basic grammar error algorithm framework with a training model to insert a deep learning algorithm for grammar error recognition, and combines big data to process the grammar error to eliminate the grammar error. The cloud storage platform allows images and webpage links processed by the image processing program to be uploaded and categorized. It also includes a search and redirect module that redirects to the searched links or redirects the images to the text matching and analysis unit for processing. The text matching and analysis unit includes a character recognition and screening program, a text grammar analysis and processing program, and a text display and download module; The text recognition and screening program identifies and highlights all text, adds background color for modeling, marks the text background area containing keywords, and establishes a coordinate interval for the text background area containing keywords. The origin of the coordinates is the upper left corner of the image, the rightward direction is the X-axis increasing direction, and the downward direction is the Y-axis increasing direction. Each character in the background color model is set as a point to obtain the coordinates. Get and set the minimum coordinate A in the model containing the keyword. The maximum coordinate is B Set the minimum coordinate C in the model without keywords. The maximum coordinate is D Set rules to identify the required text content, and remove the text content corresponding to other models that do not conform to the set rules.
2. The webpage text content matching and analysis system based on big data according to claim 1, characterized in that: The image processing program includes a webpage frame extraction module, an image stitching module, and an image fusion module; The webpage frame extraction module extracts the inherent frame area of the browser webpage and outputs the frame area data to the image stitching module. The image stitching module first crops the image to remove the inherent frame, then aligns multiple images so that their feature points correspond, and uses an image stitching algorithm to stitch the aligned images together to finally obtain a large image containing all the image content, which is then output to the image fusion module. The image fusion module will segment the stitched large image, extract the regions of each original image, perform image fusion on each region, and merge the pixels of the same region in multiple images to finally obtain a composite image with duplicate parts removed.
3. The webpage text content matching and analysis system based on big data according to claim 1, characterized in that: The rules are set as follows: ； or ； The text content corresponding to the model that meets the above rules is the text content to be extracted.
4. The webpage text content matching and analysis system based on big data according to claim 1, characterized in that: The text error analysis and processing program includes a letter recognition and translation module, an error recognition and marking module, and a marked text character replacement module. The error recognition and marking module identifies errors and marks contradictory words. If an error exists, the error is output to the letter recognition and translation module and the marked text character replacement module for processing. If no error exists, the error is directly output to the text matching and analysis unit. The letter recognition and translation module translates existing letters. If a letter cannot be translated, it is removed. If the translation is successful, it is output to the grammar error recognition and marking module for further grammar error judgment. The marked text character replacement module recognizes the marked non-alphabetic characters, combines big data to perform similar replacements on Chinese characters and symbols, and outputs the replacements to the grammar error recognition and marking module for identification. If there are no grammar errors, the text is retained; if there are grammar errors, the text is deleted.
5. The webpage text content matching and analysis system based on big data according to claim 4, characterized in that: The grammar error identification and marking module identifies lines consisting entirely of letters or numbers as formulas and does not mark them as grammar errors.
6. The webpage text content matching and analysis system based on big data according to claim 1, characterized in that: The text display and download module is used to list the processed text in segments and create a pop-up marker on the right side of the webpage. Clicking the marker will bring up the recognized text, which can be downloaded and copied.
7. The webpage text content matching and analysis system based on big data according to claim 1, characterized in that: The cloud storage platform includes an automatic upload module, a tagging and classification module, an automatic cleaning module, and a retrieval and redirection module. The automatic upload module uploads secondary and lower-level webpage links and image recording data to the tagging and classification module for classification and tagging. The tagging and classification module independently classifies the data saved from each keyword search and tags it into first-level and second-level categories according to conditions. The automatic cleanup module automatically deletes data when the cloud storage platform's memory is insufficient, following a priority order of secondary to primary and chronological order. The search and redirect module provides a window for searching content stored on the cloud storage platform and clicking on links to redirect to the relevant page.
8. The webpage text content matching and analysis system based on big data according to claim 7, characterized in that: If the link summary interface that pops up after a browser searches for keywords is set as the first-level webpage, then the second-level webpage refers to the webpage that pops up after clicking on the first-level webpage, the webpage that opens based on the second-level webpage is the third-level webpage, and so on.
9. A webpage text content matching and analysis system based on big data according to claim 7, characterized in that: The difference between level 1 and level 2 files lies in the manually marked content and the search files below the browser's bookmarks.