Sports event rebroadcasting information aggregation system and method based on intelligent analysis

Abstract

The invention provides a sports event rebroadcasting information aggregation system and method based on intelligent analysis, and the system comprises a data collection layer which comprises a web crawler engine, a request management module, an anti-crawling bypassing module and a dynamic page rendering module, and is used for collecting original HTML data from an electronic program guide forecast website; the intelligent analysis layer comprises a preprocessing cleaning module, a target match matching module, a time processing module and a double traversal analysis module and is used for intelligently analyzing the collected original HTML data; the data storage layer is used for storing the analyzed structured data; and the exception handling mechanism is used for handling network request exception of the data acquisition layer, analysis exception of the intelligent analysis layer and storage exception of the data storage layer. The invention aims to solve the problems of large data error, weak dynamic adaptation, low storage efficiency and the like in the prior art, and is suitable for automatic acquisition, standardized processing, structured storage and efficient scene retrieval of sports event rebroadcast information.

Description

Technical Field

[0001] This invention relates to the field of network information and sports event data processing technology, specifically to a sports event broadcast information aggregation system and method based on intelligent parsing, applicable to scenarios of automated collection, standardized processing, structured storage, and efficient retrieval of sports event broadcast information. Background Technology

[0002] Information such as the schedules and broadcast programs for sports events can generally be found by browsing major Electronic Program Guide (EPG) websites. This process is time-consuming and involves a lot of other useless information. When there are a large number of events to obtain, the workload is enormous.

[0003] The current mainstream solution is based on web crawler-based information gathering technology. A web crawler is a program or script written in a computer language that automatically retrieves information from the internet according to certain rules. Web crawlers typically consist of three parts: data collection (obtaining data from web pages); data processing (parses web pages); and data storage (persistently storing useful information). Using web crawlers, one can automatically obtain information such as sports schedules and broadcast programs from major Electronic Program Guide (EPG) websites and store this information. Some commercial crawling tools and services simplify this process, making it easier for people without software development experience to obtain this information.

[0004] Existing technologies for web scraping have significant errors in data acquisition, resulting in empty or inaccurate sports event information. They cannot match target events to specific needs, display event dates correctly, or show all broadcast programs completely. Their main drawbacks are as follows: 1. Parsing mechanism: single traversal (directly extracting visible elements), resulting in low data integrity; 2. Dynamic adaptability: Relying on fixed CSS selectors may result in empty data and low data accuracy utilization on web pages; 3. Storage architecture: fixed columns (maximum of 10 preset broadcast programs), resulting in low storage space utilization; 4. Time processing accuracy: Simple date matching has low date parsing accuracy; 5. Exception handling: Only raw error logs are recorded, resulting in low system availability. Summary of the Invention

[0005] This invention aims to solve the problems of large data errors, weak dynamic adaptation, and low storage efficiency in the prior art. This invention provides a sports event broadcast information aggregation system and method based on intelligent parsing, which can realize intelligent processing of the entire process from data collection to storage.

[0006] The present invention achieves the above objectives through the following technical solutions: A sports event broadcast information aggregation system based on intelligent parsing, comprising: The data acquisition layer, including a web crawler engine, a request management module, an anti-crawling bypass module, and a dynamic page rendering module, is used to collect raw HTML data from the Electronic Program Guide website. The intelligent parsing layer includes a preprocessing and cleaning module, a target event matching module, a time processing module, and a dual-traversal parsing module, used to intelligently parse the collected raw HTML data. The target event matching module uses a multi-level matching strategy to match the user-defined target event set, including primary matching using regular expressions, intelligent contextual analysis, and event-level verification. The time processing module handles the complex time formats unique to sports events. The dual-traversal parsing module determines the maximum broadcast program capacity through the first traversal, constructs the data structure through the second traversal, and captures broadcast program names through multiple signal sources. The data storage layer, including a dynamic column store module and a versioned snapshot library, is used to store parsed structured data; An exception handling mechanism is used to handle network request exceptions in the data acquisition layer, parsing exceptions in the intelligent parsing layer, and storage exceptions in the data storage layer.

[0007] According to the present invention, a sports event broadcast information aggregation system based on intelligent parsing is provided, wherein the web crawler engine, as the central control system of the data acquisition layer, performs the following: It is used to manage the URLs of websites to be crawled, and sorts the URLs according to preset priority rules to ensure that the URLs of high-priority websites are crawled first. Automatically identify the type of website to be crawled and match the best crawling strategy according to the website type; the crawling strategy includes at least request frequency control, data extraction rules, and anti-crawling mechanism to adapt to the structure and anti-crawling characteristics of different websites; Monitor the operational metrics of each module in the data acquisition layer in real time; dynamically adjust the resource configuration and operating parameters of each module based on the monitoring results.

[0008] According to the sports event broadcast information aggregation system based on intelligent parsing provided by the present invention, the request management module performs full lifecycle management of HTTP requests, including: Receive requests from the web crawler engine, and parse the target website URL, request type, data extraction rules, and associated proxy IP pool and cookie database configuration in the task. Based on the target website's anti-scraping policies or verification requirements, dynamically generate HTTP request headers that conform to the specifications, and support adaptive adjustment of request header parameters through configuration files or machine learning models to simulate real user behavior; Maintain a multi-threaded connection pool, dynamically adjust the number of connections according to the concurrency limit of the target website, and prioritize the reuse of idle connections to reduce handshake overhead; when the connection pool is full, temporarily store request tasks in a priority queue to avoid request failure due to resource contention. The system monitors HTTP response status codes in real time. When access is blocked or requests are too frequent, it automatically switches to a compliant IP from the proxy IP pool and updates the cookie. If the response contains a dynamic token or CAPTCHA link, the anti-scraping bypass module is triggered, and the updated identity credentials are synchronized to the cookie library for reuse in subsequent requests.

[0009] According to the present invention, a sports event broadcast information aggregation system based on intelligent parsing is provided, wherein the anti-crawling bypass module counters various anti-crawling technologies and performs the following: By using real user operation patterns, the behavior analysis and detection of anti-crawler systems are interfered with; After preprocessing simple graphic CAPTCHAs, character content is extracted using optical character recognition (OCR) technology; for complex CAPTCHAs, convolutional neural network (CNN) or recurrent neural network (RNN) models are used for training and real-time recognition, supporting dynamic updates of model parameters to adapt to changes in CAPTCHA styles. Integrate a third-party CAPTCHA cracking API. When local recognition fails, automatically call the service to complete the verification and return the result to the request management module. The request sending frequency is dynamically adjusted based on the target website's historical response latency, request success rate, and current system load; an exponential backoff algorithm is used to gradually extend the waiting time when rate limiting is triggered. Real-time monitoring of HTTP response status codes and anti-scraping warning fields in response headers; when a specific defense mechanism is detected, automatic switching to the backup proxy IP pool, modification of request header parameters, or activation of browser automation tools to simulate a complete browser environment. Establish data interfaces with the request management module and dynamic page rendering module to synchronize successfully identified verification codes or updated proxy IP / cookie information to the request header in real time; at the same time, receive parsing error signals from the intelligent parsing layer to trigger a retry mechanism or adjust the priority of anti-crawling strategies.

[0010] According to the sports event broadcast information aggregation system based on intelligent parsing provided by the present invention, the preprocessing and cleaning module performs preliminary cleaning and standardization processing on the original HTML data, including: Automatically detects the character encoding of the original HTML and performs encoding conversion and unification on garbled content; supports encoding conversion via HTTP response headers, <meta> Infer encoding type from tag or byte stream features and fix text display anomalies caused by encoding inconsistencies; Based on a predefined ad tag library or DOM node position features, identify and delete ad scripts, images, and pop-up code; remove developer comments, redundant line breaks, and indentation spaces from HTML; mark and isolate JavaScript code blocks while retaining key dynamic content trigger points; repair incomplete DOM structures caused by website development errors or anti-scraping mechanisms, and automatically complete missing nodes through a balanced tag algorithm; mark missing static DOM fragments after dynamic rendering with placeholders and record the missing positions for dynamic content tagging units to supplement. Identify key sports event information dynamically generated by JavaScript in HTML and mark it using the following methods: Attribute markers: Add custom attributes to dynamic nodes; Annotation anchoring: Insert special annotation tags before and after dynamic content as a basis for parsing and positioning; The cleaned HTML is transformed into a standardized DOM tree structure in memory, providing a unified access interface for the target event matching module to call; a metadata file containing node cleaning logs is generated.

[0011] According to the present invention, a sports event broadcast information aggregation system based on intelligent parsing is provided, wherein the dual-traversal parsing module extracts broadcast program information from the data matched with the target event, including: Using the target event name as the core anchor, the DOM node containing the event is located using XPath or CSS selectors, and its child nodes are recursively parsed to identify elements that may contain broadcast program information. Iterate through all located broadcast program elements, count their number, and dynamically update the maximum number of broadcast programs variable. This variable serves as the only parameter passed between the two stages. If the current page has pagination or asynchronous loading, then load the subsequent content by simulating scroll events or triggering Ajax requests, repeat the location and statistics operations until all relevant nodes are parsed; Based on the max_broadcast_count determined in the first traversal, pre-allocate storage space and initialize the data structures for broadcast programs 1 to n; Check if the target DOM node is bound to the onmouseover or onclick event, and extract the raw broadcast program data by parsing the JavaScript code in the event handler function; If information about a broadcast program is missing during the second traversal, it is marked as pending completion and an exception log is triggered, while processing of subsequent programs continues; when the signal source fails to capture, it automatically switches to the backup parsing strategy to ensure data integrity.

[0012] The generated structured dataset is transmitted to the dynamic column storage module of the data storage layer through a standardized interface.

[0013] According to the present invention, a sports event broadcast information aggregation system based on intelligent parsing is provided. When checking whether the target DOM node is bound to the onmouseover or onclick event, if the event attribute has no valid data, information is extracted from the visible text of the node, and key fields can be located through OCR or text layout analysis. The captured original broadcast program names are filtered using chained regular expressions to remove non-core information such as advertising icons, separators, and redundant spaces, while retaining standardized program names. The cleaned broadcast program names are associated with and populated with the fields extracted in the first traversal. Cross-field alignment is achieved through timestamps or event IDs to generate a structured dataset.

[0014] According to the present invention, a sports event broadcast information aggregation system based on intelligent parsing is provided, wherein the dynamic column storage module implements adaptive storage and version management for the parsed structured data, including: When receiving structured data, the system first compares the number of data fields with the number of columns in the current stored table. If a new field is detected, the system dynamically generates a new column and updates the table structure, while retaining NULL values ​​from historical data. The data type is automatically inferred from the field content, and the default type rules can be overridden through configuration files.

[0015] Data is aligned with the predefined column mapping table by field name; if no field matches, a dynamic column expansion process is triggered. When the number of data columns is less than the number of columns in the current table, the missing fields are automatically filled with NULL values, and the position of the NULL value is recorded in the metadata log; By using the data write time as the filename prefix, we ensure that each modification generates an independent version file, thus achieving versioned storage of the versioned snapshot library.

[0016] According to the present invention, a sports event broadcast information aggregation system based on intelligent parsing is provided, wherein the target event matching module uses a multi-level matching strategy to match a user-preset set of target events, specifically including: Basic Regular Expression Filtering Unit: Based on the user-preset target event keyword library, it constructs a set of regular expressions to perform the first round of matching on the input text and quickly excludes irrelevant events; Contextual intelligent analysis unit: Combining the context of the event description text, it identifies key attributes of the event through a pre-trained NLP model and excludes non-target events; it also performs secondary filtering based on a negative keyword library. Event Level Verification Unit: Based on a predefined league priority table, only top-tier league events are retained, while second-tier and lower-tier leagues are discarded; Structured Information Extraction Unit: For verified event text, extract match time, participating teams, and broadcast programs using conditional random field or rule engine; compare the extracted match time with the user-preset time range, and exclude matches that do not meet the conditions.

[0017] A method for aggregating sports event broadcast information based on intelligent parsing, characterized in that the method employs the aforementioned sports event broadcast information aggregation system based on intelligent parsing, and includes the following steps: Data Acquisition: Raw HTML data is collected from the Electronic Program Guide (EPB) website using a web crawler engine, request management module, anti-crawling bypass module, and dynamic page rendering module. Intelligent parsing: The collected raw HTML data is intelligently parsed through a preprocessing and cleaning module, a target event matching module, a time processing module, and a dual-traversal parsing module. Among them, the preprocessing and cleaning module performs encoding standardization, noise filtering, DOM structure repair, and dynamic content marking; the target event matching module uses a multi-level matching strategy to accurately match the target event set pre-defined by the user; the time processing module handles the complex time format unique to sports events; and the dual-traversal parsing module determines the maximum broadcast program capacity through the first traversal, constructs the data structure through the second traversal, and captures the broadcast program name through multiple signal sources. Data storage: The parsed structured data is stored through a dynamic column store module and a versioned snapshot library; Anomaly Handling: An anomaly handling mechanism is implemented throughout the entire data acquisition, intelligent parsing, and data storage process, including network request anomaly handling at the data acquisition layer, parsing anomaly handling at the intelligent parsing layer, and storage anomaly handling at the data storage layer.

[0018] Therefore, compared with existing technologies, the sports event broadcast information aggregation system and method based on intelligent parsing proposed in this invention achieves comprehensive optimization of sports event broadcast information collection, processing, and storage through a dual traversal mechanism, dynamic column storage structure, intelligent time parsing algorithm, multi-mode event matching engine, multi-modal broadcast program extractor, and anomaly handling mechanism, and has the following beneficial effects: 1. The system of this invention integrates the full-process automation of data collection, parsing and storage. Users only need to input the target event range (such as time interval, event type) to automatically complete information capture and processing without manual intervention in each step. For example, there is no need to configure crawler parameters, write parsing rules or design storage structure separately. The operation steps are greatly reduced compared with traditional methods.

[0019] 2. The system is developed based on a general programming language and open-source framework, eliminating the need for browser plugins, proxy tools, or third-party parsing libraries. This lowers the technical threshold and device compatibility requirements for users, and it supports direct operation on mainstream operating systems.

[0020] 3. This invention dynamically generates storage columns by counting the maximum number of broadcast programs during the first traversal, avoiding data truncation caused by fixed column design; the second traversal accurately fills the data, combined with a sparse filling strategy (filling insufficient columns with NULL), ensuring the integrity of the matrix structure and greatly improving the data integrity rate.

[0021] 4. This invention achieves intelligent matching of event names through the regular expression combination library (TARGET_COMPETITIONS), supports fuzzy matching and exclusion of secondary leagues, and allows dynamic addition and deletion of specified events, greatly reducing the data misjudgment rate.

[0022] 5. The dynamic columnar storage structure automatically expands columns based on the actual number of broadcast programs, avoiding storage waste caused by a fixed table structure; the versioned snapshot library supports incremental storage (only saving the differences), reducing storage space usage and further reducing hardware investment.

[0023] 6. This invention renders dynamic content (such as AJAX-loaded event data) through a headless browser (Puppeteer), and combines CSS selectors with a hybrid XPath parsing strategy, which can automatically adapt to changes in the target website's DOM structure (such as tag changes and class name modifications), greatly reducing the frequency of parsing rule maintenance.

[0024] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of an embodiment of a sports event broadcast information aggregation system based on intelligent parsing according to the present invention.

[0026] Figure 2 This is a flowchart of the target event matching process in an embodiment of a sports event broadcasting information aggregation system based on intelligent parsing according to the present invention.

[0027] Figure 3 This is a flowchart of time processing in an embodiment of a sports event broadcast information aggregation system based on intelligent parsing according to the present invention.

[0028] Figure 4 This is a flowchart of the double traversal parsing process in an embodiment of a sports event broadcast information aggregation system based on intelligent parsing according to the present invention.

[0029] Figure 5 This is a flowchart of dynamic column storage in an embodiment of a sports event broadcast information aggregation system based on intelligent parsing according to the present invention.

[0030] Figure 6This is a flowchart of the anomaly handling mechanism in an embodiment of a sports event broadcasting information aggregation system based on intelligent parsing according to the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0032] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0033] An Example of a Sports Event Broadcast Information Aggregation System Based on Intelligent Parsing See Figures 1 to 6 This embodiment provides a sports event broadcast information aggregation system based on intelligent parsing, including: The data acquisition layer, including a web crawler engine, a request management module, an anti-crawling bypass module, and a dynamic page rendering module, is used to collect raw HTML data from the Electronic Program Guide preview website. Among them, the web crawler engine manages the URLs of the websites to be crawled, the request management module manages the entire lifecycle of HTTP requests, the anti-crawling bypass module combats anti-crawling technologies, and the dynamic page rendering module processes JavaScript dynamic content.

[0034] The intelligent parsing layer includes a preprocessing and cleaning module, a target event matching module, a time processing module, and a dual-traversal parsing module. These modules intelligently parse the collected raw HTML data. The preprocessing and cleaning module performs encoding standardization, noise filtering, DOM structure repair, and dynamic content tagging. The target event matching module uses a multi-level matching strategy to match a user-defined set of target events, including primary matching using regular expressions, intelligent contextual analysis, and event-level verification. The time processing module handles the complex time formats unique to sports events. The dual-traversal parsing module determines the maximum broadcast program capacity through the first traversal, constructs the data structure through the second traversal, and captures broadcast program names from multiple signal sources.

[0035] The data storage layer includes a dynamic column storage module and a versioned snapshot library, which are used to store the parsed structured data. The dynamic column storage module dynamically generates storage columns based on the maximum number of broadcast programs obtained during the first traversal, and the versioned snapshot library implements versioned storage of the data.

[0036] An exception handling mechanism is implemented throughout the entire data acquisition, intelligent parsing, and data storage process to handle network request exceptions in the data acquisition layer, parsing exceptions in the intelligent parsing layer, and storage exceptions in the data storage layer.

[0037] The web crawler engine is the central control system of the entire collection layer. It manages the URLs (Uniform Resource Locators) of websites to be crawled through a priority queue, matches the best crawling strategy according to the website type, and monitors the operating metrics of each module in real time.

[0038] The request management module manages the entire lifecycle of HTTP (Hypertext Transfer Protocol) requests. It receives request tasks from the web crawler engine, constructs request headers that meet the requirements of the target website, sends requests through a connection pool, and updates the proxy IP (Internet Protocol address) and cookie (data stored on the user's local terminal by the website to identify the user) status based on the response status.

[0039] Among them, the anti-crawling bypass module is used to counter various anti-crawling technologies, such as simulating human operation behavior, using OCR (Optical Character Recognition) to recognize simple CAPTCHAs, using deep learning to recognize complex CAPTCHAs, and using an adaptive request interval algorithm to intelligently wait.

[0040] The dynamic rendering module is the core processor for handling dynamic JavaScript content and can simulate interactions. For example, it can simulate user actions such as scrolling and clicking the "load more" button.

[0041] The interaction between the web crawler engine, request management module, and anti-crawling bypass module can collect raw HTML (Hypertext Transfer Protocol) data and then output it to the intelligent parsing layer.

[0042] In this embodiment, the web crawler engine, acting as the central control system of the data acquisition layer, performs the following: Uniform Resource Locator (URL) used to manage websites to be crawled. The URLs are sorted according to preset priority rules (such as website update frequency, data importance, user-specified priority, etc.) to ensure that URLs of high-priority websites are crawled first. Automatically identify the type of website to be crawled (such as news websites, sports event websites, electronic program guide and preview websites, etc.) and match the best crawling strategy according to the website type; the crawling strategy includes at least request frequency control, data extraction rules, and anti-crawling mechanism response, so as to adapt to the structure and anti-crawling characteristics of different websites and improve the success rate and accuracy of data collection. Real-time monitoring of the operational metrics of each module in the data acquisition layer (such as the request management module, anti-crawling bypass module, dynamic page rendering module, etc.), including but not limited to request success rate, data acquisition volume, processing time, error rate, etc.; dynamic adjustment of resource configuration and working parameters of each module based on monitoring results, such as increasing the connection pool size of the request management module and optimizing the CAPTCHA recognition algorithm of the anti-crawling bypass module, to ensure the stability and efficiency of the data acquisition process.

[0043] In this embodiment, the request management module performs full lifecycle management of HTTP requests, including: Receive requests from the web crawler engine, and parse the target website URL, request type (GET / POST), data extraction rules, and associated proxy IP pool and cookie database configuration in the task; Based on the anti-scraping strategy or verification requirements of the target website, dynamically generate HTTP request headers that conform to the specifications, including but not limited to User-Agent, Referer, and Accept-Language fields, and support adaptive adjustment of request header parameters through configuration files or machine learning models to simulate real user behavior; Maintain a multi-threaded connection pool, dynamically adjust the number of connections according to the concurrency limit of the target website, and prioritize the reuse of idle connections to reduce handshake overhead; when the connection pool is full, temporarily store request tasks in a priority queue to avoid request failure due to resource contention. The system monitors HTTP response status codes in real time. When a 403 (Forbidden) or 429 (Too many requests) error is detected, it automatically switches to a compliant IP from the proxy IP pool and updates the cookie. If the response contains a dynamic token or CAPTCHA link, the anti-scraping bypass module is triggered, and the updated identity credentials are synchronized to the cookie library for reuse in subsequent requests.

[0044] In this embodiment, the anti-crawling bypass module counters various anti-crawling technology defense systems by performing the following: By simulating real user operation patterns such as mouse trajectory, keyboard input rhythm, and page scrolling pauses, it interferes with the behavior analysis and detection of anti-scraping systems; it supports configuring randomized operation intervals and interaction paths to avoid being identified as an automated program due to fixed operation patterns. For simple graphic CAPTCHAs (such as combinations of numbers and letters), after preprocessing (binarization and noise reduction), the character content is extracted using optical character recognition (OCR) technology; for complex CAPTCHAs (such as distorted text, interference lines, and sliding puzzles), a convolutional neural network (CNN) or recurrent neural network (RNN) model is used for training and real-time recognition, supporting dynamic updates of model parameters to adapt to changes in CAPTCHA style; Integrate a third-party CAPTCHA cracking API. When local recognition fails, automatically call the service to complete the verification and return the result to the request management module. Based on the target website’s historical response latency, request success rate, and current system load, dynamically adjust the request sending frequency; use an exponential backoff algorithm to gradually extend the waiting time when rate limiting is triggered to avoid being blocked due to high-frequency requests; Real-time monitoring of HTTP response status codes (such as 403, 429, 503) and anti-scraping warning fields in response headers; when specific defense mechanisms (such as IP blocking, device fingerprint verification) are detected, automatic switching to backup proxy IP pool, modification of request header parameters, or enabling browser automation tools (such as Selenium) to simulate a complete browser environment; Establish data interfaces with the request management module and dynamic page rendering module to synchronize successfully identified verification codes or updated proxy IP / cookie information to the request header in real time; at the same time, receive parsing error signals from the intelligent parsing layer to trigger a retry mechanism or adjust the priority of anti-crawling strategies.

[0045] In this embodiment, the preprocessing and cleaning module performs preliminary cleaning and standardization processing on the raw HTML data, including: Automatically detects the character encoding of the original HTML (such as UTF-8, GBK, ISO-8859-1, etc.) and performs encoding conversion and unification on garbled content; supports encoding conversion via HTTP response headers, <meta> Infer the encoding type from tags or byte stream features and fix text display anomalies (such as garbled Chinese characters) caused by inconsistent encoding. Based on a predefined advertising tag library (such as <iframe>, Identify and remove ad scripts, images, and pop-up code based on DOM node location characteristics; remove developer comments in HTML.<!-- --> Remove redundant line breaks and indentation spaces; mark and isolate JavaScript code blocks while retaining key dynamic content trigger points (such as event listener properties); repair incomplete DOM structures caused by website development errors or anti-scraping mechanisms, and automatically fill in missing nodes through a balanced tag algorithm; mark missing static DOM fragments after dynamic rendering with placeholders and record the missing positions for dynamic content tagging units to supplement. Identify key sports event information (such as scores, match times, and live stream links) dynamically generated by JavaScript in HTML and mark them using the following methods: Attribute markers: Add custom attributes to dynamic nodes (e.g., data-dynamic="true"); Annotation anchoring: Inserting special annotation tags (such as...) before and after dynamic content.<!-- DYNAMIC_CONTENT_START--> (This serves as the basis for analysis and location;) The cleaned HTML is transformed into a standardized DOM tree structure in memory, providing a unified access interface (such as XPath and CSS selectors) for the target event matching module to call; a metadata file containing node cleaning logs is generated.

[0046] As can be seen, the preprocessing module in this embodiment performs preliminary cleaning of the original HTML (Hypertext Transfer Protocol) data: encoding standardization; noise filtering to remove advertisements and comments; DOM (Document Object Model) structure repair; and dynamic content tagging. Through standardization, the preprocessing module transforms the original HTML (Hypertext Transfer Protocol) into a highly structured DOM (Document Object Model) document, providing clean and standardized input for subsequent dynamic parsing and other processes, significantly reducing the complexity of downstream processing, and is a key foundation for the accurate extraction of sports event information.

[0047] In this embodiment, the target event matching module uses a multi-level matching strategy to match the target event set preset by the user, specifically including: Basic regular expression filtering unit: Based on the user's preset target event keyword library (such as league name, team name), it constructs a set of regular expressions (such as (Premier\sLeague|La\sLiga|Bundesliga)) to perform the first round of matching on the input text and quickly exclude irrelevant events (such as training matches, friendly matches). Contextual Intelligent Analysis Unit: Combining the context of the event description text (e.g., "UEFA Champions League Group Stage Round 3"), it identifies key event attributes using a pre-trained NLP model (e.g., BERT fine-tuning) to exclude non-target events (e.g., youth team matches, regional cup competitions); it also performs secondary filtering based on a negative keyword library (e.g., '2', 'B', 'Segunda', 'Second', 'Youth', 'U21'). Example: When the input text contains "Segunda División", it will be automatically excluded and marked as a second-tier league; When the input text contains "Youth League", the youth competition exclusion rule is triggered.

[0048] Event Level Verification Unit: Based on a predefined league priority table (e.g., Premier League > Championship > League One), only top-tier league events are retained, while second-tier and lower-tier leagues are discarded (e.g., when "League One" is detected, it is directly filtered). Structured Information Extraction Unit: For verified event text, use Conditional Random Field (CRF) or rule engine to extract match time (e.g., "2024-08-20 20:00"), participating teams (e.g., "Manchester United vs Liverpool"), and broadcast program (e.g., "Sky Sports Main Event"); compare the extracted match time with the user's preset time range (e.g., only focusing on weekend events) and exclude matches that do not meet the conditions.

[0049] As can be seen, the target event matching module in this embodiment uses a multi-level matching strategy to accurately match the user's pre-defined target event set. First, a primary matching is performed using regular expressions to exclude non-target events. Then, contextual intelligent analysis and event level verification are performed. Events containing characters such as '2', 'B', 'Segunda', 'Second', and 'Youth' are excluded, and the matching results are then subjected to secondary matching. Only the top-tier league is retained, while the second-tier leagues are discarded. After that, the match time, teams, and broadcast programs of the corresponding events are extracted. The target event matching module achieves high accuracy and efficiency in the field of sports event information processing, providing a precise event screening foundation for subsequent broadcaster program information extraction.

[0050] In this embodiment, the time processing module is responsible for processing the complex time formats unique to sports events, including: First, the input string containing dates is cleaned by removing ordinal suffixes such as 1st, 2nd, 3rd, 4th, keeping only Arabic numeral dates. Special separators such as '-' and '–' are replaced. 25 May - 20:00 → 25 May 20:00. Redundant descriptions such as (Cancelled, Postponed) are removed. Then, multi-format parsing is performed, which can parse: Full date and time: "25 May 2025 20:00". Date and time without year: "25 May 20:00". Short format time: "Aug 15 17:30". Plain time: "KO 20:00". Handling cross-year dates: When the event month is less than the current month and no year is specified, the year is incremented by 1. Finally, the normalized time format output is: 2025-05-25 20:00. The time processing module takes into account the characteristics of sports event date formats, combining the base year and the current month to make year-end judgments, thereby improving the accuracy of date parsing.

[0051] In this embodiment, the dual traversal parsing module extracts broadcast program information from the data matched for the target event, including: Using the target event name as the core anchor, the DOM node containing the event is located using XPath or CSS selectors. Then, its child nodes are recursively parsed to identify elements that may contain broadcast information (such as...). , ); Iterate through all located broadcast program elements, count their number, and dynamically update the maximum number of broadcast programs variable (max_broadcast_count), which serves as the only parameter passed between the two stages. If the current page has pagination or asynchronous loading, then load the subsequent content by simulating scroll events or triggering Ajax requests, repeat the location and statistics operations until all relevant nodes are parsed; Based on the max_broadcast_count determined in the first traversal, pre-allocate storage space and initialize the data structures (such as arrays or dictionaries) for broadcast programs 1 to n, each structure containing fields such as date, time, team name, and broadcast program name; Check if the target DOM node is bound to the onmouseover or onclick event, and extract the original broadcast program data by parsing the JavaScript code in the event handler function (such as showBroadcastInfo('program name')). If there is no valid data for the event attributes, information is extracted from the visible text of the node, and key fields can be located through OCR or text layout analysis (such as bolding and color differentiation). The captured original broadcast program names are filtered using chained regular expressions to remove non-core information such as advertising identifiers (e.g., "[Live]"), separators (e.g., "-" "|"), and redundant spaces, while retaining standardized program names (e.g., "CCTV5 Live: Premier League Round 10" → "Premier League Round 10"). The cleaned broadcast program names are associated with and populated with the fields extracted in the first traversal. Cross-field alignment is achieved through timestamps or event IDs to generate a structured dataset (such as JSON or CSV format).

[0052] If information of a broadcast program is missing during the second traversal (such as an empty time field), it is marked as pending completion and an exception log is triggered. At the same time, subsequent programs continue to be processed. When the signal source fails to capture, it automatically switches to the backup parsing strategy to ensure data integrity.

[0053] The generated structured dataset is transmitted to the dynamic column storage module of the data storage layer through a standardized interface, along with parsing logs (such as cleaning rule version and signal source priority) for subsequent auditing and optimization.

[0054] As can be seen, the data after the target event is matched enters the double traversal parsing module to obtain all broadcast program information of the event. The first traversal determines the maximum broadcast program capacity, and the second traversal constructs the data structure. The first traversal first parses the DOM (Document Object Model), locates the node containing the target event name, extracts the broadcast program elements of the event, counts the number of broadcast programs, and updates the maximum broadcast program count variable value. It is determined whether the traversal is complete. If not, it locates the node containing the target event name and continues parsing. After the traversal is completed, it enters the second traversal. The maximum broadcast program count variable value is used as the only parameter passed between the two stages. The second traversal reconstructs the DOM (Document Object Model) structure, locates the node containing the target event name, and creates the data structure of broadcast program 1, broadcast program 2... broadcast program n. Then, the broadcast program name is captured through multiple signal sources: the original broadcast program data is extracted first from the onmouseover event property. onmouseover is an HTML (Hypertext Markup Language) event that is triggered when the mouse pointer moves on the specified HTML (Hypertext Markup Language) element. This event is typically used to execute specific JavaScript code when a user hovers their mouse over an element. An alternative is to scrape visible text while dynamically cleaning the pipeline: using chained regular expressions to filter out useless information such as "-", retaining only the core information of the broadcast program. After obtaining the broadcast program information, the corresponding date, target event name, time, team name, and broadcast program data are filled and aligned, and finally the dataset is output to the data storage layer.

[0055] In this embodiment, the dynamic column storage module implements adaptive storage and version management for the parsed structured data, including: When receiving structured data, the number of data fields is first compared with the number of columns in the current storage table. If a new field is detected (such as a new broadcasting platform identifier or event status label), a new column is dynamically generated and the table structure is updated, while NULL values ​​of historical data are retained and filled. The system automatically infers the data type based on the field content (e.g., converting date fields to DATETIME and numeric fields to INT), and supports overriding default type rules through configuration files.

[0056] Align data with the predefined column mapping table according to field name; if no field matches, trigger the dynamic column expansion process. When the number of data columns is less than the number of columns in the current table, the missing fields are automatically filled with NULL values, and the position of the NULL value is recorded in the metadata log; For frequently repeated fields (such as event type and league name), dictionary encoding is used for compressed storage to reduce disk usage; By using the data write time as the filename prefix (e.g., 20240815_143022_broadcast_data.csv), we ensure that each modification generates an independent version file, thus achieving versioned storage of the versioned snapshot library.

[0057] As can be seen, the parsed structured data enters the data storage layer. The dynamic column storage module first determines whether a new column greater than the current column number has been generated. If so, it automatically expands the column storage. If not, it stores the data according to the actual column number, filling in the data content. When the column number is insufficient, it fills in NULL values. Finally, a unique filename is generated using a timestamp to identify the creation or modification time of the data, thus achieving versioned storage.

[0058] In this embodiment, the exception handling mechanism is not completely independent of any one layer, but rather runs through the entire system to ensure the overall stability of the system. However, the specific implementation and responsibilities of the exception handling mechanism are different at each layer.

[0059] Data Acquisition Layer: Primarily responsible for handling network request exceptions. When timeouts or connection errors occur, it prints "Request failed" and uses a retry mechanism and sets a reasonable timeout period; when anti-scraping measures are implemented, such as IP blocking, it uses IP rotation to simulate human behavior; when verification codes are required, it uses OCR recognition or manual CAPTCHA solving; when JavaScript execution errors or elements are not loaded, it adds exception handling logic to the network request code to ensure timely processing of exceptions.

[0060] Intelligent Parsing Layer: Primarily responsible for handling parsing anomalies. When a date parsing anomaly occurs, time processing is performed to extract the date. If parsing still fails, "Date parsing failed" is printed. When a target event matching anomaly occurs, fuzzy matching of the event name is performed. If parsing still fails, "No matching event data found" is printed. When the number of broadcast programs is 0, "No broadcast information" is printed. When the data structure does not meet expectations, unified adaptation is performed.

[0061] Data storage layer: mainly responsible for handling storage anomalies, automatically cleaning up expired data when storage space is insufficient; converting data when there is a data format conflict; and performing version rollback when data writing fails.

[0062] In addition, there are cross-layer exception handling mechanisms, for example: When the data acquisition layer encounters an unhandled exception, it will pass the exception information to the intelligent parsing layer, which will then decide whether to retry or mark it as abnormal data.

[0063] If the intelligent parsing layer finds an anomaly during the parsing process, it may require the data acquisition layer to re-acquire the data, or mark the data as an anomaly and send it to the storage layer.

[0064] If the data storage layer detects an anomaly during storage (such as a data format error), it will return the data to the intelligent parsing layer for repair.

[0065] Therefore, exception handling is distributed, with each layer having its own specific exception handling logic, and the layers working together to complete exception handling through exception propagation and cooperation.

[0066] An embodiment of a method for aggregating sports event broadcast information based on intelligent parsing. The method is characterized by employing the aforementioned sports event broadcast information aggregation system based on intelligent parsing, and includes the following steps: Data Acquisition: Raw HTML data is collected from the Electronic Program Guide (EPB) website using a web crawler engine, request management module, anti-crawling bypass module, and dynamic page rendering module. Intelligent parsing: The collected raw HTML data is intelligently parsed through a preprocessing and cleaning module, a target event matching module, a time processing module, and a dual-traversal parsing module. Among them, the preprocessing and cleaning module performs encoding standardization, noise filtering, DOM structure repair, and dynamic content marking; the target event matching module uses a multi-level matching strategy to accurately match the target event set pre-defined by the user; the time processing module handles the complex time format unique to sports events; and the dual-traversal parsing module determines the maximum broadcast program capacity through the first traversal, constructs the data structure through the second traversal, and captures the broadcast program name through multiple signal sources. Data storage: The parsed structured data is stored through a dynamic column store module and a versioned snapshot library; Anomaly Handling: An anomaly handling mechanism is implemented throughout the entire data acquisition, intelligent parsing, and data storage process, including network request anomaly handling at the data acquisition layer, parsing anomaly handling at the intelligent parsing layer, and storage anomaly handling at the data storage layer.

[0067] Furthermore, the dual traversal mechanism includes: when parsing the website, a first traversal is performed to count the maximum number of broadcast programs, and then a second traversal is performed to fill the data, ensuring the correctness of dynamic column expansion.

[0068] Furthermore, dynamic column storage includes: dynamically generating storage columns (Program_1, Program_2, ...) based on the max_programs obtained from the first traversal, adapting to changes in the number of broadcast programs for different events, and retaining all broadcast programs. Sparse data padding: automatically padding NULL values ​​for columns with insufficient broadcast program counts to ensure the integrity of the matrix structure.

[0069] Furthermore, the intelligent time parsing algorithm includes: considering the characteristics of sports event date formats, combining the base year and the current month for year-to-year judgment, incorporating context awareness, and introducing a year-to-year correction algorithm: automatically incrementing the base year when the parsed month is less than the current month, improving the accuracy of date parsing. A regular expression configuration cleaning strategy is adopted, compatible with date formats such as 16th May 2025 and May 16.

[0070] Furthermore, the multi-mode event matching engine includes: using a regular expression combination library to intelligently match event names, supporting fuzzy matching and excluding secondary leagues, filtering only specified events, and allowing specified events to be added, deleted, or modified at any time.

[0071] Further, multimodal broadcast program extraction includes: Dual signal source capture: Primarily extracting raw data from the onmouseover event attribute. onmouseover is an HTML (Hypertext Markup Language) event that is triggered when the mouse pointer moves over a specified HTML (Hypertext Markup Language) element. This event is typically used to execute specific JavaScript code when the user hovers the mouse over an element. Alternatively, extracting from visible text. Dynamic pipeline cleaning: Chained regular expression filters remove interfering symbols such as "-", retaining the core program information.

[0072] In summary, the sports event broadcast information aggregation system and method based on intelligent parsing proposed in this invention achieves comprehensive optimization of sports event broadcast information collection, processing and storage through a dual traversal mechanism, dynamic column storage structure, intelligent time parsing algorithm, multi-mode event matching engine, multi-modal broadcast program extractor and anomaly handling mechanism.

[0073] The technical features of the above embodiments can be combined in any way. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0074] The above embodiments are only preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.< / iframe>

Claims

1. An intelligent parsing based sports event broadcast information aggregation system characterized in that, Comprise: A data collection layer including a web crawler engine, a request management module, an anti-crawl bypass module, and a dynamic page rendering module for collecting raw HTML data from an electronic program guide preview website; An intelligent analysis layer including a preprocessing and cleaning module, a target event matching module, a time processing module, and a double traversal analysis module for intelligently analyzing the collected raw HTML data; the target event matching module uses a multi-level matching strategy to match the target event set set by the user in advance, including using regular expressions for primary matching, context intelligent analysis, and event level verification; the time processing module is used to process the complex time format specific to sports events; the double traversal analysis module determines the maximum broadcast program capacity through the first traversal, constructs the data structure through the second traversal, and captures the broadcast program name through multiple signal sources; A data storage layer including a dynamic column storage module and a versioned snapshot library for storing the structured data after analysis; An exception handling mechanism for handling network request exceptions of the data collection layer, analysis exceptions of the intelligent analysis layer, and storage exceptions of the data storage layer.

2. The system of claim 1, wherein, The network crawler engine as the central control system of the data collection layer performs the following: A uniform resource locator (URL) for managing websites to be crawled is sorted according to the preset priority rules to ensure that the URLs of high-priority websites are crawled first; Automatically identify the type of website to be crawled and match the best crawling strategy according to the website type; the crawling strategy at least includes request frequency control, data extraction rule, and anti-crawl mechanism response to adapt to the structure and anti-crawl characteristics of different websites; Real-time monitoring of the running indicators of each module in the data collection layer; dynamically adjust the resource configuration and working parameters of each module according to the monitoring results.

3. The system of claim 1, wherein, The request management module performs full-life-cycle management of HTTP requests, including: Receiving request tasks issued by the network crawler engine, analyzing the target website URL, request type, data extraction rule, and associated proxy IP pool and Cookie library configuration in the task; According to the anti-crawl strategy or verification requirements of the target website, dynamically generate HTTP request headers that meet the specifications, and support adaptive adjustment of request header parameters through configuration files or machine learning models to simulate real user behavior; Maintain a multi-threaded connection pool, dynamically adjust the number of connections according to the concurrency limit of the target website, and preferentially reuse idle connections to reduce handshake overhead; when the connection pool is full, store the request tasks in a priority queue to avoid request failures due to resource competition; Real-time monitoring of HTTP response status codes, automatically switching to a compliant IP from the proxy IP pool and updating the Cookie when access is prohibited or requests are too frequent; if the response contains a dynamic Token or verification code link, trigger the anti-crawl bypass module to handle it, and synchronize the updated identity credentials to the Cookie library for subsequent request reuse.

4. The system of claim 1, wherein, The anti-crawl bypass module is a defense system against various anti-crawl technologies, including: Through real user operation mode, interfere with the behavior analysis detection of the anti-crawler system; After preprocessing simple CAPTCHA, the character content is extracted by optical character recognition (OCR) technology. For complex CAPTCHA, convolutional neural network (CNN) or recurrent neural network (RNN) model is used for training and real-time recognition, and model parameters are dynamically updated to adapt to the change of CAPTCHA style. Third-party CAPTCHA cracking API is integrated, and when local recognition fails, the service is automatically called to complete verification, and the result is fed back to the request management module. According to the historical response delay, request success rate and current system load of the target website, the request sending frequency is dynamically adjusted, and the exponential backoff algorithm is used to gradually extend the waiting time when flow limiting is triggered. Real-time monitoring of HTTP response status code and anti-crawl prompt field in response header, when detecting specific defense mechanism, automatically switch to backup proxy IP pool, modify request header parameters or enable browser automation tool to simulate complete browser environment. Data interface is established with request management module and dynamic page rendering module, and the recognized CAPTCHA or updated proxy IP / Cookie information is synchronized to the request header in real time. At the same time, the analysis exception signal fed back by the intelligent analysis layer triggers the retry mechanism or adjusts the priority of anti-crawl strategy.

5. The system of claim 4, wherein, The preprocessing cleaning module performs preliminary cleaning and standardization processing on the original HTML data, including: Automatic detection of the character encoding of the original HTML, encoding conversion and unification of garbled content; support through HTTP response header, <meta> Tag or byte stream feature inference encoding type, and repair the text display abnormal caused by inconsistent encoding; Based on the pre-defined advertisement tag library or DOM node position characteristics, identify and delete advertisement scripts, pictures and pop-up window codes; remove developer comments, redundant line breaks and indentation spaces in HTML; mark and isolate JavaScript code blocks while retaining key dynamic content triggers; repair incomplete DOM structure caused by website development errors or anti-crawl mechanisms, and automatically complete missing nodes through balance tag algorithm; mark placeholders for missing static DOM fragments after dynamic rendering, and record the missing position for dynamic content marking unit to supplement; Identify key sports event information generated by JavaScript in HTML, and mark by the following methods: Attribute marking: add custom attributes to dynamic nodes; Comment anchoring: insert special comment tags before and after dynamic content as analysis positioning basis; Convert the cleaned HTML into a standardized DOM tree structure in memory, providing a unified access interface for the target event matching module to call; generate metadata files containing node cleaning logs.

6. The system of claim 1, wherein, The double traversal analysis module extracts the broadcast program information from the data matched with the target event, including: Based on the target event name as the core anchor point, locate the DOM node containing the event through XPath or CSS selector, recursively analyze the elements in its child nodes that may contain broadcast program information; Traverse all located broadcast program elements, count their number and dynamically update the maximum broadcast program number variable value, which is used as the only transmission parameter between two stages; If the current page has pagination or asynchronous loading, simulate scroll events or trigger Ajax request to load subsequent content, repeat the positioning and statistical operation until all related nodes are parsed. According to the max_broadcast_count determined by the first traversal, pre-allocate storage space and initialize the data structure of the broadcast program 1 to the broadcast program n; Check whether the target DOM node is bound to the onmouseover or onclick event, and extract the original broadcast program data by parsing the JavaScript code in the event handling function; If some broadcast program information is missing in the second traversal, mark it as a to-be-completed state and trigger an exception log, and continue processing the subsequent programs; when the signal source capture fails, automatically switch to a backup analysis strategy to ensure data integrity. The generated structured data set is transmitted to the dynamic column storage module of the data storage layer through a standardized interface.

7. The system of claim 6, wherein: When checking whether the target DOM node is bound to the onmouseover or onclick event, if the event attribute has no valid data, extract information from the visible text of the node, and support positioning key fields through OCR or text layout analysis; Perform chain regular filtering on the captured original broadcast program name, sequentially remove non-core information such as advertisement identifiers, separators, and redundant spaces, and retain the standardized program name; Fill in the cleaned broadcast program name and the fields extracted in the first traversal, align the fields through timestamps or event IDs, and generate a structured data set.

8. The system according to any one of claims 1 to 6, characterized in that The dynamic column storage module implements adaptive storage and version management for the parsed structured data, including: When receiving structured data, first compare the number of data fields with the current number of storage table columns; if a new field is detected, dynamically generate a new column and update the table structure, while retaining the NULL value filling of historical data; Automatically infer the data type according to the field content, and support overriding the default type rules through a configuration file. Align the data according to the field name and the pre-defined column mapping table, and trigger the dynamic column expansion process for unmatched fields; When the number of data columns is less than the current number of table columns, automatically fill in NULL values for missing fields and record the NULL value positions to the metadata log; Use the data write time as the file name prefix to ensure that each modification generates an independent version file, and implement versioned storage of the versioned snapshot library.

9. The system according to any one of claims 1 to 6, characterized in that, The target event matching module uses a multi-level matching strategy to match the target event set preset by the user, specifically including: Primary regular filtering unit: based on the user's preset target event keyword library, construct a regular expression set, and perform the first round of matching on the input text to quickly exclude irrelevant events; Contextual intelligent analysis unit: combined with the context of the event description text, identify the key attributes of the event through a pre-trained NLP model to exclude non-target events; based on a negative keyword library for secondary screening; Event level verification unit: according to the pre-defined league priority table, only keep the top league events and discard the secondary and lower leagues; Structured information extraction unit: for the verified event text, use conditional random fields or rule engines to extract match time, participating teams, and broadcast programs; compare the extracted match time with the user's preset time range to exclude non-eligible sessions.

10. A smart parsing-based sports event broadcast information aggregation method, characterized in that, The method adopts the intelligent analysis-based sports event broadcast information aggregation system according to any one of claims 1 to 9, and comprises the following steps: Data collection: collecting original HTML data from an electronic program guide preview website through a network crawler engine, a request management module, an anti-crawling bypass module and a dynamic page rendering module; Intelligent analysis: intelligently analyzing the collected original HTML data through a preprocessing and cleaning module, a target event matching module, a time processing module and a double-traversal analysis module; wherein the preprocessing and cleaning module performs encoding standardization, noise filtering, DOM structure repair and dynamic content marking, the target event matching module uses a multi-level matching strategy to accurately match a target event set preset by a user, the time processing module processes a complex time format specific to sports events, and the double-traversal analysis module determines a maximum broadcast program capacity through a first traversal, constructs a data structure through a second traversal, and captures a broadcast program name through multiple signal sources; Data storage: storing the analyzed structured data through a dynamic column storage module and a versioned snapshot library; Exception handling: implementing an exception handling mechanism during the entire data collection, intelligent analysis and data storage process, including network request exception handling at the data collection layer, analysis exception handling at the intelligent analysis layer and storage exception handling at the data storage layer.

Images