An automatic collection and fusion application method and system of internet multi-source precipitation data

Abstract

This invention proposes an automatic acquisition and fusion application method and system for multi-source precipitation data from the Internet. The method includes crawling precipitation data from target websites and decrypting encrypted webpage data, performing preprocessing and key information extraction on the raw data, storing the data, removing abnormal data through quality verification, fusing data from the same source, and providing a standard data service interface. The system includes a data crawling module, a data preprocessing module, a data storage and quality control module, and a data service interface module. This method and system can effectively overcome font encryption anti-crawling mechanisms, improve the integrity of precipitation data acquisition, extraction accuracy, and quality reliability, reduce manual intervention, and achieve integrated automated processing from acquisition to service. It can be widely applied in meteorological and hydrological monitoring, flood control and disaster reduction, smart city management, and smart agriculture.

Description

Technical Field

[0001] This invention relates to the fields of meteorological data processing, Internet data acquisition and fusion technology, and more specifically, to a method and system for the automatic acquisition and fusion of multi-source precipitation data from the Internet. Background Technology

[0002] In the fields of meteorological monitoring, disaster early warning and smart meteorological services, the use of multi-source data acquisition, fusion and quality control technologies to obtain continuous and stable precipitation data products has been widely applied in scenarios such as intelligent grid forecasting algorithm development and meteorological big data services.

[0003] However, existing technologies for precipitation data acquisition still have many shortcomings. In the data acquisition stage, some target websites employ complex anti-scraping techniques such as font library encryption, making it difficult for conventional crawlers to accurately obtain encrypted precipitation-related data. Furthermore, existing crawler frameworks lack adaptability to the data loading characteristics of different websites, affecting the timeliness and completeness of data acquisition. In the data preprocessing stage, precipitation data is often disorganized and contains a large amount of unstructured information. Existing information extraction tools lack targeted adaptation designs for precipitation data scenarios, making it difficult to accurately extract key information such as precipitation amount, precipitation region, and precipitation time. In the data fusion and quality control stage, existing technologies lack effective methods for screening homogeneous datasets, and the anomaly identification effect of cross-validation of multi-source data is poor. Moreover, there is no reverse guidance mechanism for data acquisition and preprocessing rules in the quality control stage, making it impossible to optimize the front-end process based on data quality issues or new data source requirements, thus affecting the accuracy and service efficiency of the final data product.

[0004] Therefore, there is an urgent need for an automatic acquisition and fusion application solution for multi-source precipitation data from the Internet. This solution should be able to automatically acquire multi-source precipitation data, effectively decrypt encrypted data, efficiently complete data preprocessing and key information extraction, accurately filter datasets from the same source and remove abnormal data, embed quality control steps into the process, and provide reverse guidance for optimizing crawling and preprocessing rules. Ultimately, it should provide standardized data services and solve the problems existing in current technologies regarding the effectiveness of precipitation data acquisition, the accuracy of information extraction, data quality control, and process optimization. Summary of the Invention

[0005] Based on existing technology, the objective of this invention is to provide an automatic acquisition and fusion application method and system for multi-source precipitation data from the Internet. Through a modular, highly cohesive, and loosely coupled system architecture, it can achieve integrated automation from data acquisition to service provision of multi-source precipitation data, ensuring the continuous reliability of output data quality.

[0006] According to the present invention, the above-mentioned task is solved by an automatic acquisition and fusion application method and system for multi-source precipitation data from the Internet.

[0007] In a first aspect, the present invention proposes an automatic acquisition and fusion application method for multi-source precipitation data from the Internet, the method comprising: It crawls precipitation data from target websites, and for encrypted web page data, it provides a text decryption algorithm to decrypt the encrypted text and obtain the original data. Preprocessing and data extraction are performed on the raw data to obtain a multi-source precipitation dataset; Store the multi-source precipitation dataset, identify and remove outlier data from the multi-source precipitation dataset, and filter and fuse precipitation data from the same source; and A standard data service interface is provided, through which precipitation data services are provided.

[0008] Furthermore, the precipitation data crawled from the target website includes: To address the data loading characteristics of the target website, a customized crawler framework was adopted and adapted to automatically and in real-time retrieve data from the target website through dynamic webpage parsing.

[0009] Furthermore, when the encryption format is fixed-glyph encryption, the text decryption algorithm includes: Detecting the font features of the target website includes: The system detects whether the target website's HTML contains the `@font-face` rule and links to font library files; and Detect whether the text content of the target website contains a large number of uncommon Unicode characters; Use a font parsing tool to parse the font library file; Establishing font mapping relationships includes: Manually establish mapping relationships between glyph names and actual characters; and Based on the font library file, a glyph image of each character is automatically generated, and the correspondence between the glyph name and the glyph outline is established through OCR recognition; Extract the cmap table from the font library file, and establish a mapping relationship between the glyph names and actual numbers / characters through glyph feature matching analysis, thereby providing a mapping dictionary from garbled Unicode to plaintext; and Based on the mapping dictionary, replace the garbled Unicode characters in the target website with real text.

[0010] Furthermore, when the encryption format is dynamic glyph encryption, the text decryption algorithm includes: Each time a target webpage is accessed, the part that needs to be crawled is automatically collected visually, and the text format and / or content are obtained through the layout analysis module and OCR recognition.

[0011] Furthermore, the data preprocessing includes: Use a data processing library to remove completely duplicate data from the original data; Unify the data format of the original data and remove abnormal characters from the data; and Extracting numbers and key information from text-based data using a UIE-based fine-tuning model.

[0012] Furthermore, the cross-validation includes: Cross-validation is performed on the precipitation datasets from different sources. If data conflicts are found, the conflicting data is corrected or removed.

[0013] Furthermore, the accuracy of the cross-validation is evaluated, and when the accuracy of the cross-validation is lower than a preset threshold, the following countermeasures are taken: Automatically trigger the alarm mechanism to notify the data quality manager, locate the problem, and label the relevant data as pending verification. Before the quality verification of this part of the data is completed, use it cautiously or suspend its application in critical business. Initiate a manual intervention process to correct, remove, or label the affected data, while simultaneously tracing and correcting upstream issues; and When necessary, the threshold for cross-validation accuracy can be dynamically adjusted to ensure that the threshold setting is adapted to the actual application scenario.

[0014] Furthermore, the multi-source precipitation datasets with similar origins are filtered and merged by calculating data matching degree. The data matching degree is calculated using a five-dimensional cosine similarity algorithm, and the five-dimensional metadata of the algorithm includes: name; summary; Time resolution; Spatial resolution; and Observation frequency.

[0015] Furthermore, the formula for calculating the five-dimensional cosine similarity is: in For the i-th dimension of the precipitation dataset, a numerical vector is generated. Let be the dimension weights, and ∑ =1; When the data matching degree between any two of the multi-source precipitation datasets is greater than a preset threshold, they are determined to be from the same source, and data fusion is performed.

[0016] A second aspect of the present invention proposes an automatic acquisition and fusion application system for multi-source precipitation data from the Internet, wherein the modules of the system interact directly through a standardized data format, and the system includes: A data crawling module is configured to automatically collect precipitation data and provide raw data. The data crawling module includes a decryption algorithm module, which is configured to provide a text decryption algorithm to parse and extract encrypted text. A data preprocessing module is configured to remove duplicate data and abnormal content from the raw data and convert the raw data into a standardized format. The data storage and quality control module is configured to store precipitation data in a standardized format, remove outlier data through cross-validation, calculate the data matching degree of the dataset using a five-dimensional cosine similarity algorithm, thereby filtering precipitation data from the same source and performing data fusion; and The data service interface module is configured to use a standard data service interface to provide data support for external data products.

[0017] The present invention proposes an automatic acquisition and fusion application method and system for multi-source precipitation data from the Internet, which has at least the following beneficial effects: (1) The method and system proposed in this invention effectively overcome anti-crawling mechanisms such as font encryption, and achieve stable acquisition of multi-source precipitation-related data. Through the targeted adaptation information extraction model (UIE fine-tuning model), the accuracy and efficiency of extracting key information from messy text in multi-source precipitation data are improved. At the same time, the consistency and reliability of centralized data in multi-source precipitation data are ensured by multi-dimensional homogeneous screening and multi-source cross-validation.

[0018] (2) The system proposed in this invention relies on a modular architecture with "high cohesion and low coupling" and standardized data service interfaces to achieve convenient integration with external systems. Each module in the system has a single responsibility and highly centralized internal processing logic. The modules interact with each other through standardized data formats, which facilitates system maintenance and upgrades.

[0019] (3) The method and system proposed in this invention do not require manual intervention throughout the entire process from data collection to service provision. Through timed task triggering and automatic scheduling, the manual cost and operational errors are greatly reduced.

[0020] In summary, the method and system proposed in this invention can effectively crack webpage font encryption anti-crawling mechanisms, achieve automatic collection and accurate decryption of multi-source heterogeneous precipitation data, and improve the integrity, consistency, and accuracy of precipitation data through data cleaning, fusion, and cross-validation processes. This reduces manual intervention and lowers data processing costs, and the generated high-resolution long-sequence precipitation data products can meet the application needs of different scenarios. The method and system proposed in this invention can fully meet the needs of meteorological and hydrological monitoring for acquiring accurate precipitation data, providing data support for disaster assessment and early warning information dissemination in flood control and disaster reduction work, scientific decision-making in smart city management, and irrigation regulation in smart agriculture scenarios. Attached Figure Description

[0021] To further illustrate the advantages and other features of the various embodiments of the present invention, a more specific description of the embodiments of the present invention will be presented with reference to the accompanying drawings. It is understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by the same or similar reference numerals for clarity.

[0022] Figure 1 A flowchart illustrating the method of the present invention is shown.

[0023] Figure 2 The crawler configuration page of this invention is shown.

[0024] Figure 3 The crawler task statistics page of this invention is shown.

[0025] Figure 4 A schematic diagram of a font encryption anti-scraping mechanism is shown.

[0026] Figure 5 This shows a diagram of a TTF file in a webpage.

[0027] Figure 6 This diagram illustrates how to manually establish a mapping relationship between glyph names and characters.

[0028] Figure 7 A visual viewing interface for the target glyph is shown.

[0029] Figure 8 A real-time recognition scheme for dynamic character encryption is shown.

[0030] Figure 9 This shows a comparison between the source website information and the mapped information.

[0031] Figure 10 A flowchart illustrating the fine-tuning process of the UIE fine-tuning model of the present invention is shown. Detailed Implementation

[0032] It should be noted that the components in the various figures may be shown exaggeratedly for illustrative purposes and are not necessarily to scale. In each figure, the same reference numerals are used for components that are identical or have the same function.

[0033] In this invention, the various embodiments are merely intended to illustrate the solutions of the invention and should not be construed as limiting.

[0034] In this invention, unless otherwise specified, the quantifiers “a” and “one” do not exclude scenarios involving multiple elements.

[0035] It should also be noted that, in the embodiments of the present invention, only a portion of the components or parts may be shown for clarity and simplicity. However, those skilled in the art will understand that, under the teachings of the present invention, necessary components or parts can be added as needed for specific scenarios. Furthermore, unless otherwise stated, features in different embodiments of the present invention can be combined with each other. For example, a feature in the second embodiment can replace a corresponding or functionally identical or similar feature in the first embodiment, and the resulting embodiment will also fall within the scope of disclosure or description of this application.

[0036] It should also be noted that within the scope of this invention, the terms "same", "equal", and "equal to" do not mean that the two values ​​are absolutely equal, but allow for a certain reasonable error. In other words, the terms also cover "substantially the same", "substantially equal", and "substantially equal to".

[0037] In this invention, the modules of the system according to the invention can be implemented using software, hardware, firmware, or a combination thereof. When a module is implemented using software, its function can be implemented through computer program flow. For example, the module can be implemented using code segments (such as code segments in languages ​​like C and C++) stored in a storage device (such as a hard disk, memory, etc.), wherein the corresponding function of the module can be implemented when the code segment is executed by a processor. When a module is implemented using hardware, its function can be implemented by setting a corresponding hardware structure. For example, the module's function can be implemented by hardware programming a programmable device such as a field-programmable gate array (FPGA), or by designing an application-specific integrated circuit (ASIC) that includes multiple transistors, resistors, capacitors, and other electronic devices. When a module is implemented using firmware, the module's function can be written into a read-only memory such as an EPROM or EEPROM in the form of program code, and the corresponding function of the module can be implemented when the program code is executed by a processor. In addition, some functions of the module may need to be implemented by separate hardware or by working in cooperation with the hardware. For example, the detection function is implemented by a corresponding sensor (such as a proximity sensor, accelerometer, gyroscope, etc.), the signal transmission function is implemented by a corresponding communication device (such as a Bluetooth device, infrared communication device, baseband communication device, Wi-Fi communication device, etc.), the output function is implemented by a corresponding output device (such as a display, speaker, etc.), and so on.

[0038] Furthermore, the steps of the methods of the present invention are not limited in terms of the execution order of the method steps. Unless otherwise specified, the method steps may be executed in different orders.

[0039] To address the shortcomings of existing technologies in terms of the effectiveness of precipitation data acquisition, the accuracy of information extraction, data quality control, and process optimization, this invention proposes an automatic acquisition and fusion application method and system for multi-source precipitation data from the Internet. This system enables efficient and automated data crawling and processing of multi-source precipitation data, providing high-quality, multi-source precipitation data for applications.

[0040] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0041] Figure 1 A flowchart of the method of the present invention is shown.

[0042] This invention proposes an automatic acquisition and fusion application method for multi-source precipitation data from the Internet, the method comprising: Step S100, Webpage Data Crawling and Identification: Crawling precipitation data from the target website, including providing a text decryption algorithm for encrypted webpage data to decrypt the encrypted text and obtain the original data; Step S200, Data Preprocessing and Extraction: Perform preprocessing and extraction on the raw data to obtain a multi-source precipitation dataset.

[0043] Step S300, Data Storage and Verification Fusion: Store the multi-source precipitation dataset, perform cross-validation on the multi-source precipitation dataset, identify and remove abnormal data records, and use a multi-dimensional similarity algorithm to calculate the data matching degree of the multi-source precipitation dataset, and filter and fuse precipitation data from the same source.

[0044] Step S400, Service Interface Construction and Application: Provide a standard data service interface, and provide precipitation data services through the standard data service interface.

[0045] In one embodiment of the present invention, step S100, "encrypted" webpage data crawling and identification, includes: automatically accessing and crawling multi-source precipitation data such as global precipitation data, disaster data, and precipitation images from a specified website using a programming script. For dynamically loaded precipitation data, it is obtained by simulating browser behavior or calling API interfaces; for webpage content using encryption methods, a text decryption algorithm is constructed to parse font files and establish mapping relationships to locate and extract target fields.

[0046] Figure 2 and Figure 3 The crawler configuration page and crawler task statistics page of the present invention are shown. In one embodiment of the present invention, such as Figure 2 and Figure 3 As shown, a customized Python web crawler framework is used and adapted to the data loading characteristics of the target website. Through dynamic webpage parsing technology, data from the specified website is automatically fetched and retrieved in real time at regular intervals. The crawler simulates user requests to crawl dynamically loaded data.

[0047] In one embodiment of the invention, the crawler periodically visits the target website at set time intervals (e.g., hourly or daily) and attempts to extract precipitation data from the webpage.

[0048] In one embodiment of the present invention, if the target website is a simple website without anti-crawler mechanisms, the Requests library is used first for data requests.

[0049] In this invention, the Requests library is an open-source HTTP request utility library within the Python programming language ecosystem. Its core function is to simplify the HTTP / HTTPS protocol interaction process between the client and the target server. It supports quick invocation of various HTTP request methods through a concise application programming interface (API), allowing for customized request headers, maintenance of cookie states, and parsing of response content. This eliminates the need to write complex underlying network protocol interaction code, enabling efficient resource requests and response data reception from the target server. In one embodiment of this invention, the lightweight nature of the Requests library can reduce the system resource consumption of the crawler framework while ensuring the integrity of precipitation data acquisition, thereby improving data collection efficiency in simple scenarios.

[0050] In one embodiment of the present invention, for data on encrypted web pages, the method proposed in this invention establishes an automated processing flow through anti-crawler strategies, ensuring the stable and efficient operation of the data collection task.

[0051] Figure 4 A schematic diagram of a font encryption anti-scraping mechanism is shown. (For example...) Figure 4 As shown, in one embodiment of the present invention, the data on the target website employs data obfuscation technology (text encryption). Users see clear and correct text in a normal browser, while simple web crawlers directly capture garbled text or meaningless characters. Furthermore, each time the page is reloaded, the same text will be reloaded with garbled content. Font encryption anti-crawling mechanisms typically use multiple font encoding libraries to dynamically encrypt text on web pages. These are mainly divided into fixed font encryption (font name and font shape correspond one-to-one, only the encoding changes) and dynamic font encryption (font name, font shape, and encoding all change randomly).

[0052] In one embodiment of the present invention, a text decryption algorithm is provided, which can parse font files, establish font mapping relationships, restore the crawled garbled data into regular plaintext data, and achieve lossless data acquisition.

[0053] In one embodiment of the present invention, when the text encryption is a fixed font encryption, the text decryption algorithm includes: (1)Font feature detection: After the crawler obtains the complete HTML source code of the target web page, it first locates all CSS style-related areas in the source code, including inline styles, external styles, and inline styles. Traverse all CSS style contents and retrieve whether they contain the @font-face core rule (i.e., the exclusive rule for customizing fonts in CSS). If the @font-face rule is detected, further parse the src attribute within the rule to extract the link address pointing to the font file, and filter out the font file links (e.g., font files with.woff or.ttf suffixes). Optionally, verify whether the extracted font file links are accessible to exclude interference from invalid or expired font file links. At the same time, strip the tag content from the web page HTML source code and extract the plain text data. Traverse the Unicode encoding of each character in the plain text and filter out the characters belonging to the non-common range. The non-common range includes the Unicode private use area (encoding range E000-F8FF), the unassigned encoding area, and the non-universal Chinese / number encoding area. Calculate the proportion of non-common Unicode characters in the total number of characters in the text. If the proportion reaches the preset threshold, it is determined that the text contains a large number of non-common Unicode characters.

[0054] (2)Font file parsing: Use the fontTools library (Python) to parse the downloaded font file.

[0055] (3)Establishing font mapping relationships: There are two methods, manual creation and automatic retrieval, where: Manually create the mapping relationship by observing and analyzing with a professional font editing tool FontCreator, and manually establish the mapping relationship from the glyph name to the text. For example, as Figure 6 shown, the glyph name "glyph46" corresponds to the character "东".

[0056] Automatically retrieve the mapping relationship by automatically generating a glyph image for each character based on the font library, and establishing the corresponding relationship between the glyph name and the glyph in the font library through OCR recognition.

[0057] (4)Extracting the intermediate mapping: During crawling, synchronously capture the font library file (.ttf file), extract the cmap table of the font (the mapping from character encoding to glyph outline), and obtain the mapping from scrambled Unicode to glyph name.

[0058] Figure 7 Shows the visual viewing interface of the target glyph. As Figure 7 shown, this interface includes the glyph name, outline features, encoding, and attribute information, and supports the detailed viewing and feature analysis of the glyph. In an embodiment of the present invention, through glyph feature matching analysis, establish the mapping relationship between the glyph name and the real text, and construct a mapping dictionary from scrambled text to plain text.

[0059] In one embodiment of the present invention, the accuracy of the mapping can be verified by multiple font files. After the mapping dictionary is constructed, it can be reused directly without re-establishing the mapping relationship.

[0060] Figure 8 A real-time recognition scheme for dynamic character encryption is shown.

[0061] In one embodiment of the present invention, when the text encryption is dynamic font encryption, a real-time recognition scheme is provided: each time the target webpage is accessed, the part that needs to be crawled is automatically captured, and the text format and content are obtained by recognizing the image through the layout analysis module and OCR.

[0062] Figure 9 This shows a comparison between the source website information and the mapped information. For example... Figure 9 As shown, the text decryption algorithm proposed in this embodiment of the invention can accurately extract text information from the target webpage, and map the garbled Unicode characters in the HTML to replace them with real text.

[0063] In one embodiment of the present invention, the data preprocessing and extraction in step S200 includes data cleaning and data extraction.

[0064] In one embodiment of the present invention, the data cleaning includes: Data cleansing involves using a data processing library to perform preliminary cleaning of the crawled raw data, removing completely duplicate records; and Data format cleaning involves standardizing non-compliant data formats and / or data content in the original data. Specifically, the focus is on unifying the display formats of time, date, and numerical values, and removing abnormal characters from the data content.

[0065] In one embodiment of the present invention, the deduplication cleaning and the data format cleaning are implemented by writing corresponding deduplication cleaning rules and data format cleaning rules.

[0066] In one embodiment of the present invention, the data extraction includes: For text-based meteorological data, a fine-tuned model based on the Universal Information Extraction Framework (UIE) is used to accurately extract numbers and key information from messy text.

[0067] Figure 10 A flowchart illustrating the fine-tuning process of the UIE fine-tuning model of the present invention is shown.

[0068] In one embodiment of the present invention, the UIE model is fine-tuned to achieve efficient and accurate extraction of multi-source precipitation data. Specifically, the fine-tuning process includes the following steps: Data collection involves acquiring multi-source raw precipitation data required for training the fine-tuning model. Data annotation: The multi-source raw precipitation data are manually annotated, and the annotated sample data are divided into training set, validation set and test set in a ratio of 8:1:1. Model training involves offline learning and parameter optimization of the fine-tuned model using training and validation sets. By iteratively adjusting the structure and hyperparameters of the fine-tuned model, the generalization ability and prediction accuracy of the fine-tuned model on the target task are improved. Evaluation and optimization: Use independent test sets to verify the effect of the trained model, such as the accuracy of garbled character decryption and the completeness of precipitation information extraction. If the model performance does not meet the preset standard, iteratively adjust the model parameters and supplement labeled samples based on the test results, and repeatedly optimize the model until the performance meets the standard. The optimized model is then encapsulated into a callable functional module and deployed to the multi-source precipitation data acquisition system proposed in this invention.

[0069] In one embodiment of the present invention, the effective data after data cleaning and extraction are uniformly packaged into a standardized JSON format to complete the standardization and preliminary structuring of precipitation data, which facilitates the extraction of key information in subsequent stages.

[0070] In one embodiment of the present invention, the data storage and verification fusion described in step S300 includes data storage, cross-validation, and data fusion of multi-source precipitation data.

[0071] In one embodiment of the present invention, the data storage step includes: The preprocessed and standardized data from step S200 are stored in a relational database as a dataset to obtain a multi-source precipitation dataset.

[0072] In one embodiment of the present invention, the cross-validation step includes: When multiple precipitation datasets come from different sources, cross-correlation verification is first performed on the multi-source precipitation datasets. If the analysis finds that they conflict with each other, the contradictory data is corrected or removed.

[0073] The algorithm system for cleaning multi-source precipitation data is abstracted into a binary classification model. Based on the quality status of precipitation data records, a binary category label is set: Category 1 (positive class) represents "clean data", that is, precipitation data that meets the preset quality standards; Category 0 (negative class) represents "dirty data", that is, precipitation data with problems such as numerical deviation, spatiotemporal misalignment, and attribute contradiction.

[0074] In one embodiment of the present invention, the results of cross-validation are verified by algorithm comparison or manual verification, and the accuracy A of cross-validation is calculated as follows: A = (TP + TN) / (TP + TN + FP + FN) × 100% in, TP (True Positive): Precipitation data that the cleaning algorithm classifies as Category 1 and that actually has no quality issues; TN (True Negative): Precipitation data that the cleaning algorithm classifies as category 0 and actually has quality problems (e.g., the algorithm identifies a record value that exceeds a reasonable threshold, and verification confirms that it was caused by a transmission error). FP (False Positive): Precipitation data that the cleaning algorithm mistakenly classifies as category 1 but is actually category 0 (e.g., the algorithm fails to identify a spatiotemporal misalignment of a record, and upon review, it is found that the geographic unit labeling is incorrect). FN (False Negative): Precipitation records that were mistakenly classified as category 0 by the cleaning algorithm but were actually category 1 (e.g., the algorithm mistakenly judged the data as abnormal due to small fluctuations in the values, but the verification confirmed that the fluctuations were within the natural and reasonable range).

[0075] In one embodiment of the present invention, the threshold for the accuracy A of cross-validation is set to 95%. When the accuracy A of cross-validation is less than the set threshold of 95%, the following countermeasures are taken: (1) Automated early warning and preliminary diagnosis The system automatically triggers an alarm mechanism to notify the data quality manager and simultaneously initiate the problem localization process. Through data analysis, it clarifies whether the decline in accuracy is concentrated in a specific time period, specific region, or specific data source. Subsequently, the data corresponding to the affected time period, region, or data source is labeled as pending verification. Before the quality verification of this part of the data is completed, its application in critical business should be used with caution or suspended to avoid low-quality data affecting subsequent processing results.

[0076] (2) Manual intervention and deep treatment After the system performs an initial diagnosis of the problem, a manual intervention process is initiated, where experts use their professional experience to directly correct, remove, or label the affected data. Simultaneously, the source of the affected data is traced. If the accuracy decline is determined to stem from upstream issues such as rain gauge malfunction or radar calibration anomalies, the relevant maintenance unit is promptly contacted to conduct on-site repairs or data reprocessing. Furthermore, based on the specific circumstances of the event, it is assessed whether it is an isolated extreme event. If necessary, the cross-validation accuracy threshold is dynamically adjusted to ensure that the threshold setting is adapted to the actual application scenario.

[0077] In one embodiment of the present invention, a tiered response process from automatic alarm to human expert intervention is established for situations where the accuracy rate is lower than a preset threshold. This ensures that data problems can be detected, located, and properly handled in a timely manner, and continuously optimizes the entire data production and cleaning process, ultimately providing a stable and reliable data foundation for downstream applications.

[0078] In one embodiment of the present invention, the data fusion includes: Constructing an Internet-oriented multi-source precipitation fusion framework: Integrating multi-source precipitation meteorological datasets from the global Internet, such as the Global Precipitation and Climate Centre (GPCC), the European Meteorological Satellite Organization (EUMETSAT), and the Copernicus Climate Change Service (C3S), and adopting a two-level fusion of "data layer-feature layer" to combine heterogeneous precipitation information into a consistent, high-resolution, long-sequence precipitation data product, thereby enhancing the value of data application.

[0079] In one embodiment of the present invention, a multi-dimensional similarity algorithm is used to calculate the data matching degree of the multi-source precipitation dataset, and to filter and merge precipitation datasets from the same source. Specifically, the multi-dimensional similarity algorithm is a five-dimensional cosine similarity algorithm, and the five-dimensional metadata includes: The name, which is a summary identifier of the core features of the precipitation dataset, is the core identity label of the precipitation dataset and directly reflects the core coverage, data type and key attributes of the data. The abstract, which is a detailed description of the precipitation dataset, supplements and extends the name information; Temporal resolution, or the minimum observation interval of a precipitation dataset in the time dimension, is the time span between two adjacent data collections or records, and directly reflects the level of refinement of the data in the time dimension. Spatial resolution, or the smallest observational unit in the spatial dimension of a precipitation dataset, i.e., the smallest geographical area covered by the data, directly reflects the level of detail in the spatial dimension of the data; and Observation frequency refers to the number of times a precipitation dataset is observed or recorded per unit of time, clarifying the temporal pattern of data collection.

[0080] In one embodiment of the present invention, the formula for calculating the five-dimensional cosine similarity of two precipitation datasets A and B from different sources is as follows: in Let A be a numerical vector representing the i-th dimension of the precipitation datasets A and B. Let be the dimension weights, and ∑ =1; When the data matching degree between the precipitation datasets A and B is greater than a preset threshold, they are determined to be from the same source, and data fusion is performed.

[0081] In one embodiment of the present invention, data fusion includes extracting core feature dimensions of precipitation data from a data hierarchy library and assigning weights based on data source characteristics and meteorological industry standards. The core feature dimensions include: Spatiotemporal characteristics include spatial resolution, temporal continuity, and observation frequency. For example, the spatial resolution of high-resolution data sources is given greater weight, and the temporal continuity of long-sequence data sources is given greater weight. Accuracy characteristics include numerical error range and data source credibility. For example, the numerical error weight of ground station data is higher than that of satellite remote sensing data, and the credibility weight of official authoritative data sources is higher than that of non-official data sources. Semantic features: including precipitation intensity levels, to ensure that the fused values ​​match the intensity levels.

[0082] In one embodiment of the present invention, after extracting and weighting the core feature dimensions, heterogeneous data from the same spatiotemporal unit are weighted and fused according to the feature weights: In the spatial dimension, the accuracy of low spatial resolution data is improved through fusion; In the time dimension, numerical fluctuations are smoothed by fusing data from different observation frequencies; In terms of accuracy, the error of a single data source is reduced by fusion.

[0083] In one embodiment of the present invention, multi-source precipitation datasets are fused into a single high-resolution, long-sequence precipitation data product, and the data product is subjected to feature consistency verification, including: Numerical and semantic feature verification, such as verifying whether the precipitation value matches the corresponding intensity level, and correcting it according to meteorological industry standards if they do not match. Spatiotemporal feature verification, such as verifying whether the spatial resolution and time series of the fused data meet the preset targets. If resolution degradation or sequence breakpoints occur, the interpolation algorithm or weight assignment is readjusted. Cross-data source consistency checks, such as randomly sampling fused data and comparing it with the core features of the original heterogeneous data, ensure that the fusion results are free from systematic bias.

[0084] In one embodiment of the present invention, in step S400, the service interface is constructed and applied based on the fused data stored in the database, and a standard data service interface is developed and released to provide high-quality and reliable data support for meteorological analysis, disaster early warning and / or scientific research applications.

[0085] Specifically, in one embodiment of the present invention, a RESTful API is developed using Python's Flask or FastAPI framework, and the following key interfaces are provided: GET / api / precipitation?start_time=xxx&end_time=xxx&bbox=min_lon,min_lat,max_lon,max_lat: Query precipitation data for a specified time and / or geographic area; GET / api / disaster?location=xxx: Query rainfall-related disaster records for a specified area; and GET / api / image / {image_id}: Retrieves the corresponding precipitation image based on the ID.

[0086] This invention also proposes an automatic acquisition and fusion application system for multi-source precipitation data from the Internet. The system is characterized in that the modules of the system interact directly through a standardized data format. The system includes a data crawling module, a data preprocessing module, a data storage and quality control module, and a data service interface module.

[0087] In one embodiment of the present invention, the data crawling module adopts a customized Python crawler framework to adapt to the data loading characteristics of different target websites. For websites without anti-crawling mechanisms, it initiates data requests through the requests library. For websites that dynamically load data, it obtains data through dynamic webpage parsing technology or by simulating browser behavior. It automatically collects global precipitation data, precipitation-related disaster data, and precipitation images. It supports periodic fetching at preset time intervals to achieve incremental synchronization. At the same time, it integrates a text decryption algorithm to detect the font encryption features in webpages, parse the cmap table of font files, establish a mapping relationship between garbled text and plaintext, and complete the lossless extraction of encrypted fields.

[0088] In one embodiment of the present invention, the data preprocessing module uses a data processing library to remove completely duplicate records from the original data, standardizes the data format, unifies the display format of key information such as time, date, geographic coordinates and values, removes abnormal content such as non-standard characters and redundant symbols from the data, integrates the UIE model fine-tuned by meteorological text data, accurately extracts key information from messy text-based meteorological data, and uniformly encapsulates the processed effective data into a standardized JSON format to ensure the consistency of data interaction between modules.

[0089] In one embodiment of the present invention, the data storage and quality control module stores standardized data in JSON format into a relational database, establishes an index structure adapted to spatiotemporal queries, performs cross-validation of multi-source precipitation data, abstracts the data cleaning algorithm into a binary classifier, evaluates data quality through a cross-validation accuracy formula, and identifies and removes abnormal data. A preset accuracy threshold is provided for the cross-validation accuracy; when the accuracy falls below the threshold, automated early warning and preliminary diagnosis, as well as manual intervention and in-depth processing measures are initiated. Simultaneously, a five-dimensional cosine similarity algorithm is used to filter homogeneous precipitation datasets based on five types of metadata. Through data layer integration of validated valid data to remove redundancy and feature layer fusion of spatiotemporal and accuracy features from different data sources, a high-resolution, long-sequence consistent precipitation data product is generated.

[0090] In one embodiment of the present invention, the data service interface module uses the Flask framework or the FastAPI framework to develop a RESTful API interface, providing three core data services: querying precipitation data for a specified time and geographical range, querying precipitation-related disaster records for a specified area, and obtaining corresponding precipitation images based on an identifier ID. It supports integration with external GIS platform disaster early warning models and meteorological business systems, uses JSON format to achieve data interaction between modules to ensure the standardization and compatibility of data transmission, receives fused data products output by the data storage and quality control module, and responds to external query requests in real time to achieve automated delivery of data services.

[0091] Although various embodiments of the invention have been described above, it should be understood that they are presented by way of example only and not as limitations. It will be apparent to those skilled in the art that various combinations, modifications, and alterations can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely by the appended claims and their equivalents.

Claims

1. An automatic collection and fusion application method of Internet multi-source precipitation data, characterized in that, The method includes: It crawls precipitation data from target websites, and for encrypted web page data, it provides a text decryption algorithm to decrypt the encrypted text and obtain the original data. Preprocessing and data extraction are performed on the raw data to obtain a multi-source precipitation dataset; Store the multi-source precipitation dataset, identify and remove outlier data from the multi-source precipitation dataset, and filter and fuse precipitation data from the same source; and A standard data service interface is provided, through which precipitation data services are provided.

2. The method of claim 1, wherein, The precipitation data crawled from the target website includes: To address the data loading characteristics of the target website, a customized crawler framework was adopted and adapted to automatically and in real-time retrieve data from the target website through dynamic webpage parsing.

3. The method of claim 1, wherein, When the encryption format is fixed glyph encryption, the text decryption algorithm includes: Detecting the font features of the target website includes: The system detects whether the target website's HTML contains the `@font-face` rule and links to font library files; and Detect whether the text content of the target website contains a large number of uncommon Unicode characters; Use a font parsing tool to parse the font library file; Establishing font mapping relationships includes: Manually establish mapping relationships between glyph names and actual characters; and Based on the font library file, a glyph image of each character is automatically generated, and the correspondence between the glyph name and the glyph outline is established through OCR recognition; Extract the cmap table from the font library file, and establish a mapping relationship between the glyph names and actual numbers / characters through glyph feature matching analysis, thereby providing a mapping dictionary from garbled Unicode to plaintext; and Based on the mapping dictionary, replace the garbled Unicode characters in the target website with real text.

4. The method of claim 1, wherein, When the encryption format is dynamic glyph encryption, the text decryption algorithm includes: Each time a target webpage is accessed, the part that needs to be crawled is automatically collected visually, and the text format and / or content are obtained through the layout analysis module and OCR recognition.

5. The method of claim 1, wherein, The data preprocessing includes: Use a data processing library to remove completely duplicate data from the original data; Unify the data format of the original data and remove abnormal characters from the data; and Extracting numbers and key information from text-based data using a UIE-based fine-tuning model.

6. The method of claim 1, wherein, The cross-validation includes: Cross-validation is performed on the precipitation datasets from different sources. If data conflicts are found, the conflicting data is corrected or removed.

7. The method according to claim 6, characterized in that, The accuracy of the cross-validation is evaluated, and when the accuracy of the cross-validation is lower than a preset threshold, the following measures are taken: Automatically trigger the alarm mechanism to notify the data quality manager, locate the problem, and label the relevant data as pending verification. Before the quality verification of this part of the data is completed, use it cautiously or suspend its application in critical business. Initiate a manual intervention process to correct, remove, or label the affected data, while simultaneously tracing and correcting upstream issues; and When necessary, the threshold for cross-validation accuracy can be dynamically adjusted to ensure that the threshold setting is adapted to the actual application scenario.

8. The method according to claim 1, characterized in that, The multi-source precipitation datasets are filtered and fused based on their similarity by calculating data matching degree. The data matching degree is calculated using a five-dimensional cosine similarity algorithm, and the five-dimensional metadata of the algorithm includes: name; summary; Time resolution; Spatial resolution; and Observation frequency.

9. The method according to claim 8, characterized in that, The formula for calculating the five-dimensional cosine similarity is: in For the i-th dimension of the precipitation dataset, a numerical vector is generated. Let be the dimension weights, and ∑ =1; When the data matching degree between any two of the multi-source precipitation datasets is greater than a preset threshold, they are determined to be from the same source, and data fusion is performed.

10. An automatic acquisition and fusion application system for multi-source precipitation data from the Internet, characterized in that, The various modules of the system interact directly through a standardized data format. The system includes: A data crawling module is configured to automatically collect precipitation data and provide raw data. The data crawling module includes a decryption algorithm module, which is configured to provide a text decryption algorithm to parse and extract encrypted text. A data preprocessing module is configured to remove duplicate data and abnormal content from the raw data and convert the raw data into a standardized format. The data storage and quality control module is configured to store precipitation data in a standardized format, remove outlier data through cross-validation, calculate the data matching degree of the dataset using a five-dimensional cosine similarity algorithm, thereby filtering precipitation data from the same source and performing data fusion; and The data service interface module is configured to use a standard data service interface to provide data support for external data products.

Images