Meteorological data metadata extraction method and system based on large language model

Abstract

The application provides a meteorological data metadata extraction method and system based on a large language model. The large language model is used to perform unsupervised or small sample mode discovery on meteorological data filenames, a user-provided specification document is combined to construct a field knowledge context, and initial metadata extraction logic is generated. The logic is dynamically optimized through human-computer interaction verification and correction. Finally, the verified logic is automatically compiled into high-performance executable code for efficient metadata extraction in large-scale production environments. The application not only realizes adaptive analysis of any naming rules, but also opens up an automated path from unstructured specification documents to structured extraction code, significantly reducing manual intervention costs and improving the system's rapid response capability when facing new data sources.

Description

Technical Field

[0001] This application belongs to the field of meteorological data processing technology, and in particular relates to a method and system for extracting meteorological data metadata based on a large language model. Background Technology

[0002] Meteorological data, as a fundamental strategic resource for the nation, plays an irreplaceable role in many key areas such as weather forecasting, climate prediction, disaster prevention and mitigation, ecological environment monitoring, aerospace, and agriculture and water conservancy. With the rapid development of global integrated observation systems (such as the Fengyun series satellites, automatic weather station networks, radar networks, and sounding systems) and numerical weather prediction models (such as ECMWF, GFS, and CMA-GFS), meteorological data exhibits significant characteristics of being "multi-source, heterogeneous, massive, and high-frequency." According to statistics from the World Meteorological Organization (WMO), the daily increase in global meteorological data has reached the petabyte (PB) level and continues to grow rapidly.

[0003] For metadata extraction from meteorological data filenames, current mainstream technical solutions primarily rely on rule-driven methods. Specifically, system developers need to manually write regular expressions or parsing scripts for each type of data product (such as FY-4A AGRI L1 data, GFS global forecast data, ECMWF ERA5 reanalysis data, etc.) to decompose the filenames into structured fields according to a fixed pattern. For example: For Fengyun-4 (FY-4) L1 level data, the China Meteorological Administration industry standard "Naming Specifications for Meteorological Satellite Data Files" (QX / T) applies. WMO Standard 387–2017 specifies naming structures such as “FY4A-_AGRI-_H_DISK_1050E_L1-_FDI-_C001_NOM_20141022100000_20141022101459_0500M_V0001.HDF”, requiring manual parsing of fields such as satellite, instrument, region, time, and resolution. For GFS data from the US NCEP, the typical filename is “gfs.t00z.pgrb2.1p00.f000”, requiring the extraction of forecast cycle time (t00z) and forecast lead time (f000) through preset rules. Although there are guiding documents such as the WMO Core Metadata Specification (WMO-No. 1170) for the WMO-recommended GRIB2 data exchange format, many custom extensions still exist in actual applications by various centers.

[0004] In existing technologies, some systems attempt to introduce configuration files or rule engines (such as Apache NiFi, Talend, and other ETL tools) to manage multiple sets of parsing rules, but they are essentially still within the paradigm of "manual coding + hard rule matching". In recent years, although some research has attempted to introduce machine learning methods (such as sequence labeling and named entity recognition) for filename parsing, they have not yet been widely implemented in business systems due to the scarcity of labeled data and poor domain adaptability.

[0005] Existing methods for extracting meteorological data metadata based on rules or traditional machine learning have the following significant drawbacks: (1) Poor generalization ability, making it difficult to cope with unknown or dynamically changing naming rules. Existing technologies rely heavily on predefined regular expressions or parsing scripts. Once they encounter uncovered data sources, new product versions, or non-compliant "wild files," the system cannot parse them correctly, and may even cause the process to be interrupted. This requires maintenance personnel to continuously and manually maintain and update the rule base, which is costly and has a slow response time.

[0006] (2) Low utilization rate of standard documents, making it difficult to accumulate and reuse knowledge. Although there are a large number of official or industry naming standards (such as QX / T 387–2017, WMO-No.1170, etc.), existing systems cannot automatically understand the semantic rules in these unstructured documents. The standards still need to be manually "translated" into code logic, which is cumbersome and error-prone, resulting in the failure to effectively transform the value of standard documents into system intelligence.

[0007] (3) Lack of intelligent classification capabilities, relying on manual intervention. Existing systems usually only complete field extraction and cannot automatically determine data types. They require subsequent manual annotation or additional configuration, which hinders the automated cataloging and management of data assets.

[0008] (4) The system is rigid and lacks human-machine collaboration and continuous learning mechanisms. When there are deviations in the automatic extraction results, the existing system cannot accept feedback from users in the form of natural language, and it cannot automatically correct the extraction logic based on the feedback, resulting in the accumulation of errors, poor user experience, and the system intelligence cannot evolve with use.

[0009] (5) It is difficult to balance extraction performance and flexibility. If we rely entirely on online parsing of large models, although it is flexible, the performance is low and it is difficult to support the processing of massive amounts of data; if we rely entirely on static code, we lose flexibility. Existing solutions are difficult to achieve a balance between the two. Summary of the Invention

[0010] In view of this, this application aims to propose a method and system for extracting meteorological data metadata based on a large language model, in order to solve at least one of the above problems.

[0011] To achieve the above objectives, the technical solution of this application is implemented as follows: Firstly, this application provides a method for extracting meteorological data metadata based on a large language model, including: Input a list of meteorological data files to be processed, and select representative samples from the list. Use a large language model to perform structural parsing on the file names and identify their naming structure. Extract key information fields that constitute the file names and establish an initial metadata pattern model. The initial metadata pattern model is used to parse unstructured original file names into a sequence of structured metadata fields. Based on the meteorological data naming convention document uploaded by the user, the full text of the document is semantically parsed by calling a large language model, the structured rules related to file naming are extracted and converted into knowledge representation, and the knowledge representation is fused with the initial metadata pattern model to obtain a standardized enhanced metadata pattern model. The input list of meteorological data files is parsed in batches based on the fused standardized enhanced metadata pattern model to generate structured metadata instances, and intelligent classification is performed based on the key semantic fields in the metadata instances. The automatic extraction and classification results are presented to the user for confirmation through visualization. The user can correct the identification errors or inaccuracies through natural language. Based on the captured natural language feedback and combined with contextual information, the prompt words of the large language model are dynamically optimized, the standard enhanced metadata pattern model is regenerated or partially adjusted, and the extraction process is iteratively executed until the results are verified by the user. The validated, standardized, enhanced metadata schema model is compiled into executable code, which is then used to extract metadata from massive amounts of meteorological data.

[0012] Secondly, based on the same inventive concept, this application also provides a meteorological data metadata extraction system based on a large language model, including: The pattern recognition and modeling module is configured to take a list of meteorological data files to be processed as input, select representative samples from the list of meteorological data files, perform structural parsing of the file names and identify the naming structure of the file names by calling a large language model, extract the key information fields that constitute the file names, and establish an initial metadata pattern model; wherein, the initial metadata pattern model is used to parse the unstructured original file names into a sequence of structured metadata fields; The knowledge fusion module is configured to name the meteorological data uploaded by the user into a standard document, perform semantic parsing of the full text of the document by calling a large language model, extract the structured rules related to file naming and convert them into knowledge representation, and then fuse the knowledge representation with the initial metadata pattern model to obtain a standardized enhanced metadata pattern model. The automatic extraction and classification module is configured to perform batch parsing of the input meteorological data file list based on the fused standardized enhanced metadata schema model to generate structured metadata instances, and perform intelligent classification based on the key semantic fields in the metadata instances; The human-computer collaboration and online learning module is configured to present the automatic extraction and classification results to the user for confirmation through visualization. The user can correct the identification errors or inaccuracies through natural language. Based on the captured natural language feedback and combined with contextual information, the module dynamically optimizes the prompt words of the large language model, regenerates or partially adjusts the standard enhanced metadata pattern model, and iteratively executes the extraction process until the results are verified by the user. The code generation and execution module is configured to compile a validated, standardized, enhanced metadata schema model into executable code, and then extract metadata from massive amounts of meteorological data by calling the executable code.

[0013] Thirdly, based on the same inventive concept, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0014] Fourthly, based on the same inventive concept, this application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method as described in the first aspect.

[0015] Compared with existing technologies, the meteorological data metadata extraction method and system based on a large language model described in this application have the following advantages: This application utilizes a large language model to perform unsupervised or few-sample pattern discovery on meteorological data filenames. Combined with user-provided specification documents, it constructs a domain knowledge context to generate initial metadata extraction logic. This logic is dynamically optimized through human-computer interaction for verification and correction. Finally, the verified logic is automatically compiled into high-performance executable code for efficient metadata extraction in large-scale production environments. This solution not only achieves adaptive parsing of arbitrary naming rules but also establishes an automated path from unstructured specification documents to structured extraction code, significantly reducing manual intervention costs and improving the system's rapid response capability when facing new data sources. Attached Figure Description

[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1This is a flowchart of a meteorological data metadata extraction method based on a large language model, as described in an embodiment of this application. Figure 2 This is a schematic diagram of the structure of a meteorological data metadata extraction system based on a large language model, as described in an embodiment of this application. Figure 3 This is a schematic diagram of the hardware structure of the electronic device described in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0018] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0019] The technical solution of this application revolves around five core links: "intelligent recognition, knowledge fusion, automatic extraction, human-machine collaboration, and code solidification," forming a closed-loop metadata extraction process. This technical solution constructs an end-to-end intelligent extraction framework from the original file name to high-performance executable code by integrating the semantic understanding capabilities of large language models, the structured guidance of domain specification knowledge, and the online learning mechanism of human-machine collaboration.

[0020] The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0021] Please see Figure 1 As shown in the figure, this embodiment provides a method for extracting meteorological data metadata based on a large language model, which specifically includes the following steps: Step S10: Input the list of meteorological data files to be processed, select representative samples from the list, and use a large language model to perform structural parsing on the file names and identify the naming structure of the file names. Extract the key information fields that constitute the file names and establish an initial metadata pattern model. The initial metadata pattern model is used to parse the unstructured original file names into a sequence of structured metadata fields.

[0022] Specifically, in this embodiment, the main purpose of this step is to automatically identify and model the metadata pattern structure. Based on the input list of meteorological data files, a large language model is used to perform pattern analysis on the filenames through small-sample learning or supervised fine-tuning. This automatically identifies their naming structure, extracts key information fields constituting the filenames (such as station number, forecast time, regional range, data source identifier, data category identifier, etc.), and establishes a generalizable initial metadata pattern model. This model can parse unstructured original filenames into a structured sequence of metadata fields, laying the foundation for subsequent processing.

[0023] Specifically, based on the list of meteorological data files submitted by users (e.g., ["20250924 / 00 / atmos / gfs.t00z..."]), N (N≥5) samples are randomly selected and input into a large language model (LLM, such as Qwen, DeepSeek, etc.). Using specific prompts, the LLM parses the filenames into alternating "delimiter-field" sequences, identifies the semantic name, variability, content pattern, and normalization rule for each field, and outputs a structured segments template (JSON format) as the initial metadata schema model.

[0024] This step first addresses the fundamental problem of "how to discover structure from irregular filenames," and the specific technical implementation is as follows: (1) Input sample data The system receives a list of filenames to be processed as input (e.g., ["FY4A_...HDF", "gfs.t00z...grb2", "ecmwf_20230101_00.grib"]), which can come from the local file system or remote FTP server, object storage, NAS, etc.

[0025] (2) Small sample pattern discovery N (N≥5) representative samples are randomly selected from the input list or selected using a heuristic strategy (such as clustering by filename length) as a small training set. These samples are then input into a pre-trained large language model (such as DeepSeek, Qwen, Kimi, Claude, ChatGPT, etc.), and the model is guided to perform structural parsing by designing special prompts. Example prompts are as follows: "You are a meteorological data expert. Below is a set of meteorological data file names. Please analyze their common naming structure, identify the semantic fields they contain (such as satellite name, forecast time, data time, data subclass, region code, data level, etc.), and indicate the start and end positions of each field in the file name (indicated by character index). If there are separators between fields (such as underscores, periods, hyphens, etc.), please identify them as well." The implementation code is as follows: 1. Using delimiters as boundaries: Decompose the path into alternating sequences of literal delimiters and semantic fields.

[0026] 2. Fields must contain standardized rules: - Each semantic field needs to provide: -`name`: Standard field name (e.g., "data_format"); -`semantic_type`: semantic type ("semantic" / "fixed_literal" / "placeholder"); -`variability`: "fixed" or "variable_length"; - If variability="fixed", it is recommended to provide `length`; - If variability="variable_length", a pattern must be provided (e.g., "[az]+"). - If the original value of this field is in abbreviated, encoded, or non-standard form, a `normalization_rule` must be provided to specify how to convert it to a standard value. For example: - Original value "pgrb2" → Standard value "grib2" - The "00" in the original value "t00z" → Standard value 0 (integer) - Original value "f003" → Standard value 3 (integer) -Original value "0p25" → Standard value "0.25deg" 3. Context dependency handling: If a field semantically depends on other fields (such as a subtype depending on the domain), mark `context_dependent: true`.

[0027] 4. Output format: Strictly valid JSON, containing only the "segments" array. Each element: -`type="separator"`: contains `literal` - `type="field"`: Includes `name`, `semantic_type`, `variability`, and on-demand `length` / `pattern` / `normalization_rule`, etc. [Input Sample Example] 20250924 / 00 / atmos / gfs.t00z.pgrb2_abc_.0p25.f000 20250924 / 06 / wave / gfs.t06z.pgrb2_abcd_.0p25.f012 20250924 / 12 / chemical / gfs.t12z.pgrb2_xyz_.0p50.f024 Pattern template generation The model returns the following JSON structure, which defines the metadata sequence of the file path and the related business transformation rules: { "segments": [ { "type": "field", "name": "reference_date", "semantic_type": "semantic", "variability": "fixed", "length": 8, "normalization_rule": "Parse as YYYYMMDD string and output inISO format 'YYYY-MM-DD'." }, { "type": "separator", "literal": " / " }, { "type": "field", "name": "cycle_hour", "semantic_type": "semantic", "variability": "fixed", "length": 2, "normalization_rule": "Convert 2-digit zero-padded hour stringto integer (e.g., '00' → 0, '06' → 6)." }, { "type": "separator", "literal": " / " }, { "type": "field", "name": "domain", "semantic_type": "semantic", "variability": "variable_length", "pattern": "[a-z]+" }, { "type": "field", "name": "resolution", "semantic_type": "semantic", "variability": "fixed", "length": 5, "normalization_rule": "Replace 'p' with '.', then append 'deg'(e.g., '0p25' → '0.25deg', '0p50' → '0.50deg')." }, { "type": "separator", "literal": ".f"} ... ] } To address unseen naming rules, this embodiment employs a delimiter-aware pattern generalization strategy. First, potential delimiters are identified through character frequency analysis (e.g., _, ., - appear significantly more frequently than other characters). Then, filenames are segmented into "token" sequences based on these delimiters. A Large Language Model (LLM) performs semantic annotation on this, allowing the system to infer the category of a field even if its content is unknown (e.g., "20230101120000" is highly likely a timestamp). This mechanism significantly improves adaptability to new naming styles.

[0028] The metadata pattern automatic identification method provided in this step has a low barrier to entry. It can automatically discover and model naming structures based on a small number of filenames without requiring manual pre-setting of regular expressions or parsing rules, significantly reducing reliance on professional developers and lowering system maintenance costs.

[0029] Step S20: Based on the meteorological data naming specification document uploaded by the user, the full text of the document is semantically parsed by calling the large language model, the structured rules related to file naming are extracted and converted into knowledge representation, and the knowledge representation is fused with the initial metadata schema model to obtain the specification-enhanced metadata schema model.

[0030] Specifically, in this embodiment, this step primarily aims to facilitate the access and fusion of user-defined naming convention documents. This method supports users uploading various meteorological data naming convention documents (such as the "Fengyun-4 Data File Naming Convention," "GFS Data Naming Rules," and "WMO Core Metadata Standards"). A large language model is used to perform deep parsing of these unstructured documents, automatically extracting key information such as defined field semantics, encoding rules, value ranges, and naming examples, and converting them into machine-readable knowledge representations. This knowledge representation is then fused with the aforementioned initial metadata model to form an enhanced metadata extraction logic that balances general naming rules with specific specification constraints (i.e., planning an enhanced metadata model), significantly improving the accuracy and compliance of the extraction.

[0031] To improve extraction accuracy, this embodiment supports incorporating human expert knowledge (i.e., naming conventions) into the extraction process. The specific implementation is as follows: (1) Standard document access. Users can upload meteorological data naming standard documents in PDF, DOCX or TXT format through the Web interface, such as "Fengyun-4 Meteorological Satellite Data File Naming Standard" and "WMO GRIB2 Encoding Manual".

[0032] (2) Structured knowledge extraction. The system calls LLM to perform semantic parsing on the entire document. The prompt words are designed as follows: "Please extract all rules related to file naming from the following meteorological data naming convention document. This includes: field name, field definition, allowed values ​​(encoding table), field position, and example file name. The output should be a structured JSON array, where each object contains 'field_name', 'definition', 'allowed_values', 'position', and 'example'." After the model outputs, the system stores it as structured knowledge entries and constructs a lightweight knowledge graph. For example, it maps "FY4A" to satellite ID "A", "AGRI" to instrument ID "#", and "DISK" to region code "1050E", etc.

[0033] (3) Standardization Integration and Enhanced Prompts. In the subsequent metadata extraction stage, the system injects the above knowledge as context into the LLM prompts. For example: "You are a meteorological data expert. Below is a set of meteorological data file names. Please analyze their common naming structure, identify the semantic fields they contain (such as satellite name, forecast time, data time, data subclass, area code, data level, etc.), and indicate the start and end positions of each field in the file name (indicated by character index). If there are separators between fields (such as underscores, periods, hyphens, etc.), please identify them as well."

[0034] The "Naming Convention for L1-Level Data Files of Fengyun-4 Meteorological Satellite" defines the following key field rules: - Field "satellite" (satellite code): Allowed values: {"FY4A": "FY4A", "FY4B": "FY4B"} Location: At the beginning of the filename; - Field "instrument" (load instrument): Permissible value: {"AGRI": "Advanced Earth Radiation Imager", "GIIRS": "Interferometric Atmospheric Vertical Sounder"} Location: Immediately following the satellite designation; - Field "region" (observation area): Allowed values: {"DISK": "Full disk (105.0°E)", "REGC": "China region", "REGX": "Custom region"} Location: After the instrument field; - Field "resolution" (spatial resolution): Permissible values: {"0500M": "0.5 km", "1000M": "1 km", "2000M": "2 km"} Location: After the time range field.

[0035] The implementation code is as follows: 1. Using delimiters as boundaries: Decompose the path into alternating sequences of literal delimiters and semantic fields.

[0036] 2. Fields must contain standardized rules: - Each semantic field needs to provide: -`name`: Standard field name (e.g., "data_format"); -`semantic_type`: semantic type ("semantic" / "fixed_literal" / "placeholder"); -`variability`: "fixed" or "variable_length"; - If variability="fixed", it is recommended to provide `length`; - If variability="variable_length", a pattern must be provided (e.g., "[az]+"). - If the original value of this field is in abbreviated, encoded, or non-standard form, a `normalization_rule` must be provided to specify how to convert it to a standard value. For example: - Original value "pgrb2" → Standard value "grib2" - The "00" in the original value "t00z" → Standard value 0 (integer) - Original value "f003" → Standard value 3 (integer) -Original value "0p25" → Standard value "0.25deg" 3. Context dependency handling: If a field semantically depends on other fields (such as a subtype depending on the domain), mark `context_dependent: true`.

[0037] 4. Output format: Strictly valid JSON, containing only the "segments" array. Each element: -`type="separator"`: contains `literal` -`type="field"`: Includes `name`, `semantic_type`, `variability`, and, as needed, `length` / `pattern` / `normalization_rule`, etc. [Input Sample] FY4A-_AGRI-_H_DISK_1050E_L1-_FDI-_C001_NOM_20151022100000_20151022101459_0500M_V0002.HDF FY4A-_AGRI-_H_DISK_1030E_L1-_FDI-_C001_NOM_20141025100000_20141025101459_0500M_V0003.HDF The metadata extraction template generated from the prompt words "FY4A-_AGRI-_H_DISK_1040E_L1-_FDI-_C002_NOM_20141022100000_20141022101459_0500M_V0012.HDF" after integrating the above structured information is as follows: { "segments": [ { "type": "field", "name": "satellite", "semantic_type": "semantic", "variability": "fixed", "length": 4, "normalization_rule": "Use raw value as standard identifier(eg, 'FY4A')." }, { "type": "separator", "literal": "-_" }, { "type": "field", "name": "instrument", "semantic_type": "semantic", "variability": "fixed", "length": 4, "normalization_rule": "Use raw value as standard identifier(e.g., 'AGRI')." }, { "type": "separator", "literal": "_" }, { "type": "field", "name": "central_longitude", "semantic_type": "semantic", "variability": "fixed", "length": 5, "normalization_rule": "Interpret as longitude in 0.1°E units;output as string (e.g., '1050E' → '105.0°E')." }, { "type": "separator", "literal": "_" }, { "type": "field", "name": "data_level", "semantic_type": "semantic", "variability": "fixed", "length": 2, "normalization_rule": "Keep as standard level code (e.g., 'L1')." }, { "type": "field", "name": "product_type", "semantic_type": "semantic", "variability": "fixed", "length": 3, "normalization_rule": "Map 'FDI' to full product name 'FullDisk Image' if needed, but retain 'FDI' as standard code." }, { "type": "field", "name": "channel_group", "semantic_type": "semantic", "variability": "fixed", "length": 4, "normalization_rule": "Interpret as channel group code (e.g., 'C001' → channel group 1)." }, { "type": "separator", "literal": "_" }, { "type": "separator", "literal": "_" }, { "type": "field", "name": "start_time", "semantic_type": "semantic", "variability": "fixed", "length": 14, "normalization_rule": "Parse as YYYYMMDDHHMMSS and output inISO 8601 format 'YYYY-MM-DDTHH:MM:SSZ'." }, { "type": "field", "name": "end_time", "semantic_type": "semantic", "variability": "fixed", "length": 14, "normalization_rule": "Parse as YYYYMMDDHHMMSS and output inISO 8601 format 'YYYY-MM-DDTHH:MM:SSZ'." }, ... ] } In this way, LLM can refer to authoritative rules during parsing, significantly reducing ambiguity and improving the accuracy of field alignment.

[0038] This step effectively improves the accuracy of metadata extraction through standardized document automatic parsing and knowledge fusion mechanisms. It can automatically convert user-provided naming conventions (such as QX / T, WMO standards, custom conventions, etc.) into structured knowledge and guide the extraction process, effectively avoiding field mismatches caused by misunderstandings of rules.

[0039] Step S30: Based on the fused standardized enhanced metadata model, the input list of meteorological data files is parsed in batches to generate structured metadata instances, and intelligent classification is performed based on the key semantic fields in the metadata instances.

[0040] Specifically, in this embodiment, the purpose of this step is mainly for automatic metadata extraction and classification. Based on the fused extraction logic, the system performs batch parsing of the input list of meteorological data files to generate structured metadata instances. Simultaneously, based on key semantic fields in the metadata instances (such as data level, product type, spatial dimension, and time attributes), the system automatically determines the data category, achieving intelligent classification of forecast data, observation data, grid data, station data, and other types, providing semantic support for subsequent data organization and services.

[0041] After completing pattern modeling and knowledge fusion, the system enters the batch processing stage: (1) Metadata instance generation For each input filename, a structured metadata object (JSON format) is generated by invoking LLM prompts based on fused knowledge. This object contains field name-value pairs. For example, the extracted metadata structure of the file FY4A-_AGRI-_H_DISK_1040E_L1-_FDI-_C002_NOM_20141022100000_20141022101459_0500M_V0012.HDF is as follows: { "satellite": "FY4A", "instrument": "AGRI", "region": "DISK", "central_longitude": "104.0°E", "data_level": "L1", "product_type": "FDI", "channel_group": "C002", "processing_flag": "NOM", "start_time": "2014-10-22T10:00:00Z", "end_time": "2014-10-22T10:14:59Z", "resolution": "0.5km", "version": "0012" }

[0042] (2) Automatic data classification The system automatically categorizes data based on key fields in the metadata. Categorization rules can be predefined or generated with the assistance of an LLM (Local Management System). If the level is "L0" or "L1" and contains an instrument field, then it is observation data; If it contains the forecast_hour or cycle_time field, it is forecast data; If it contains grid_resolution but has no site ID, then it is grid data; If it contains station_id, then it is station data.

[0043] The classification results are used as part of the metadata for subsequent database modeling and retrieval.

[0044] This step supports automatic metadata extraction and intelligent classification. It is easy to use and can automatically determine the data type (such as forecast data, observation data, grid data, etc.) while parsing the file name. It can complete the initial cataloging of data assets without manual intervention, thus improving data management efficiency.

[0045] Step S40: The automatic extraction and classification results are presented to the user for confirmation through visualization. The user corrects the identification errors or inaccuracies using natural language. Based on the captured natural language feedback and contextual information, the large language model prompts are dynamically optimized, the standard enhanced metadata pattern model is regenerated or partially adjusted, and the extraction process is iteratively executed until the results are verified by the user.

[0046] Specifically, in this embodiment, the purpose of this step is mainly for human-machine collaborative correction and online learning, visually presenting the results of automatic extraction and classification to the user for confirmation. Users can correct erroneous or inaccurate identifications using natural language (e.g., "This file belongs to reanalysis data, not prediction data"). Upon receiving such feedback, the system dynamically optimizes the prompt word project of the large language model by combining the original filename, the standardized knowledge base, and user instructions. It then regenerates or partially adjusts the metadata extraction logic and iteratively executes the extraction process until the result is verified by the user. This mechanism enables continuous learning and self-optimization of the system.

[0047] Specifically, to ensure the reliability of the results, this embodiment introduces a closed-loop human-machine collaboration mechanism: First, the system presents the automatically extracted and classified results to the user in a visual manner. For items with uncertain identification, the system highlights the uncertain parts and provides the reasoning basis (i.e., prompts and model output), allowing the user to easily make corrections.

[0048] Secondly, when a user submits a correction, such as saying "This file is not forecast data, but reanalysis data," the system will capture this natural language feedback. At this point, the system will enter online learning mode. It will collect current context information, including the filename, the system's original extraction results, the user's correction comments, and relevant specification knowledge. It will then regenerate optimized prompts. For example: "The user pointed out that the file 'ecmwf_20230101_00.grib' belongs to reanalysis data. According to WMO specifications, reanalysis data typically does not contain forecast lead time fields. Please re-parse this filename." Then, the LLM regenerates the optimized extraction logic based on this information. This is not simply replacing a rule, but a deeper level of cue engineering optimization. The system may generate a new, more targeted cue word, or fine-tune certain parameters of the model to make the correct judgment when encountering similar situations in the future. For example, the system may learn that, under a specific data source, the "PRF" suffix usually means precipitation observation data rather than forecast data, thereby correcting previous misclassifications. Finally, this optimized extraction logic is temporarily stored and immediately applied to the remaining files in the queue. This process can be iterated repeatedly until the user confirms that all results are accurate. This closed-loop feedback mechanism ensures the long-term reliability and adaptability of the system.

[0049] (1) Results Visualization and Feedback. First, the system presents the automatically extracted and classified results to the user in a visual manner. For items with uncertain identification, the system highlights the uncertain parts and provides the reasoning basis (i.e., prompts and model outputs). Users can easily make corrections. Users can make corrections by inputting natural language, such as: "This file is reanalysis data, not forecast data" or "The region should be 'China' instead of 'DISK'".

[0050] (2) Online prompt optimization. When a user submits a correction, such as saying "This file is not forecast data, but reanalysis data," the system will capture this natural language feedback. At this time, the system will enter online learning mode. It will collect the current context information, including the file name, the system's original extraction results, the user's correction comments, and relevant specification knowledge. It will then regenerate the optimized prompt words. For example: "The user pointed out that the file 'ecmwf_20230101_00.grib' belongs to reanalysis data. According to WMO specifications, reanalysis data usually does not contain the forecast lead time field. Please re-parse the file name." After capturing the user feedback, the system will construct a new context together with the original file name, the current extraction results, and relevant specification knowledge, and regenerate the optimized prompt words. For example: "The user pointed out that the file 'ecmwf_20230101_00.grib' belongs to reanalysis data. According to WMO specifications, reanalysis data usually does not contain the forecast lead time field. Please re-parse the file name." (3) Logical Iterative Update. LLM regenerates metadata based on the new prompt words. The system records the context and result of this correction to form a "corrected sample" for rapid reasoning in similar scenarios. This process can be iterated multiple times until the user confirms the accuracy of the result.

[0051] This step, through human-machine collaboration and online learning mechanisms, effectively ensures the business compliance of the extraction results, allows users to correct the results in natural language, and enables the system to dynamically optimize the extraction logic accordingly, effectively avoiding metadata errors caused by model misjudgment and ensuring that the final results meet the actual needs of users.

[0052] Step S50: Compile the validated, standardized, enhanced metadata schema model into executable code, and extract metadata from massive amounts of meteorological data by calling the executable code.

[0053] Specifically, in this embodiment, the purpose of this step is mainly for the automatic generation and execution of extraction logic code. After the extraction logic is confirmed by the user, the system automatically compiles it into high-performance, independently runnable executable code (such as Python or Java functions). This code completely solidifies the verified extraction rules and no longer relies on online calls to large language models. In a real production environment, the system directly calls this code to perform fast, batch, and low-latency metadata extraction from massive meteorological data files, balancing intelligence and execution efficiency.

[0054] To balance flexibility and performance, this embodiment "solidifies" the intelligent logic into code after verification: (1) Code Generation. The system analyzes the final verified extraction logic (including field regular expression patterns, mapping relationships, and conditional branches), calls the code generation module, and automatically generates Python functions. For example: You are a code generation engine whose task is to generate a high-performance, executable Python function based on a given structured pattern definition (segments) of meteorological data filenames, to parse such filenames and output standardized metadata.

[0055] Input: You will receive a JSON array named `segments`, which describes the exact structure of the filenames. Each element is: - If type="separator": contains literal (fixed string separator). - If type="field": includes name, semantic_type, variability, length, and optional normalization_rule description.

[0056] Settings: 1. Function signature: def parse_filename(filename: str) -> Optional[Dict[str, Any]] 2. Parsing logic: - Strictly follow the order of segments, alternating between delimiters and fields; - The field is truncated from the current position by a fixed length; - The separator must exactly match the literal; - If any step does not match, return None.

[0057] 3. Standardization process: - Fields are added to the output dictionary only if semantic_type == "semantic"; - Implement value transformations (such as time formatting, encoding mapping, etc.) based on normalization_rule; - Implement the corresponding transformation logic inside the function for each field that needs to be standardized.

[0058] 4. Code quality: -Does not depend on external libraries (except typing, datetime, and re); - Includes complete error handling; - The code is clear, efficient, and free of redundancy; - Include a comment at the beginning of the function to indicate that the function was automatically generated by this system.

[0059] Structured schema definition (segments): json [ {"type": "field", "name": "satellite", "semantic_type": "semantic", "variability": "fixed", "length": 4}, {"type": "separator", "literal": "-_"}, {"type": "field", "name": "instrument", "semantic_type": "semantic", "variability": "fixed", "length": 4}, ... ] ``` The generated result code is as follows: import re import datetime from datetime from typing import Optional, Dict, Any def parse_filename(filename: str) -> Optional[Dict[str, Any]]: """ Parse the file name of Fengyun-4 (FY-4) L1-level data and return structured metadata.

[0060] """ segments = {"type": "field", "name": "satellite", "semantic_type": "semantic", "variability": "fixed", "length": 4}, {"type": "separator", "literal": "-_"}, {"type": "field", "name": "instrument", "semantic_type": "semantic", "variability": "fixed", "length": 4}, {"type": "separator", "literal": "-_H_"}, {"type": "field", "name": "region", "semantic_type": "semantic", "variability": "fixed", "length": 4} ... def normalize_value(field_name: str, raw: str): if field_name == "central_longitude": if raw.endswith('E') and raw[:-1].isdigit(): return f"{float(raw[:-1]) / 10:.1f}°E" return raw elif field_name == "resolution":​ return {"0500M": "0.5km", "1000M": "1.0km", "2000M": "2.0km"}.get(raw, raw) elif field_name in ("start_time", "end_time"): if len(raw) == 14 and raw.isdigit(): return datetime.strptime(raw, "%Y%m%d%H%M%S").strftime("%Y-%m-%dT%H:%M:%SZ") return raw elif field_name == "version": return raw[1:] if raw.startswith('V') and len(raw) == 5else raw else: return raw pos = 0 result = {} n = len(filename) for seg in segments: if pos > n: return None if seg["type"] == "separator": lit = seg["literal"] if not filename.startswith(lit, pos): return None pos += len(lit) elif seg["type"] == "field": length = seg["length"] if pos + length > n: return None raw_val = filename[pos:pos + length] pos += length if seg.get("semantic_type") == "semantic": result[seg["name"]] = normalize_value(seg["name"],raw_val) # Ensure a complete match if pos != n: return None return result (2) High-performance execution. The generated code is compiled and registered to the metadata collection engine. In the production environment, the system directly calls this function (by inputting the final verified segment template into the code generation engine, which automatically generates a high-performance Python parsing function) to process massive files, avoiding repeated calls to LLM and achieving millisecond-level single-file parsing performance.

[0061] This step provides automatic code generation for extraction logic, offering superior performance. It can solidify proven intelligent extraction logic into high-performance executable code, directly calling local functions to complete batch processing during actual operation. This eliminates the need for repeated calls to large language models, effectively avoiding high latency and high cost issues and significantly improving processing efficiency.

[0062] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0063] Based on the same inventive concept, and corresponding to the methods of any of the above embodiments, the embodiments of this application also provide a meteorological data metadata extraction system based on a large language model.

[0064] like Figure 2 As shown, the meteorological data metadata extraction system based on a large language model includes: The pattern recognition and modeling module 11 is configured to take a list of meteorological data files to be processed as input, select representative samples from the list of meteorological data files, perform structural parsing of the file names and identify the naming structure of the file names by calling a large language model, extract the key information fields that constitute the file names, and establish an initial metadata pattern model; wherein, the initial metadata pattern model is used to parse the unstructured original file names into a sequence of structured metadata fields. The knowledge fusion module 12 is configured to name the meteorological data uploaded by the user into a standard document, perform semantic parsing of the full text of the document by calling a large language model, extract the structured rules related to file naming and convert them into knowledge representation, and then fuse the knowledge representation with the initial metadata schema model to obtain a standardized enhanced metadata schema model. The automatic extraction and classification module 13 is configured to perform batch parsing of the input meteorological data file list based on the fused standardized enhanced metadata schema model to generate structured metadata instances, and perform intelligent classification based on the key semantic fields in the metadata instances. The human-computer collaboration and online learning module 14 is configured to present the automatic extraction and classification results to the user for confirmation through visualization. The user can correct the identification errors or inaccuracies through natural language. Based on the captured natural language feedback and combined with context information, the module dynamically optimizes the prompt words of the large language model, regenerates or partially adjusts the standard enhanced metadata pattern model, and iteratively executes the extraction process until the results are verified by the user. The code generation and execution module 15 is configured to compile a validated, standardized, enhanced metadata schema model into executable code, and extract metadata from massive amounts of meteorological data by calling the executable code.

[0065] For ease of description, the above system is described by dividing it into various modules based on their functions. Of course, in implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0066] The system described in the above embodiments is used to implement the corresponding method in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0067] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in any of the above embodiments.

[0068] Figure 3This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0069] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0070] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0071] The input / output interface 1030 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0072] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0073] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0074] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0075] The electronic devices described above are used to implement the corresponding methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0076] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to perform the methods described in any of the above embodiments.

[0077] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0078] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the methods described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0079] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0080] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the claims of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A method for extracting meteorological data metadata based on a large language model, characterized in that, include: Input a list of meteorological data files to be processed, and select representative samples from the list. Use a large language model to perform structural parsing on the file names and identify their naming structure. Extract key information fields that constitute the file names and establish an initial metadata pattern model. The initial metadata pattern model is used to parse unstructured original file names into a sequence of structured metadata fields. Based on the meteorological data naming convention document uploaded by the user, the full text of the document is semantically parsed by calling a large language model, the structured rules related to file naming are extracted and converted into knowledge representation, and the knowledge representation is fused with the initial metadata pattern model to obtain a standardized enhanced metadata pattern model. The input list of meteorological data files is parsed in batches based on the fused standardized enhanced metadata pattern model to generate structured metadata instances, and intelligent classification is performed based on the key semantic fields in the metadata instances. The automatic extraction and classification results are presented to the user for confirmation through visualization. The user can correct the identification errors or inaccuracies through natural language. Based on the captured natural language feedback and combined with contextual information, the prompt words of the large language model are dynamically optimized, the standard enhanced metadata pattern model is regenerated or partially adjusted, and the extraction process is iteratively executed until the results are verified by the user. The validated, standardized, enhanced metadata schema model is compiled into executable code, which is then used to extract metadata from massive amounts of meteorological data.

2. The method according to claim 1, characterized in that: Guided by prompt words, the large language model parses filenames into alternating sequences of "delimiters-fields," identifies the semantic name, variable-length attributes, content patterns, and standardization rules of each field, and outputs structured segment templates as the initial metadata schema model.

3. The method according to claim 2, characterized in that: By extracting key information that includes at least field definitions, encoding rules, value ranges, and named instances, the key information is converted into knowledge representations to construct structured knowledge entries. These structured knowledge entries serve as context-injected prompts for the large language model, thereby forming a standardized enhanced metadata schema model.

4. The method according to claim 1, characterized in that: The system iterates through the list of input meteorological data files, generates structured metadata instances by calling prompts from a large language model that integrates normative knowledge, and performs intelligent classification based on key semantic fields in the metadata instances to determine the category to which the data belongs.

5. The method according to claim 4, characterized in that: The intelligent classification rules are generated either through predefined rules or with the assistance of a large language model, and the categories to which the data belongs include at least observation data, forecast data, grid data, and site data.

6. The method according to claim 1, characterized in that: The system collects current contextual information based on the online learning model and regenerates optimized prompts. The contextual information includes filenames, the system's original extraction results, user corrections, and relevant standard knowledge.

7. The method according to claim 6, characterized in that: The final verified segment templates are input into the code generation engine to generate Python parsed functions.

8. A meteorological data metadata extraction system based on a large language model, characterized in that, include: The pattern recognition and modeling module is configured to take a list of meteorological data files to be processed as input, select representative samples from the list of meteorological data files, perform structural parsing of the file names and identify the naming structure of the file names by calling a large language model, extract the key information fields that constitute the file names, and establish an initial metadata pattern model; wherein, the initial metadata pattern model is used to parse the unstructured original file names into a sequence of structured metadata fields; The knowledge fusion module is configured to name the meteorological data uploaded by the user into a standard document, perform semantic parsing of the full text of the document by calling a large language model, extract the structured rules related to file naming and convert them into knowledge representation, and then fuse the knowledge representation with the initial metadata pattern model to obtain a standardized enhanced metadata pattern model. The automatic extraction and classification module is configured to perform batch parsing of the input meteorological data file list based on the fused standardized enhanced metadata schema model to generate structured metadata instances, and perform intelligent classification based on the key semantic fields in the metadata instances; The human-computer collaboration and online learning module is configured to present the automatic extraction and classification results to the user for confirmation through visualization. The user can correct the identification errors or inaccuracies through natural language. Based on the captured natural language feedback and combined with contextual information, the module dynamically optimizes the prompt words of the large language model, regenerates or partially adjusts the standard enhanced metadata pattern model, and iteratively executes the extraction process until the results are verified by the user. The code generation and execution module is configured to compile a validated, standardized, enhanced metadata schema model into executable code, and then extract metadata from massive amounts of meteorological data by calling the executable code.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium, characterized in that, in, The non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform the method described in any one of claims 1-7.

Images