A method, system, device, and medium for encapsulating environmental observation data.
By using structured requirements specification and field-level validation, combined with iterative optimization of the large language model, the accuracy problem in environmental observation data encapsulation was solved, and efficient and accurate data processing logic generation was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HONG KONG UNIV OF SCI & TECH (GUANGZHOU)
- Filing Date
- 2026-04-22
- Publication Date
- 2026-06-02
Smart Images
Figure CN122133641A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data encapsulation, and in particular to a method, system, device, and medium for encapsulating environmental observation data. Background Technology
[0002] Environmental observation data (such as ocean temperature and salinity, meteorological time series, ecological monitoring, and geological disaster data) are characterized by multi-source heterogeneity, strong spatiotemporal correlation, rich metadata, and massive data volume. In scientific research and operational applications, these data often require a series of processing steps, including cleaning, alignment, aggregation, enhancement, and anonymization, before they can be transformed into standard datasets that can be analyzed, shared, or published. Traditional data processing workflows heavily rely on manually written query statements and customized programs. Faced with constantly changing observation variables, data formats, and business requirements, the development cycle is long and the maintenance cost is high, severely restricting the usability and reuse efficiency of scientific data.
[0003] Currently, the encapsulation of environmental observation data mainly relies on two technical approaches. The first is based on fixed templates or manually written Structured Query Language (SQL) scripts. These scripts pre-define processing flows for specific data structures and business logic. When data fields change or need to adapt to new observation scenarios, the templates must be manually modified, making it difficult to handle the dynamic field expansion and multimodal fusion requirements common in scientific data. The second approach directly utilizes Large Language Models (LLMs) to generate data processing statements in one go based on natural language descriptions. While this method improves the speed of initial draft generation, it often suffers from problems such as missing fields, incorrect data type mapping, incomplete aggregation logic, and syntax incompatibility due to a lack of stable understanding of input field structures, business constraints, and output requirements. The generated results are often not directly executable and require repeated manual debugging. Summary of the Invention
[0004] This invention provides a method, system, device, and medium for encapsulating environmental observation data, which can improve the accuracy of environmental observation data encapsulation.
[0005] The first aspect of this invention provides a method for encapsulating environmental observation data, comprising: Obtain a natural language task description and corresponding data pattern information for environmental observation data, parse the natural language task description and the data pattern information, and generate a structured requirement specification; The structured requirement specification is input into a preset large language model to generate a first candidate processing logic for processing the environmental observation data; The input and output fields involved in the first candidate processing logic are parsed, and the input and output fields are compared with the expected fields in the structured requirement specification to generate a first comprehensive quality score and missing field information. If the first comprehensive quality score does not reach the preset threshold, the missing field information is fed back to the large language model to regenerate the second candidate processing logic. This continues until the second comprehensive quality score recalculated based on the second candidate processing logic reaches the preset threshold and the current iteration number does not reach the preset maximum iteration number. Then, the corresponding second candidate processing logic is output as the target processing logic for encapsulating and processing the environmental observation data.
[0006] This invention, through acquiring natural language task descriptions and data pattern information and parsing them to generate structured requirement specifications, transforms ambiguous and unstructured user intentions into an intermediate representation that machines can accurately understand and execute. This eliminates the interference of natural language ambiguity on the generation of subsequent processing logic, ensuring the accuracy and completeness of requirement expression from the source. Secondly, the structured requirement specification is input into a large language model to generate first candidate processing logic. This ensures that the code generation process no longer relies on one-off, unconstrained open generation, but is based on explicit specification constraints, significantly reducing the probability of arbitrary model speculation or omission of key processing steps, and improving the consistency between the generated logic and the original requirements. Next, by parsing the input and output fields involved in the candidate processing logic and comparing them one by one with the expected fields in the structured requirement specification, a first comprehensive quality score and missing field information are generated. This achieves automated, fine-grained quantitative evaluation of the generated logic. This field-level comparison mechanism can accurately identify specific defects such as missing fields, type mismatches, or incomplete logic, avoiding the bias and subjective errors of evaluations based solely on syntax checking or manual inspection, thus providing an objective and quantifiable basis for subsequent corrections. Finally, when the overall quality score fails to reach the preset threshold, the missing field information is fed back to the large language model to regenerate the second candidate processing logic. This process is iterated repeatedly until the score meets the standard or the maximum number of iterations is reached, forming a closed-loop optimization mechanism of "generation-verification-feedback-correction". This mechanism can specifically supplement missing fields, correct erroneous logic, and continuously improve the quality of the processing logic within a limited number of rounds until the preset accuracy requirements are met. In summary, the technical solution of this invention, through the systematic combination of structured specification constraints, field-level verification scoring, and iterative feedback correction, effectively overcomes the problem of inaccurate encapsulation caused by natural language understanding bias, field omissions, and logical deficiencies in the prior art, and significantly improves the final accuracy of environmental observation data encapsulation.
[0007] Furthermore, the step of parsing the natural language task description and the data pattern information to generate a structured requirement specification includes: Based on the data pattern information, the key field types and field names in the environmental observation data are identified, wherein the key field types include grouping key fields, time fields, numerical fields, metadata fields, quality fields, and sensitive fields; The natural language task description is preprocessed to obtain the target task description, and data processing rules are extracted from the target task description. The data processing rules include time window configuration, statistical criteria, quality judgment rules, anomaly detection rules, and de-identification rules. The identified key field types, field names, and data processing rules are organized according to a preset format to generate the structured requirement specification.
[0008] By identifying key field types such as grouping keys, time, values, metadata, quality, and sensitivity, and extracting time windows, statistical definitions, quality and desensitization rules, the original data patterns and task requirements can be systematically and structurally mapped into the specification, avoiding field omissions or rule ambiguities, and laying a solid foundation for the accurate generation of subsequent processing logic.
[0009] Further, the preprocessing of the natural language task description to obtain the target task representation includes: Obtain a domain terminology database corresponding to the domain to which the environmental observation data belongs; The semantic information in the natural language task description is aligned using the domain terminology database to obtain a first processing result; The text in the first processing result is enhanced to obtain the target task description.
[0010] By using a domain terminology database to perform semantic alignment and text enhancement on natural language task descriptions, ambiguity in domain terms can be eliminated, implicit semantics can be supplemented, and the generated target task descriptions can be made more in line with the expression habits of specific scientific fields, thereby improving the accuracy of requirement analysis and indirectly improving the accuracy of data encapsulation.
[0011] Further, the step of inputting the structured requirement specification into a preset large language model to generate a first candidate processing logic for processing the environmental observation data includes: Obtain the data source access information of the environmental observation data, as well as the initial missing field information recorded in the previous round of verification feedback; The structured requirement specification, the data source access information, and the initial missing field information are input into the large language model, so that the large language model can generate candidate processing logic that includes a data source definition part, a data processing operation part, and a result output definition part.
[0012] In this way, when generating candidate processing logic, the data source access information and the missing field information from the previous round are input simultaneously. This guides the large language model to supplement missing fields and correct errors in a targeted manner, avoiding the recurrence of similar defects. It achieves precise correction based on feedback, significantly improving the efficiency of iterative optimization and the accuracy of final encapsulation.
[0013] Further, the step of parsing the input and output fields involved in the first candidate processing logic, and comparing the input and output fields with the expected fields in the structured requirement specification to generate a first comprehensive quality score and missing field information, includes: Check whether the first candidate processing logic includes the data source definition part, the data processing operation part, and the result output definition part, and check whether the functions and keywords used conform to the preset syntax rules; If the check passes, the data processing operation part in the first candidate processing logic is parsed to obtain the data source reference and the result generation expression, and the input field and the output field are extracted from the data source reference and the result generation expression respectively. The ratio of the number of extracted input fields to the number of expected input fields in the structured requirement specification is used as the first field coverage rate, and the ratio of the number of output fields to the number of expected output fields in the structured requirement specification is used as the second field coverage rate. The first comprehensive quality score and the missing field information are generated based on the first field coverage rate and the second field coverage rate.
[0014] By performing structural integrity checks and function and keyword syntax validations on the candidate processing logic, and by extracting input / output fields and comparing them with the expected fields in the specification to calculate field coverage, dual verification of syntax and business semantics is achieved. This enables precise identification of missing fields or type mismatch issues, providing a reliable basis for quantitative evaluation.
[0015] Furthermore, the present invention also includes: if the check fails, generating syntax error information and feeding the syntax error information as part of the missing field information back to the large language model to update the large language model.
[0016] In this way, when the structure or syntax check fails, the specific syntax error information is fed back to the large language model as part of the missing field. This allows the model to clearly know the defects at the code format level, so that it can specifically correct the syntax problems in the next generation and avoid the encapsulated logic from failing to execute due to syntax errors.
[0017] Further, generating the first comprehensive quality score and the missing field information based on the first field coverage and the second field coverage includes: The first field in the expected input field that does not appear in the extracted input field, and the second field in the expected output field that does not appear in the extracted output field, are jointly recorded as the missing field information; Based on the field function types defined in the structured requirements specification, corresponding weight coefficients are assigned to the first field coverage rate and the second field coverage rate, respectively. The first field coverage rate and the second field coverage rate are then weighted and summed according to the weight coefficients to obtain the first comprehensive quality score.
[0018] By recording the missing input and output fields in the expected fields and assigning weights according to the field function type, a weighted sum is obtained to obtain a comprehensive quality score. This makes the score more scientific and reasonable, accurately reflecting the completeness of the encapsulated logic and guiding the iteration towards optimization in the direction of completing key fields.
[0019] Another embodiment of the present invention also provides an environmental observation data encapsulation system, comprising: The first module is used to acquire natural language task descriptions and corresponding data pattern information for environmental observation data, parse the natural language task descriptions and data pattern information, and generate structured requirement specifications. The second module is used to input the structured requirement specification into a preset large language model to generate a first candidate processing logic for processing the environmental observation data. The third module is used to parse the input fields and output fields involved in the first candidate processing logic, and compare the input fields and output fields with the expected fields in the structured requirement specification to generate a first comprehensive quality score and missing field information. The fourth module is used to feed back the missing field information to the large language model to regenerate the second candidate processing logic if the first comprehensive quality score does not reach the preset threshold. This process continues until the second comprehensive quality score recalculated based on the second candidate processing logic reaches the preset threshold and the current iteration number does not reach the preset maximum iteration number. Then, the corresponding second candidate processing logic is output as the target processing logic for encapsulating and processing the environmental observation data.
[0020] Another embodiment of the present invention provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the steps of the environmental observation data encapsulation method as described in the present invention.
[0021] Another embodiment of the present invention provides a computer-readable storage medium item, including: a stored computer program, which, when the computer program is running, controls the device where the computer-readable storage medium is located to perform the steps of the environmental observation data encapsulation method as described in the present invention. Attached Figure Description
[0022] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0023] Figure 1 This is a flowchart illustrating one embodiment of the environmental observation data encapsulation method provided in this application; Figure 2 This is a flowchart illustrating one embodiment of steps S201 to S203 provided in this application; Figure 3 This is a flowchart illustrating one embodiment of steps S301 to S302 provided in this application; Figure 4 This is a flowchart illustrating one embodiment of steps S401 to S403 provided in this application; Figure 5 This is a schematic diagram of the structure of one embodiment of the environmental observation data encapsulation system provided in this application; Figure 6 This is a schematic diagram of one embodiment of the system architecture provided in this application. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.
[0026] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.
[0027] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0028] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.
[0029] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).
[0030] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.
[0031] See Figure 1To improve the accuracy of environmental observation data encapsulation, an embodiment of the present invention provides a method for encapsulating environmental observation data, including steps S101 to S104: Step S101: Obtain the natural language task description and corresponding data pattern information for environmental observation data, parse the natural language task description and the data pattern information, and generate a structured requirement specification. In some embodiments, obtaining the natural language task description and corresponding data pattern information for environmental observation data involves two types of basic information: one is the data processing task description submitted by the user in natural language, and the other is the data pattern information of the environmental observation data itself to be processed. Specifically, the system obtains the data pattern information of the input data from object storage (OBS), message queue (Kafka), or relational database (RDS) by reading the configuration of the data source layer. This information includes field names, data types, and relationships between fields.
[0032] It should be noted that natural language task descriptions are usually proposed by researchers or data engineers based on specific business scenarios. For example, "group according to platform identifier, observed variables and platform type, complete the aggregation process in a 30-minute time window, and output statistical fields, range enhancement fields, metadata fields, quality fields and anomaly fields" or "automatically identify sensitive fields such as contact person name, contact number and internal device number, and generate sensitive tags, sensitive reasons and de-identification results".
[0033] It should be noted that in a single-table scenario, the system directly parses the complete schema information of the table, such as fields like "platform identifier, observation time, longitude, latitude, depth, pressure, observed value, and quality control code" in ocean observation data. In a multi-table scenario, the system simultaneously obtains schema information from both the master fact table and the auxiliary dimension tables. For example, it might use observation time-series data as the fact table and the platform metadata table as the dimension table, and identify the association key between the two tables (such as "platform identifier"). Additionally, the system can optionally receive a small sample of data (such as several rows of records) to aid in understanding the actual value characteristics of the fields and the distribution of the data.
[0034] Please refer to Figure 2 In some embodiments, parsing the natural language task description and the data pattern information to generate a structured requirement specification includes steps S201 to S203: Step S201: Based on the data pattern information, identify the key field types and field names in the environmental observation data, wherein the key field types include grouping key fields, time fields, numerical fields, metadata fields, quality fields, and sensitive fields; In some embodiments, after receiving the data pattern information of the input data, the system parses the field names and data types in the data pattern information, and determines the category of each key field according to a preset field type dictionary and rules. Grouping key fields are the identifiers used for grouping in the user task, such as "platform identifier," "observation variable," and "platform type," which the system directly marks as grouping keys. Time fields are usually identified by their field name (like "time" or "observation time") and data type (timestamp type), and are used later to define event time and time windows. Numerical fields require statistical calculations, such as longitude, latitude, depth, pressure, observed values, and quality control codes. The system automatically generates five types of statistical results for each numerical field: mean, standard deviation, minimum, maximum, and count. Metadata fields are descriptive information that does not require aggregation calculations and only needs to be transmitted as is or have a representative value retrieved, such as institutional information, document identifiers, and product identifiers. Quality fields mainly refer to control codes or status fields related to data quality, such as quality control codes. The system uses these to determine whether the data is qualified and calculates the pass rate and the anomaly rate. Sensitive fields need to be anonymized before publication, such as contact person's name, contact number, and internal device number. The system identifies them by matching field names or by user-initiated annotations.
[0035] Step S202: Preprocess the natural language task description to obtain the target task description, and extract data processing rules from the target task description, wherein the data processing rules include time window configuration, statistical caliber, quality judgment rules, anomaly detection rules, and desensitization rules; In some embodiments, the preprocessing of the natural language task description to obtain the target task representation includes: acquiring a domain terminology library corresponding to the domain to which the environmental observation data belongs; aligning the semantic information in the natural language task description using the domain terminology library to obtain a first processing result; and enhancing the text in the first processing result to obtain the target task description. Specifically, corresponding domain terminology libraries are prepared in advance for different domains (such as oceanography, ecology, and geology). Taking the ocean domain as an example, the terminology library contains professional terms such as "platform identifier," "observation variable," "quality control code," and "temperature-salinity profile," as well as their standard spellings. The system first loads the terminology library that matches the current data and then uses it to scan the paragraph written by the user. If the user says "grouped by buoy ID," but the standard field name is "platform identifier," the system automatically aligns it to the standard terminology; if the user's words are ambiguous, such as whether "depth" refers to water depth or instrument descent depth, the system will also correct it according to the terminology library. After this step, an intermediate text is obtained, in which ambiguous expressions have been replaced with standard terms. The system then enhances this intermediate text: on one hand, it completes implicit information based on lexical relationships (such as synonyms and hyponyms) in the terminology database. For example, if the user only says "calculate the average," the system automatically completes it to "calculate the average of each numerical field within each grouping window" based on the context. On the other hand, the system can also add structural tags to the enhanced text, such as marking which words are grouping conditions, which are time windows, and which are statistical functions. After these two steps, a target task description with unified terminology and precise expression is obtained.
[0036] By using a domain terminology database to perform semantic alignment and text enhancement on natural language task descriptions, ambiguity in domain terms can be eliminated, implicit semantics can be supplemented, and the generated target task descriptions can be made more in line with the expression habits of specific scientific fields, thereby improving the accuracy of requirement analysis and indirectly improving the accuracy of data encapsulation.
[0037] In some embodiments, data processing rules are extracted from the target task description. Specifically, when extracting data processing rules from the natural language task description, it is not a simple keyword matching process, but rather the business intent in the description is parsed into configuration items that the machine can execute, and the data processing rules are obtained.
[0038] In some embodiments, the specific methods for obtaining data processing rules include: (1) Time window configuration: If the description states "a time window of 30 minutes" or "aggregate every 30 minutes", the system extracts a window length of 30 minutes and a window type of rolling window (corresponding to "rolling window" in the stream computing engine). (2) Statistical scope: Depending on which fields the user requires to perform calculations, such as "calculate the average, standard deviation, minimum, maximum and count for each numerical field", the system generates these five statistical functions for each numerical field; if only "the mean of the statistical observations" is requested, then only the average is extracted. (3) Quality judgment rules: Depends on the quality control code field in the data model and the definition of "qualified" or "abnormal" in the description, such as "a quality control code of 0 or 1 indicates pass, the rest indicates fail", the system generates conditional judgment statements and ratio calculation formulas accordingly (quality pass ratio = number of pass records divided by total number of records). (4) Anomaly detection rules: For example, if the description requires "generating anomaly markers and anomaly reasons based on the quality control results and whether the statistical deviation within the window exceeds a preset multiple", the system will extract threshold conditions such as "exceeding 3 times the standard deviation" or "exceeding 2 times the mean" and associate them with the corresponding numerical fields. At the same time, it will also extract the concatenation method of the anomaly reason field - for example, recording which conditions were triggered in the reason. (5) Desensitization rules: If the description says "retain the first and last characters of the contact name and replace the middle characters", "retain the first three and last two digits of the contact number and replace the middle digits", "retain the first and last characters of the internal device number and replace the middle characters", the system will convert these natural language into specific string operation sequences: first use the "get string length" function to determine the length, then use the "substring" function to get the first and last digits, and fill the middle with mask characters using the "repeat" or "connect" function. If the description also implies a degradation rule of "replace all when the length is too short", the system will also extract it.
[0039] Step S203: The identified key field types, field names, and data processing rules are organized according to a preset format to generate the structured requirement specification.
[0040] In some embodiments, after the key field types and names, and data processing rules are extracted, the system organizes them into a structured requirement specification that can be directly read by a machine, using a strict "key-value pair" text format (i.e., JSON format). Specifically, the organization involves setting up dedicated arrays of field categories within the specification. For example, a "grouping key array" stores a list of grouping key field names, a "time field" stores time field names, a "numerical field array" stores the names of all numerical fields requiring statistical calculations, a "metadata field array" stores metadata field names, a "quality field array" stores quality field names, and a "sensitive field array" stores sensitive field names and their corresponding de-identification rules. For the data processing rules, a "window configuration" object is set, containing the window type (e.g., scrolling window) and window length; an "aggregation configuration" object is set, defining a list of statistical functions to be calculated for each numerical field; a "quality rule" object is set, describing the quality judgment conditions and grade classification methods; an "anomaly rule" object is set, describing the threshold and triggering conditions for anomaly detection; and a "de-identification rule" object is set, specifying a specific combination of de-identification functions for each sensitive field.
[0041] Here are a few examples: In a single-table aggregation scenario, the generated requirement specification should include at least a grouping key, a time field, seven numeric fields, three metadata fields, a window parameter (30 minutes), and a list of business enhancement fields. In a multi-table scenario, the specification will also include a "table information array" describing the roles, join keys, and join types of the fact table and dimension tables. In sensitive handling scenarios, the specification will explicitly require the output of which sensitive marker fields (e.g., "with name sensitive marker," "with phone number sensitive marker," "comprehensive sensitive marker") and de-identification result fields (e.g., "de-identified contact name").
[0042] By identifying key field types such as grouping keys, time, values, metadata, quality, and sensitivity, and extracting time windows, statistical definitions, quality and desensitization rules, the original data patterns and task requirements can be systematically and structurally mapped into the specification, avoiding field omissions or rule ambiguities, and laying a solid foundation for the accurate generation of subsequent processing logic.
[0043] Step S102: Input the structured requirement specification into a preset large language model to generate a first candidate processing logic for processing the environmental observation data; Please refer to Figure 3 In some embodiments, step S102 includes steps S301 to S302: Step S301: Obtain the data source access information of the environmental observation data, as well as the initial missing field information recorded in the previous round of verification feedback; In some embodiments, before code generation, data source access information and initial missing field information recorded in the previous round of validation feedback are obtained. The data access information tells the large language model where the data is read from, in what format, and how to connect. This information comes from the data source layer configuration: if the data is in object storage, the bucket path and file format (e.g., PAL format) are needed; if the data comes in real-time from a message queue, the topic name and access address are needed; if the data is in a relational database, the database name, table name, and connection address are needed. The initial missing field information recorded in the previous round of validation feedback is only available during iterative optimization—it is empty in the first round of code generation because it doesn't include the previous round; however, in the second round and subsequent rounds, the system will return the list of missing fields and syntax error information found in the previous round of field validation intact. For example, if the previous check found that the "exception reason" field was missing from the output, the missing field information in this round will explicitly state "missing exception reason field". The system prepares these two things, waiting to be fed into the large language model along with the structured requirements specification.
[0044] Step S302: The structured requirement specification, the data source access information, and the initial missing field information are input into the large language model so that the large language model can generate candidate processing logic that includes a data source definition part, a data processing operation part, and a result output definition part.
[0045] In some embodiments, the structured requirements specification, the data source access information, and the initial missing field information are input together into the large language model. After receiving these, the large language model generates a complete candidate processing logic. This logic must contain three parts: The first part is the data source definition, which describes the table structure of the input data, including field names, field types, and connector configuration—such as which connector (file system, message queue, or database), file path, file format (Para, CSV, or JSON), and watermark definition (to handle out-of-order data arrival). The second part is the data processing operation part, which is the core. It clearly defines how to read data from the input table, perform window aggregation (e.g., scrolling window), perform various statistical calculations (mean, standard deviation, minimum, maximum, count), and handle multi-table joins (e.g., left join) or desensitization of sensitive fields (e.g., string truncation, mask character concatenation). The third part is the result output definition, which describes the field structure of the output table and where the processed results are written (e.g., to another storage location or message queue). The generated candidate processing logic is a complete, directly executable code snippet.
[0046] It should be noted that if there is missing field information in the previous round, the large language model will pay special attention to filling in those missing fields during generation.
[0047] In this way, when generating candidate processing logic, the data source access information and the missing field information from the previous round are input simultaneously. This guides the large language model to supplement missing fields and correct errors in a targeted manner, avoiding the recurrence of similar defects. It achieves precise correction based on feedback, significantly improving the efficiency of iterative optimization and the accuracy of final encapsulation.
[0048] Step S103: parse the input fields and output fields involved in the first candidate processing logic, and compare the input fields and output fields with the expected fields in the structured requirement specification to generate a first comprehensive quality score and missing field information; Please refer to Figure 4 In some embodiments, step S103 includes steps S401 to S403: Step S401: Check whether the first candidate processing logic includes the data source definition part, the data processing operation part, and the result output definition part, and check whether the functions and keywords used conform to the preset syntax rules; In some embodiments, after receiving the first candidate processing logic generated by the large model, the system first checks whether the first candidate processing logic contains three necessary parts: a data source definition, a data processing operation, and a result output definition. If any part is missing, such as having only data processing operations without defining where the input table comes from, or forgetting to write the structure of the output table, the system will directly determine that it fails. After the structure check is passed, the system checks whether the functions and keywords used in it conform to the preset syntax rules. Specifically, it checks several things: First, whether the aggregate functions are in the whitelist, such as only allowing the average (AVG), sample standard deviation (STDDEV_SAMP), minimum (MIN), maximum (MAX), and count (COUNT). If a function outside the whitelist is used, an error is reported; second, whether the time window is written correctly, such as the complete syntax of the rolling window (TUMBLE); third, whether the definition of the watermark (WATERMARK) conforms to the specifications, and the out-of-order tolerance time interval cannot be missing; fourth, whether key field names conflict with system reserved words and whether they are escaped with backticks. Only after all these checks are passed can the system proceed to the next step.
[0049] Step S402: If the check passes, the data processing operation part in the first candidate processing logic is parsed to obtain the data source reference and the result generation expression, and the input field and the output field are extracted from the data source reference and the result generation expression respectively. In some embodiments, after both structural and syntax checks pass, the data processing operation portion of the first candidate processing logic is parsed. This portion is typically a query statement (SELECT ... FROM ... WHERE ... GROUP BY ...). The system then finds the data source reference within this statement, specifically the table name following "FROM". Based on this table name, it queries the previously defined data source definition section and extracts all referenced input fields. Simultaneously, the system extracts output fields from the result generation expressions and parses these expressions one by one, extracting the name of each field to be output. Thus, the system obtains two lists: one for input fields and another for output fields.
[0050] It's important to note that the result generation expression is typically the long string following "SELECT," which may contain the original field names, statistical functions (such as average and minimum values), calculation expressions (such as time difference and string concatenation), desensitization functions (such as substring extraction and masking), and aliases. For example, if the expression is "COUNT(value) AS sample_count," the system will extract the "sample_count" field as the output field; if it's "AVG(temperature)," the system will extract the corresponding average value field.
[0051] Step S403: The ratio of the number of extracted input fields to the number of expected input fields in the structured requirement specification is taken as the first field coverage rate, and the ratio of the number of output fields to the number of expected output fields in the structured requirement specification is taken as the second field coverage rate. The first comprehensive quality score and the missing field information are generated based on the first field coverage rate and the second field coverage rate.
[0052] In some embodiments, after obtaining the list of input fields and the list of output fields, the number of extracted input fields is divided by the expected number of input fields in the specification to obtain a ratio, which is the first field coverage. Then, the number of extracted output fields is divided by the expected number of output fields in the specification to obtain the second field coverage.
[0053] In some embodiments, generating the first comprehensive quality score and the missing field information based on the first field coverage and the second field coverage includes: recording the first field in the expected input field that does not appear in the extracted input field, and the second field in the expected output field that does not appear in the extracted output field, together as the missing field information; Based on the field function types defined in the structured requirements specification, corresponding weight coefficients are assigned to the first field coverage rate and the second field coverage rate, respectively. The first field coverage rate and the second field coverage rate are then weighted and summed according to the weight coefficients to obtain the first comprehensive quality score.
[0054] In some embodiments, after calculating the first and second field coverage rates, all fields in the expected input fields but not found in the candidate processing logic's input field list, and fields in the expected output fields but not found in the output field list, are recorded and compiled into a missing field list. Each item in this list clearly indicates which field is missing and its type (e.g., grouping key, time field, numeric field, or sensitive marker field), facilitating its later transmission back to the larger model for targeted inclusion. Simultaneously, different weights are assigned according to the field function types defined in the specification, and a formula is selected based on the corresponding scenario. The calculated first and second field coverage rates are then substituted into the formula, and a weighted sum is obtained to obtain the first comprehensive quality score.
[0055] In some embodiments, the formula for a typical aggregation scenario (such as statistics based on a time window) is: ; In the formula, This is a comprehensive quality score for typical aggregation scenarios. For the coverage of the first field, For the coverage of the second field, For aggregation coverage, For metadata coverage, For quality and anomaly coverage, the calculation method is to divide the actual number generated by the expected number.
[0056] In some embodiments, the formula for sensitive processing scenarios (such as desensitization) is: ; In the formula, A comprehensive quality score for sensitive processing scenarios. For the coverage of the first field, For the second output coverage, Metadata coverage.
[0057] It should be noted that the actual number of aggregated fields (such as average, standard deviation, minimum, maximum, count, etc.) generated in the candidate processing logic is divided by the total number of aggregated fields expected to be generated in the specification. Metadata coverage measures whether the transparent or representative value fields of metadata fields are complete. Metadata fields are typically descriptive information that does not undergo aggregation calculations and only needs to be retained as is or have a representative value, such as organization name, document identifier, product identifier, etc. This coverage is equal to the actual number of generated metadata fields divided by the expected number of metadata fields.
[0058] By recording the missing input and output fields in the expected fields and assigning weights according to the field function type, a weighted sum is obtained to obtain a comprehensive quality score. This makes the score more scientific and reasonable, accurately reflecting the completeness of the encapsulated logic and guiding the iteration towards optimization in the direction of completing key fields.
[0059] By performing structural integrity checks and function and keyword syntax validations on the candidate processing logic, and by extracting input / output fields and comparing them with the expected fields in the specification to calculate field coverage, dual verification of syntax and business semantics is achieved. This enables precise identification of missing fields or type mismatch issues, providing a reliable basis for quantitative evaluation.
[0060] In some embodiments, the present invention further includes: if the check fails, generating syntax error information and feeding the syntax error information as part of the missing field information back to the large language model to update the large language model. Specifically, if the first candidate processing logic fails the structure check and syntax rule check in step S401, the system will not discard it directly, but will first record where the problem occurred. For example, the check may find that the data source definition is missing, or the data processing operation is incomplete, or a function not in the whitelist is used (the whitelist only allows five types of aggregation functions: mean, sample standard deviation, minimum, maximum, and count), or the scrolling window is missing a time interval, the watermark definition forgets to set the out-of-order tolerance time, or the keyword conflicts with a system reserved word without being escaped with backticks. The system organizes these error messages into one or more syntax error prompts, such as "missing result output definition" or "using the non-whitelist function 'median'". Then, the system combines this batch of syntax error information as part of the missing field information with the previously calculated missing fields (such as the omitted "abnormal reason" field) and packages it back to the large language model. After receiving this feedback, the large language model will correct the code based on these specific error messages when generating the second candidate processing logic in the next round. For example, it may add missing output definitions or replace disallowed functions with allowed combinations. This process of modification continues round after round until the structure is complete and the syntax is compliant.
[0061] In this way, when the structure or syntax check fails, the specific syntax error information is fed back to the large language model as part of the missing field. This allows the model to clearly know the defects at the code format level, so that it can specifically correct the syntax problems in the next generation and avoid the encapsulated logic from failing to execute due to syntax errors.
[0062] Step S104: If the first comprehensive quality score does not reach the preset threshold, the missing field information is fed back to the large language model to regenerate the second candidate processing logic until the second comprehensive quality score recalculated based on the second candidate processing logic reaches the preset threshold and the current iteration number does not reach the preset maximum iteration number. Then, the corresponding second candidate processing logic is output as the target processing logic for encapsulating and processing the environmental observation data.
[0063] In some embodiments, after obtaining the first comprehensive quality score, it is compared with a pre-set threshold. If the score is higher than the preset threshold (e.g., 80 points), the system uses this version of the candidate processing logic as the final result. If the score is lower than the preset threshold (e.g., 80 points), it indicates that the generated code still has problems, either due to incorrect syntax or missing fields. At this time, the system uses the previously compiled missing field information (e.g., "missing exception reason field" or "missing comprehensive sensitivity flag") as error correction prompts, along with the structured requirement specification and data source access information, and feeds it back to the large language model. The large language model receives this missing list and, combined with the previous structured requirement specification and data source information, regenerates a second set of candidate processing logic. The system then performs syntax checks and field coverage scoring on the second set of logic in the same way, calculating a new comprehensive quality score. If the second set of logic scores 80 points and the current number of iterations has not exceeded the maximum limit (e.g., 3 rounds), then this second candidate processing logic is output as the final target processing logic. The output includes not only a runnable code snippet but also a quality report containing the final score and historical records for each round. The historical records include the complete code generated in each round, the score, any errors reported, and any missing fields—all recorded for future auditing and review. If the score is still below 80 points in the second round, feedback, regeneration, and re-evaluation are conducted again, until the third round. After three rounds, regardless of whether the score reaches 80 points, the system outputs the logic generated in the final round as the result. Once this final target processing logic is obtained, it can be directly used to encapsulate and process environmental observation data, such as performing aggregation statistics by time window or adding mosaic effects to sensitive fields, and then writing the processed data to the designated storage location.
[0064] This invention, through acquiring natural language task descriptions and data pattern information and parsing them to generate structured requirement specifications, transforms ambiguous and unstructured user intentions into an intermediate representation that machines can accurately understand and execute. This eliminates the interference of natural language ambiguity on the generation of subsequent processing logic, ensuring the accuracy and completeness of requirement expression from the source. Secondly, the structured requirement specification is input into a large language model to generate first candidate processing logic. This ensures that the code generation process no longer relies on one-off, unconstrained open generation, but is based on explicit specification constraints, significantly reducing the probability of arbitrary model speculation or omission of key processing steps, and improving the consistency between the generated logic and the original requirements. Next, by parsing the input and output fields involved in the candidate processing logic and comparing them one by one with the expected fields in the structured requirement specification, a first comprehensive quality score and missing field information are generated. This achieves automated, fine-grained quantitative evaluation of the generated logic. This field-level comparison mechanism can accurately identify specific defects such as missing fields, type mismatches, or incomplete logic, avoiding the bias and subjective errors of evaluations based solely on syntax checking or manual inspection, thus providing an objective and quantifiable basis for subsequent corrections. Finally, when the overall quality score fails to reach the preset threshold, the missing field information is fed back to the large language model to regenerate the second candidate processing logic. This process is iterated repeatedly until the score meets the standard or the maximum number of iterations is reached, forming a closed-loop optimization mechanism of "generation-verification-feedback-correction". This mechanism can specifically supplement missing fields, correct erroneous logic, and continuously improve the quality of the processing logic within a limited number of rounds until the preset accuracy requirements are met. In summary, the technical solution of this invention, through the systematic combination of structured specification constraints, field-level verification scoring, and iterative feedback correction, effectively overcomes the problem of inaccurate encapsulation caused by natural language understanding bias, field omissions, and logical deficiencies in the prior art, and significantly improves the final accuracy of environmental observation data encapsulation.
[0065] To illustrate the effectiveness of this invention, please refer to Table 1, which is a comparison table of experimental results for technical solutions based on a multi-stage Workflow engine: Where: * represents the experimental result submitted after one modification; — represents the result submitted after two modifications but still failed; the rest represent the result submitted and run successfully on the first attempt; bold indicates the best result; evaluation metric: field completeness rate = number of generated fields / expected number of fields × 100%.
[0066] like Figure 5 As shown, based on the above method embodiments, corresponding apparatus embodiments are provided; An embodiment of the present invention provides an environmental observation data encapsulation system, comprising: The first module 100 is used to acquire a natural language task description for environmental observation data and the corresponding data pattern information, parse the natural language task description and the data pattern information, and generate a structured requirement specification. The second module 200 is used to input the structured requirement specification into a preset large language model to generate a first candidate processing logic for processing the environmental observation data. The third module 300 is used to parse the input fields and output fields involved in the first candidate processing logic, and compare the input fields and output fields with the expected fields in the structured requirement specification, respectively, to generate a first comprehensive quality score and missing field information; The fourth module 400 is used to feed back the missing field information to the large language model to regenerate the second candidate processing logic if the first comprehensive quality score does not reach the preset threshold. The second comprehensive quality score recalculated based on the second candidate processing logic reaches the preset threshold and the current iteration number does not reach the preset maximum iteration number. Then, the corresponding second candidate processing logic is output as the target processing logic for encapsulating and processing the environmental observation data.
[0067] like Figure 6 As shown, based on the above method implementation examples, a corresponding system architecture is provided, which consists of five layers: data source layer, streaming computing engine layer (Flink), multi-stage Workflow engine layer, resource scheduling layer (Kubernetes), and application layer. The core is the multi-stage Workflow engine, which performs closed-loop processing around the intermediate object `requirement_spec` (structured requirement specification).
[0068] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention, and can implement the environmental observation data encapsulation method provided by any of the above-described method embodiments of the present invention.
[0069] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.
[0070] Based on the above embodiments of the environmental observation data encapsulation method, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the environmental observation data encapsulation method of any embodiment of the present invention.
[0071] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.
[0072] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.
[0073] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.
[0074] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the environmental observation data encapsulation method described in any of the above-described method embodiments of the present invention.
[0075] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0076] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A method for encapsulating environmental observation data, characterized in that, include: Obtain a natural language task description and corresponding data pattern information for environmental observation data, parse the natural language task description and the data pattern information, and generate a structured requirement specification; The structured requirement specification is input into a preset large language model to generate a first candidate processing logic for processing the environmental observation data; The input and output fields involved in the first candidate processing logic are parsed, and the input and output fields are compared with the expected fields in the structured requirement specification to generate a first comprehensive quality score and missing field information. If the first comprehensive quality score does not reach the preset threshold, the missing field information is fed back to the large language model to regenerate the second candidate processing logic. This continues until the second comprehensive quality score recalculated based on the second candidate processing logic reaches the preset threshold and the current iteration number does not reach the preset maximum iteration number. Then, the corresponding second candidate processing logic is output as the target processing logic for encapsulating and processing the environmental observation data.
2. The method for encapsulating environmental observation data according to claim 1, characterized in that, The step of parsing the natural language task description and the data pattern information to generate a structured requirement specification includes: Based on the data pattern information, the key field types and field names in the environmental observation data are identified, wherein the key field types include grouping key fields, time fields, numerical fields, metadata fields, quality fields, and sensitive fields; The natural language task description is preprocessed to obtain the target task description, and data processing rules are extracted from the target task description. The data processing rules include time window configuration, statistical criteria, quality judgment rules, anomaly detection rules, and de-identification rules. The identified key field types, field names, and data processing rules are organized according to a preset format to generate the structured requirement specification.
3. The method for encapsulating environmental observation data according to claim 2, characterized in that, The preprocessing of the natural language task description to obtain the target task representation includes: Obtain a domain terminology database corresponding to the domain to which the environmental observation data belongs; The semantic information in the natural language task description is aligned using the domain terminology database to obtain a first processing result; The text in the first processing result is enhanced to obtain the target task description.
4. The method for encapsulating environmental observation data according to claim 1, characterized in that, The step of inputting the structured requirement specification into a preset large language model to generate a first candidate processing logic for processing the environmental observation data includes: Obtain the data source access information of the environmental observation data, as well as the initial missing field information recorded in the previous round of verification feedback; The structured requirement specification, the data source access information, and the initial missing field information are input into the large language model, so that the large language model can generate candidate processing logic that includes a data source definition part, a data processing operation part, and a result output definition part.
5. The method for encapsulating environmental observation data according to claim 4, characterized in that, The process involves parsing the input and output fields of the first candidate processing logic, comparing the input and output fields with the expected fields in the structured requirement specification, and generating a first comprehensive quality score and missing field information, including: Check whether the first candidate processing logic includes the data source definition part, the data processing operation part, and the result output definition part, and check whether the functions and keywords used conform to the preset syntax rules; If the check passes, the data processing operation part in the first candidate processing logic is parsed to obtain the data source reference and the result generation expression, and the input field and the output field are extracted from the data source reference and the result generation expression respectively. The ratio of the number of extracted input fields to the number of expected input fields in the structured requirement specification is used as the first field coverage rate, and the ratio of the number of output fields to the number of expected output fields in the structured requirement specification is used as the second field coverage rate. The first comprehensive quality score and the missing field information are generated based on the first field coverage rate and the second field coverage rate.
6. The method for encapsulating environmental observation data according to claim 5, characterized in that, Also includes: If the check fails, a syntax error message is generated and fed back to the large language model as part of the missing field information to update the large language model.
7. The method for encapsulating environmental observation data according to claim 5, characterized in that, The process of generating the first comprehensive quality score and the missing field information based on the first field coverage and the second field coverage includes: The first field in the expected input field that does not appear in the extracted input field, and the second field in the expected output field that does not appear in the extracted output field, are jointly recorded as the missing field information; Based on the field function types defined in the structured requirements specification, corresponding weight coefficients are assigned to the first field coverage rate and the second field coverage rate, respectively. The first field coverage rate and the second field coverage rate are then weighted and summed according to the weight coefficients to obtain the first comprehensive quality score.
8. A system for encapsulating environmental observation data, characterized in that, include: The first module is used to acquire natural language task descriptions and corresponding data pattern information for environmental observation data, parse the natural language task descriptions and data pattern information, and generate structured requirement specifications. The second module is used to input the structured requirement specification into a preset large language model to generate a first candidate processing logic for processing the environmental observation data. The third module is used to parse the input fields and output fields involved in the first candidate processing logic, and compare the input fields and output fields with the expected fields in the structured requirement specification to generate a first comprehensive quality score and missing field information. The fourth module is used to feed back the missing field information to the large language model to regenerate the second candidate processing logic if the first comprehensive quality score does not reach the preset threshold. This process continues until the second comprehensive quality score recalculated based on the second candidate processing logic reaches the preset threshold and the current iteration number does not reach the preset maximum iteration number. Then, the corresponding second candidate processing logic is output as the target processing logic for encapsulating and processing the environmental observation data.
9. A terminal device, characterized in that, include: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the environmental observation data encapsulation method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, include: A stored computer program, wherein, when the computer program is executed, the device containing the computer-readable storage medium controls the execution of the steps of the environmental observation data encapsulation method as described in any one of claims 1-7.