Data aggregation method, apparatus, device, medium, and program product

Abstract

The embodiment of the specification discloses a data aggregation method, device, equipment, medium and program product, which can improve the efficiency and accuracy of data aggregation. Wherein, the method comprises: acquiring data flow transfer files of multiple different formats; performing semantic analysis on the text content of each data flow transfer file by a large language model to obtain at least one data flow transfer record corresponding to each data flow transfer file; checking each data flow transfer record based on a preset data checking rule; the data checking rule comprises a data consistency checking rule, a data repeatability checking rule and a data continuity checking rule; and generating a data aggregation file according to each data flow transfer record that passes the check.

Description

Technical Field

[0001] This specification relates to the field of data processing technology, and in particular to a data aggregation method, apparatus, device, medium, and program product. Background Technology

[0002] In multi-source data aggregation scenarios, data from different data sources often have different formats. Related technologies typically employ manual sorting or pre-set template matching to aggregate data of different formats.

[0003] However, with the increase in data sources and the frequent changes in data formats, the above-mentioned data aggregation methods face challenges in terms of efficiency and accuracy, and a more efficient and accurate data aggregation method is needed. Summary of the Invention

[0004] This specification provides a data aggregation method, apparatus, device, medium, and program product that can improve the efficiency and accuracy of data aggregation.

[0005] Firstly, embodiments of this specification provide a data processing method, including: Acquire data stream files in various formats; By using a large language model to perform semantic parsing on the text content of each data transfer file, at least one data transfer record corresponding to each data transfer file can be obtained. Based on preset data verification rules, each data flow record is verified; the data verification rules include data consistency verification rules, data duplication verification rules, and data continuity verification rules. A data aggregation file is generated based on each verified data flow record.

[0006] In one possible implementation, before semantically parsing the text content of each data transfer file using a large language model to obtain at least one data transfer record corresponding to each data transfer file, the method further includes: Identify the file format type of each data stream file; If the file format of any data stream file is identified as an image type or a portable document type, optical character recognition is performed on any data stream file to obtain the text content of any data stream file. If the file format of any data transfer file is identified as a spreadsheet, cell text extraction is performed on any data transfer file to obtain the text content of the data transfer file.

[0007] In one possible implementation, the text content includes at least one data flow detail; Semantic parsing of the text content of each data transfer file is performed using a large language model to obtain at least one data transfer record corresponding to each data transfer file, including: Using a large language model, extract the data flow elements of each data flow detail in each data flow file; Based on the data flow elements of each data flow detail, obtain the data flow direction corresponding to each data flow detail; Based on the data flow direction corresponding to each data flow detail, determine the data flow participants in each data flow detail; By using data flow elements, data flow direction, and data flow participants as target fields, at least one data flow record is generated for each data flow file.

[0008] In one possible implementation, data flow elements include data flow time, data flow volume, data remaining volume, data flow summary, and data flow identifier; Based on the data flow elements of each data flow detail, obtain the data flow direction corresponding to each data flow detail, including: Sort each data flow detail in ascending order according to its data flow time. Iterate through each sorted data flow detail in sequence. For the current data flow detail, obtain the first reference data balance, the first data flow volume, and the first data balance. The first reference data balance is the data balance of the previous data flow detail. If the first reference data margin, the first data flow volume, and the first data margin satisfy the preset operation relationship, the data flow direction corresponding to the current data flow detail is determined according to the preset operation relationship; If the first reference data margin, the first data flow volume, and the first data margin do not satisfy the preset operation relationship, determine whether the data flow summary contains data flow direction semantics. When the data flow summary contains the semantics of data flow direction, the data flow direction corresponding to the current data flow detail is determined based on the data flow summary. If the data flow summary does not contain the semantics of the data flow direction, the data flow direction corresponding to the current data flow detail is determined based on the data flow identifier.

[0009] In one possible implementation, if the first reference data margin, the first data flow volume, and the first data margin satisfy a preset calculation relationship, the data flow direction corresponding to the current data flow detail is determined according to the preset calculation relationship, including: If the sum of the first reference data reserve and the first data flow volume equals the first data reserve, the data flow direction corresponding to the current data flow detail is determined to be data transfer in. If the difference between the first reference data margin and the first data flow volume is equal to the first data margin, the data flow direction corresponding to the current data flow detail is determined to be data outflow.

[0010] In one possible implementation, the participants in data flow include the data flow owner and the data flow related parties; Based on the data flow direction corresponding to each data flow detail, determine the data flow participants for each data flow detail, including: If the data flow direction corresponding to any data flow detail is data transfer in, the data transfer recipient in any data flow detail shall be determined as the data flow owner, and the data transfer sender in any data flow detail shall be determined as the data flow related party. If the data flow direction corresponding to any data flow detail is data outflow, the data outflow party in any data flow detail is determined as the data flow owner, and the data inflow receiving party in any data flow detail is determined as the data flow related party.

[0011] In one possible implementation, each data flow record is validated based on preset data validation rules, including: Based on data consistency verification rules, verify whether the values ​​of each field in any data flow record match the text content in the corresponding data flow file; Based on the data duplication verification rules, check whether any data flow record is duplicated with another data flow record that has already been generated; Based on the data continuity verification rules, the data balance of any data flow record is verified to meet the flow continuity conditions.

[0012] In one possible implementation, based on data continuity verification rules, the data remaining amount of any data flow record is checked to see if it meets the flow continuity condition, including: Obtain the second reference data balance, second data flow volume, and second data balance for any data flow record; the second reference data balance is the data balance of the previous data flow record for any data flow record; the previous data flow record for any data flow record is determined based on the ascending order of data flow time for each data flow record. When the sum of the second reference data reserve and the second data flow volume equals the second data reserve, or when the difference between the second reference data reserve and the second data flow volume equals the second data reserve, the data reserve of any data flow record is checked to see if it meets the flow continuity condition based on the data flow direction of any data flow record. If the sum of the second reference data reserve and the data flow volume is not equal to the second data reserve, and the difference between the second reference data reserve and the second data flow volume is not equal to the second data reserve, then the data reserve of any data flow record does not meet the flow continuity condition.

[0013] In one possible implementation, when the sum of the second reference data reserve and the second data flow volume equals the second data reserve, or the difference between the second reference data reserve and the second data flow volume equals the second data reserve, the data reserve of any data flow record is checked against the data flow direction of any data flow record to determine whether it meets the continuity condition, including: If the sum of the second reference data reserve and the second data flow volume equals the second data reserve, determine whether the data flow direction of any data flow record is data transfer in; if yes, determine that the data reserve of any data flow record meets the flow continuity condition; if no, determine that the data reserve of any data flow record does not meet the flow continuity condition. If the difference between the second reference data reserve and the second data flow volume is equal to the second data reserve, determine whether the data flow direction of any data flow record is data out. If yes, determine that the data reserve of any data flow record meets the flow continuity condition. If no, determine that the data reserve of any data flow record does not meet the flow continuity condition.

[0014] In one possible implementation, after verifying each data flow record based on preset data verification rules, the method further includes: In response to the existence of abnormal data transfer records that fail verification, determine the abnormal information associated with the abnormal data transfer records; By using a large language model to correct abnormal data flow records based on abnormal information, corrected data flow records are obtained. The corrected data flow record is used as the new data flow record, and the steps of verifying each data flow record based on the preset data verification rules are executed again until each data flow record passes the verification or the preset number of iterations is reached.

[0015] Secondly, embodiments of this specification provide a data aggregation device, including: The acquisition module is used to acquire data stream files in various formats. The parsing module is used to perform semantic parsing on the text content of each data transfer file using a large language model, so as to obtain at least one data transfer record corresponding to each data transfer file. The verification module is used to verify each data flow record based on preset data verification rules; the data verification rules include data consistency verification rules, data repeatability verification rules, and data continuity verification rules. The generation module is used to generate a data aggregation file based on each data flow record that has passed verification.

[0016] Thirdly, embodiments of this specification provide an electronic device, including: a processor and a memory; the memory stores a computer program, and the processor executes the computer program to implement the method steps provided in the first aspect of embodiments of this specification.

[0017] Fourthly, embodiments of this specification provide a computer storage medium storing multiple instructions adapted for loading by a processor and executing the method steps provided in the first aspect of embodiments of this specification.

[0018] Fifthly, embodiments of this specification provide a computer program product, including a computer program; when the computer program is executed by a processor, it implements the method steps provided in the first aspect of embodiments of this specification.

[0019] The aforementioned data aggregation methods, devices, electronic equipment, computer storage media, and computer program products, in the file parsing stage, acquire data flow files of various formats and use a large language model to perform semantic parsing on the text content of each data flow file. This adaptively identifies text content with different structural organizations, field definitions, and encoding methods, and extracts data flow records with a unified structure, achieving automated and unified parsing of heterogeneous data flow files, thus improving the efficiency of data aggregation. In the verification and aggregation stage, based on preset data verification rules, data consistency, data repeatability, and data continuity checks are performed on the parsed data flow records. A data aggregation file is generated based on the verified data flow records, achieving multi-dimensional automatic verification of the parsing results and reliable generation of the data aggregation file, thus improving the accuracy of data aggregation. Throughout the entire data aggregation process, by combining the semantic parsing capabilities of the large language model with preset multi-dimensional data verification rules, automated parsing, verification, and integration of heterogeneous data flow files are achieved, effectively improving the efficiency and accuracy of data aggregation. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram illustrating the implementation process of a data processing method provided in related technologies; Figure 2 A schematic diagram illustrating the implementation process of a data processing method provided for an exemplary embodiment of this specification; Figure 3 A schematic diagram illustrating the application environment of a data processing method provided as an exemplary embodiment of this specification; Figure 4 A flowchart illustrating a data processing method provided for an exemplary embodiment of this specification; Figure 5 A flowchart illustrating another data processing method provided as an exemplary embodiment of this specification; Figure 6 A flowchart illustrating yet another data processing method provided as an exemplary embodiment of this specification; Figure 7 A schematic diagram of the structure of a data processing apparatus provided for an exemplary embodiment of this specification; Figure 8 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this specification. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this specification clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this specification.

[0023] In the description of this specification, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of these terms in this specification based on the specific circumstances. Furthermore, in the description of this specification, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0024] In multi-source heterogeneous data processing, data aggregation is typically used to build a unified data view for subsequent data analysis and management. For example, in practical applications, different data providers can periodically provide data flow files to the data aggregator; the data aggregator collects and organizes these data flow files from different data providers, aggregating them into a unified data set for subsequent data processing.

[0025] In related technologies, the implementation process of a data aggregation method is as follows: Figure 1 As shown. Please see below. Figure 1 After collecting data flow files from different data providers, the data aggregator performs format unification, data cleaning, and classification and labeling on the multi-source heterogeneous data flow files through manual file sorting to obtain data flow records. Then, the data flow records are manually verified, and the manually verified data flow records are manually uploaded to the system for subsequent data analysis and data management.

[0026] Understandably, because data stream files output by different data providers typically differ in file format, structure, and field definitions, and with the increase in data sources and frequent changes in data formats, Figure 1 The data aggregation scheme shown has low efficiency and accuracy.

[0027] To address the aforementioned issues, this specification provides a data aggregation method, such as... Figure 2 As shown. Please see below. Figure 2 : During the file parsing phase, the data aggregator collects various data flow files in different formats provided by data providers, identifies the file format type of each data flow file, and performs automated format parsing processing such as optical character recognition or cell text extraction on each data flow file according to the file format type. The text content is extracted and split into at least one data flow detail. Then, a large language model is used to perform semantic parsing on each data flow detail to generate a unified structured data flow record.

[0028] During the verification and aggregation phase, the data aggregator performs data consistency, duplication, and continuity checks on the generated data flow records based on preset data verification rules, and determines whether any abnormal data flow records fail verification. In response to the existence of abnormal data flow records that fail verification, the data aggregator identifies the associated anomaly information and corrects the abnormal data flow records based on this anomaly information using a large language model, resulting in corrected data flow records. These corrected data flow records are then used as new data flow records, and the steps for verifying each data flow record based on the preset data verification rules are executed again until each data flow record passes verification or reaches the preset number of iterations. For records that still fail after reaching the iteration limit, the data aggregator can mark them as records awaiting review for manual verification. Finally, the data aggregator generates a data aggregation file based on the verified data flow records and uploads the data aggregation file to the data storage system for subsequent data analysis and management.

[0029] Figure 2 In the data aggregation scheme shown, during the data processing stage, the data aggregator acquires data flow files in various formats and performs automated format parsing based on the file format type of each data flow file. It then uses a large language model to perform semantic analysis on the text content obtained from the format parsing, adaptively identifying text content with different structural organization, field definitions, and encoding methods, and extracting data flow records with a unified structure. This achieves automated and unified parsing of heterogeneous data flow files, improving the efficiency of data aggregation. During the verification and aggregation stage, the data aggregator performs data consistency, data duplication, and data continuity checks on the parsed data flow records based on preset data verification rules. Abnormal data flow records that fail the checks are corrected, and a data aggregation file is generated based on the verified data flow records. This achieves multi-dimensional automatic verification of the parsing results and reliable generation of the data aggregation file, improving the accuracy of data aggregation.

[0030] The entire data aggregation process combines the semantic parsing capabilities of a large language model with pre-defined multi-dimensional data verification rules to achieve automated parsing, verification, and integration of multi-source heterogeneous data transfer files, effectively improving the efficiency and accuracy of data aggregation.

[0031] It is worth noting that the data aggregation method provided in the embodiments of this specification can be adapted to various data aggregation scenarios. For example: In logistics scheduling scenarios, multiple data sources correspond to different logistics service providers, and the logistics data transfer files (such as inbound and outbound records) exported by each service provider have different formats. After receiving logistics data transfer files of various formats from multiple service providers, the data aggregation server uses a large language model to semantically parse the text content of each file, obtaining at least one logistics data transfer record (such as inbound and outbound details) for each file. Based on preset data verification rules, each logistics data transfer record is verified. Based on the verified records, a logistics data aggregation file (such as an inbound and outbound aggregation record) is generated for use in logistics progress tracking, item verification, and other processing.

[0032] In network monitoring scenarios, multiple data sources correspond to different server nodes, and the network data flow files (such as network traffic logs) exported by each server node have different formats. After receiving network data flow files of various formats from multiple server nodes, the data aggregation server performs semantic parsing on the text content of each network data flow file using a large language model to obtain at least one network data flow record (such as a network traffic record) corresponding to each network data flow file. Based on preset data verification rules, each network data flow record is verified. Based on the verified network data flow records, a network data aggregation file (such as a network traffic aggregation log) is generated for use in network security risk assessment, network status monitoring, and other processing.

[0033] In asset management scenarios, multiple data sources correspond to different asset custodians, and the asset data flow files (such as transaction flow reports) exported by each custodian have different formats. After receiving asset data flow files of various formats from multiple asset custodians, the data aggregation server performs semantic parsing of the text content of each asset data flow file using a large language model to obtain at least one asset data flow record (such as transaction details) corresponding to each asset data flow file. Based on preset data verification rules, each asset data flow record is verified. Based on the verified asset data flow records, an asset data aggregation file (such as a transaction flow aggregation report) is generated for use in asset liquidity monitoring, accounting reconciliation, and other processing.

[0034] The data aggregation method provided in the embodiments of this specification can be applied to, for example... Figure 3 In the application environment shown, multiple data source terminals 10 and data aggregation terminals 20 communicate with the data aggregation server 30 via a communication network. The data storage system may include, but is not limited to, the data flow files that the server 30 needs to process and the generated data aggregation files. The data storage system can be integrated on the server 30 or placed on the cloud or other network servers.

[0035] In some possible embodiments, multiple data source terminals 10, in response to a data submission operation, upload data stream files of various different formats to the data aggregation server 30. The data aggregation server 30 receives the data stream files of various different formats and stores them in the data storage system.

[0036] In response to the data aggregation operation, the data aggregation terminal 20 generates a data aggregation request and sends the data aggregation request to the data aggregation server 30; wherein, the data aggregation request carries data aggregation information, including the source information and / or the identification information of the file to be aggregated.

[0037] The data aggregation server 30 receives a data aggregation request sent by the data aggregation terminal 20. In response to the data aggregation request, it loads various data flow files in different formats from the data storage system according to the data aggregation information carried in the data aggregation request. It performs semantic parsing on the text content of each data flow file using a large language model to obtain at least one data flow record corresponding to each data flow file. Based on preset data verification rules, it verifies each data flow record. The data verification rules include data consistency verification rules, data repeatability verification rules, and data continuity verification rules. Based on each verified data flow record, it generates a data aggregation file. The data aggregation file is stored in the data storage system for subsequent processing modules to call.

[0038] Understandably, the multiple data source terminals 10 and data aggregation terminals 20 can be, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, etc. The data aggregation server 30 can be implemented using a standalone server or a server cluster composed of multiple servers. The data storage system can include multiple storage partitions, such as a storage partition for storing the data stream files to be processed, and a storage partition for storing the generated data aggregation files, etc.

[0039] In one embodiment, such as Figure 4 As shown, a data aggregation method is provided. Taking the application of this method to a data aggregation server as an example, the method includes the following steps: S402: Acquire data stream files in various formats.

[0040] These include various formats, including but not limited to portable document formats (such as PDF), spreadsheet formats (such as XLSX, XLS, CSV), image formats (such as PNG, JPG), and plain text formats (such as TXT). Data transfer files are used to record detailed information generated when data flows between different entities.

[0041] Optionally, the data aggregation server receives data stream files uploaded by various data source terminals and performs preliminary verification on the integrity of each data stream file, such as checking if the file size is zero or if the file format is abnormal. Data stream files that pass verification are then uniformly stored in the data storage system. Upon receiving a data aggregation request from a data aggregation terminal, the data aggregation server responds to the request by loading various data stream files in different formats from the data storage system based on the data aggregation information carried in the request, such as portable document format data stream files, spreadsheet format data stream files, and image format data stream files.

[0042] It is worth noting that, because the information systems used by each data provider are built independently, the data transfer files uploaded by different data source terminals not only differ in file format, but also in structure organization and field definition. In other words, the data aggregation server obtains multi-source heterogeneous data transfer files.

[0043] To achieve unified aggregation and processing of multi-source heterogeneous data transfer files, the data aggregation server, after acquiring the files, first performs format parsing based on their file format type. For example, optical character recognition is performed on data transfer files in image or portable document formats, and cell text extraction is performed on data transfer files in spreadsheet formats. This converts the various formats of data transfer files into unified text content, eliminating differences in file format among the multi-source heterogeneous data transfer files. Then, a large language model is used to perform semantic parsing on the parsed text content. For example, key fields are identified and extracted from the text content, and data transfer records in a unified standard format are generated based on these key fields, thereby eliminating differences in structural organization and field definitions among the heterogeneous data transfer files.

[0044] S404: Semantically analyze the text content of each data transfer file using a large language model to obtain at least one data transfer record corresponding to each data transfer file.

[0045] The text content consists of raw text data extracted from the data flow file, including at least one data flow detail. Each data flow detail includes data flow elements such as data flow time, data flow volume, data remaining amount, data flow summary, and data flow identifier. The data flow record is a formatted data entry formed by associating and integrating the various data flow elements in the data flow detail, and it is used to represent detailed information about a single data flow event.

[0046] To ensure the accuracy and reliability of the output of the large language model, the data aggregation server pre-injects semantic parsing prompts into the large language model. This enables the large language model to process at least one data flow detail in each data flow file according to the semantic parsing strategy indicated by the prompts, thereby obtaining at least one data flow record corresponding to each data flow file. The semantic parsing prompts include role definitions, task descriptions, parsing rules, and output format constraints. Specifically, role definitions define the role and professional background that the large language model should assume when performing semantic parsing tasks; task descriptions define the specific operations that the large language model needs to perform; parsing rules specify the specific logic for information processing such as field extraction and direction determination; and output format constraints limit the data structure and format requirements of the large language model's output results.

[0047] Optionally, the large language model performs the following steps according to the semantic parsing strategy indicated by the semantic parsing prompts: First, extract the data flow elements of each data flow detail in each data flow file, including data flow time, data flow volume, data remaining volume, data flow summary, and data flow identifier; then, based on the data flow elements of each data flow detail, deduce the data flow direction corresponding to each data flow detail, where the data flow direction corresponding to any data flow detail can be data inflow or data outflow; next, based on the data flow direction corresponding to each data flow detail, determine the data flow participants in each data flow detail, and the data flow participants... The term "party" refers to the main roles involved in the data flow process, including the data flow owner and the data flow related parties. The data flow owner is the entity that owns the data rights in a single data flow transaction, while the data flow related parties are the entities that interact with this data flow transaction. Finally, the data flow elements, data flow direction, and data flow participants of each data flow detail are used as target fields to generate structured data flow records corresponding to each data flow detail. It can be understood that the structured data flow records generated from each data flow detail corresponding to the same data flow file constitute at least one structured data flow record corresponding to that data flow file.

[0048] In this embodiment, during the file parsing stage, the data aggregation server acquires data transfer files in various formats and uses a large language model to perform semantic parsing on the text content of each data transfer file. This enables it to adaptively identify text content with different structural organization, field definitions, and encoding methods, and extract data transfer records with a unified structure. This achieves automated and unified parsing of heterogeneous data transfer files, which helps improve the efficiency of data aggregation.

[0049] S406: Verify each data flow record based on preset data verification rules.

[0050] Among them, the data verification rules are pre-configured judgment rules used to verify the accuracy and integrity of data flow records. The data verification rules include data consistency verification rules, data repeatability verification rules, and data continuity verification rules.

[0051] Specifically, data consistency verification rules are used to check whether each field in the data flow record matches the text content in the original data flow file, to prevent misaligned lines, missing items, or misplaced fields in the data flow records used for aggregation. Data duplication verification rules are used to check whether the current data flow record is duplicated with other generated data flow records, to avoid duplicate aggregation. Data continuity verification rules are used to check whether the data balance between adjacent data flow records meets the flow continuity condition, to prevent logical errors in the data flow records used for aggregation. It is worth noting that the above flow continuity condition is: for any data flow record, the data balance of its predecessor data flow record and the data flow volume of this data flow record, after being processed by a preset operation method (addition or subtraction), equal the data balance of this data flow record, and the preset operation method matches the data flow direction of this data flow record.

[0052] Optionally, for any data flow record, the data aggregation server first checks whether the values ​​of each field in the record match the text content in the corresponding data flow file, based on data consistency verification rules. If they do not match, the data aggregation server marks the record as failing verification. If they match, the server further checks whether the record is a duplicate of another generated record. If they are duplicates, the record is marked as failing verification. If they are not duplicates, the server further checks whether the data balance of the record meets the continuity requirements, based on data continuity verification rules. If the continuity requirements are not met, the record is marked as failing verification. If the continuity requirements are met, the record is marked as passing verification.

[0053] S408: Generate a data aggregation file based on each data flow record that has passed verification.

[0054] Among them, the data aggregation file refers to a file that organizes multiple verified data flow records according to a unified field definition and structural format, and is used for subsequent data analysis, data management and other processing.

[0055] Optionally, the data aggregation server filters out all data flow records marked as verified, sorts them in ascending order according to the data flow time field, and then assembles the sorted data flow records according to a preset field order and structural format to generate a data aggregation file. During the generation process, the data aggregation server can also add metadata information to the data aggregation file, such as data aggregation time, data source identifier, data aggregation scope, and total number of data records. After generation, the data aggregation server stores the data aggregation file in the data storage system and can send aggregation completion information to the data aggregation terminal to prompt the data aggregation party to view it through the data aggregation terminal.

[0056] In this embodiment, during the verification and aggregation phase, the data aggregation server performs data consistency verification, data duplication verification, and data continuity verification on the parsed data flow records based on preset data verification rules. It then generates a data aggregation file based on the verified data flow records, thereby achieving multi-dimensional automatic verification of the parsing results and reliable generation of the data aggregation file, which helps improve the accuracy of data aggregation.

[0057] In the above data aggregation method, the data aggregation server combines the semantic parsing capabilities of the large language model with preset multi-dimensional data verification rules to achieve automated parsing, verification, and integration of heterogeneous data transfer files, effectively improving the efficiency and accuracy of data aggregation.

[0058] In one embodiment, such as Figure 5 As shown, another data aggregation method is provided. Taking the application of this method to a data aggregation server as an example, the following steps are included: S502: Acquire data stream files in various formats.

[0059] Specifically, S502 is the same as S402, and will not be repeated here.

[0060] S504: Identify the file format type of each data transfer file.

[0061] Optionally, the data aggregation server identifies the file format type of each data transfer file based on its extension or header information. File format types include, but are not limited to, portable document types, spreadsheet types, image types, and plain text types.

[0062] S506: Based on the file format type of each data transfer file, perform format parsing on each data transfer file to obtain at least one data transfer detail in each data transfer file.

[0063] Optionally, for data transfer files in image or portable document format, the data aggregation server uses an optical character recognition tool to convert the text in the image or portable document into editable text content; for data transfer files in spreadsheet format, the data aggregation server uses a table parsing tool to extract the text content by cell and organize the text content according to the table header level and the table's row and column organization; for data transfer files in plain text format, the data aggregation server directly reads the text content.

[0064] Furthermore, after extracting the text content from each data flow file, the data aggregation server splits the extracted text content into at least one data flow detail according to preset splitting rules. Understandably, the preset splitting rules can be line-by-line, delimiter-by-separator, etc. The data flow detail is the original text information unit extracted from the data flow file.

[0065] In this embodiment, during the file parsing stage, the data aggregation server identifies the file format type of each data transfer file and adopts corresponding processing methods such as optical character recognition and table parsing for different file formats such as images, portable documents, and spreadsheets. This enables the heterogeneous format files to be uniformly converted into standardized text input that can be used for semantic parsing by subsequent large language models, effectively improving the system's compatibility with different file formats and the efficiency of data aggregation.

[0066] S508: Extract data flow elements from each data flow detail in each data flow file using a large language model.

[0067] The data flow elements include data flow time, data flow volume, data remaining amount, data flow summary, and data flow identifier. Specifically, data flow time is used to characterize the timestamp of a single data flow event; data flow volume is used to characterize the amount of data transferred in a single data flow event; data remaining amount is used to characterize the total amount of data remaining held by the data flow owner after a single data flow event is completed; data flow summary is used to characterize additional descriptive information for a single data flow event; and data flow identifier is used to indicate the data flow direction of the data flow details.

[0068] Optionally, the large language model uses a semantic parsing strategy based on semantic parsing prompts to extract key fields such as data flow time, data flow volume, data remaining volume, data flow summary, and data flow identifier from each data flow detail. Then, the extracted key fields are converted into a unified standardized format as data flow elements for each data flow detail. For example, the large language model converts the data flow time field extracted from data flow details of different data flow files into the standardized format YYYY-MM-DD; and the large language model converts the data flow volume and data remaining volume fields extracted from data flow details of different data flow files into the standardized format ##.##.

[0069] S510: Based on the data flow elements of each data flow detail, obtain the data flow direction corresponding to each data flow detail.

[0070] Optionally, the data aggregation server sorts each data flow detail in ascending order according to the data flow time of each data flow detail using a large language model. It then iterates through each sorted data flow detail sequentially. For the current data flow detail, it executes the following steps in priority order: Based on the first priority, it uses state value deduction to determine whether the first reference data balance, the first data flow volume, and the first data balance of the current data flow detail satisfy a preset operational relationship; where the first reference data balance is the data balance of the previous data flow detail. If the first reference data balance, the first data flow volume, and the first data balance of the current data flow detail satisfy the preset operational relationship, the state value deduction is deemed valid. At this point, the data flow direction corresponding to the current data flow detail is determined according to the preset operational relationship. If the first reference data balance, the first data flow volume, and the first data balance in the current data flow detail do not satisfy the preset operational relationship, then the state value deduction is determined to be invalid. In this case, based on the second priority, a semantic association deduction method is used to determine the data flow direction corresponding to the current data flow detail based on the data flow summary. If the data flow direction corresponding to the current data flow detail cannot be determined based on the data flow summary, then the semantic association deduction is determined to be invalid. In this case, based on the third priority, a flow identifier deduction method is used to determine the data flow direction corresponding to the current data flow detail based on the data flow identifier.

[0071] Specifically, the preset operational relationships involved in the above-mentioned state value deduction method include addition or subtraction operations. When the sum of the first reference data reserve and the first data flow amount in the current data flow detail equals the first data reserve, the large language model deduces that the data flow direction corresponding to the current data flow detail is data inflow; when the difference between the first reference data reserve and the first data flow amount in the current data flow detail equals the first data reserve, the large language model deduces that the data flow direction corresponding to the current data flow detail is data outflow. It is worth noting that if the current data flow detail is the first data flow detail in the current data flow file, then there is no previous data flow detail in the current data flow file. In this case, the first reference data reserve is taken from the data reserve of the last data flow detail in the previous data flow file of the current data flow file. Understandably, when each data flow file records data flow details on a daily basis, the data remaining amount of the last data flow detail in the previous data flow file is the data remaining amount at the end time of the previous day.

[0072] The aforementioned semantic association deduction method can be implemented using keyword matching. Specifically, for example, when the data flow summary contains any keyword from a preset first keyword set, the large language model determines the data flow direction as data inflow; where the first keyword set includes, but is not limited to: receive, receive, deposit, transfer in, collect payment, and credit. When the data flow summary contains any keyword from a preset second keyword set, the large language model determines the data flow direction as data outflow; where the second keyword set includes, but is not limited to: send, issue, withdraw, transfer out, pay, deduct payment, pay, and credit. It should be noted that the above keyword matching is only one specific implementation of the large language model's semantic analysis capabilities. The large language model can also handle semantic variations beyond the keyword set based on contextual semantic understanding. For example, when the data flow summary is expressed as "deduct and pay account management fees on behalf of others," the large language model can accurately identify its implicit fund outflow attribute without relying on precise literal matching of preset keywords; it can automatically identify the data flow direction semantics contained therein through its semantic generalization capabilities.

[0073] When a clear data flow direction cannot be determined based on the data flow summary, the large language model further determines the data flow direction based on the data flow identifier. For example, when the data flow identifier is a first identifier, the large language model determines the data flow direction as data inflow; wherein, the first identifier can be, but is not limited to, an identifier value representing the data receiver's perspective, such as a marker symbol (e.g., "borrow") or conventional code (e.g., "IN") used in the original data flow details to represent the data receiver. When the data flow identifier is a second identifier, the large language model determines the data flow direction as data outflow; wherein, the second identifier can be, but is not limited to, an identifier value representing the data sender's perspective, such as a marker symbol (e.g., "loan") or conventional code (e.g., "OUT") used in the original data flow details to represent the data sender. It should be noted that the data flow identifier is only used as an auxiliary determination basis when the aforementioned state value deduction method fails and the data flow summary cannot provide clear directional information; its determination priority is lower than the state value deduction method and the semantic analysis method of the data flow summary.

[0074] S512: Determine the data flow participants in each data flow detail based on the data flow direction corresponding to each data flow detail.

[0075] In this context, "data flow participants" refers to the entities involved in the data flow process, including the data flow owner and related parties. Specifically, the data flow owner is the data holder recorded in the data flow details, i.e., the entity to which the remaining data belongs after the data flow occurs. Related parties are the entities that interact with the data flow owner in the data flow process; i.e., other parties different from the data flow owner in the data flow activity.

[0076] Optionally, if the data flow direction corresponding to any data flow detail is data inflow, the data aggregation server determines the data inflow party in that data flow detail as the data flow owner and the data outflow party in that data flow detail as the data flow related party; if the data flow direction corresponding to any data flow detail is data outflow, the data aggregation server determines the data outflow party in that data flow detail as the data flow owner and the data inflow party in that data flow detail as the data flow related party.

[0077] S514: Using data flow elements, data flow direction, and data flow participants as target fields, generate at least one data flow record corresponding to each data flow file.

[0078] Among them, at least one data flow record refers to a formatted data entry with a unified data structure generated after semantic parsing of the data flow details. Each data flow record is used to represent the detailed information of a single data flow event.

[0079] Optionally, the large language model organizes data flow elements, data flow directions, and data flow participants according to a preset unified format (such as JSON format), generating structured data flow records corresponding to each data flow detail. The structured data flow records generated from each data flow detail corresponding to the same data flow file constitute at least one data flow record corresponding to that data flow file.

[0080] In this embodiment, during the record generation stage, the data aggregation server extracts data flow elements from unstructured data flow details using a large language model. Based on these data flow elements, it employs a priority-based strategy of state value deduction, semantic association deduction, and flow identifier deduction to obtain the data flow direction. Based on the data flow direction, it determines the data flow participants. Finally, it uses the data flow elements, data flow direction, and data flow participants as target fields to generate a unified structured data flow record. This achieves an adaptive conversion from unstructured details to structured records, effectively improving the efficiency and accuracy of data aggregation.

[0081] S516: Verify each data flow record based on preset data verification rules.

[0082] Among them, the data verification rules are pre-configured judgment rules used to verify the accuracy and integrity of data flow records. The data verification rules include data consistency verification rules, data repeatability verification rules, and data continuity verification rules.

[0083] In one possible implementation, the data aggregation server verifies, based on data consistency verification rules, whether the values ​​of each field in any data flow record match the text content in the corresponding data flow file; based on data duplication verification rules, it verifies whether any data flow record is duplicated with another data flow record that has already been generated; and based on data continuity verification rules, it verifies whether the data reserve of any data flow record meets the flow continuity condition.

[0084] Optionally, the data aggregation server verifies each data flow record sequentially based on data consistency verification rules, data duplication verification rules, and data continuity verification rules. For any given data flow record, the data aggregation server performs the following verification steps: S5161. Based on the data consistency verification rules, verify whether the values ​​of each field in the data flow record match the text content in the corresponding data flow file to prevent issues such as incorrect lines, missing items, or misaligned fields in the data flow records used for aggregation. If the values ​​of each field in the data flow record match the text content in the corresponding data flow file, continue with the subsequent verification in S5162; if the values ​​of each field in the data flow record do not match the text content in the corresponding data flow file, mark the data flow record as failing and record the names of the mismatched fields and their positions, etc., so that the data flow records that fail verification can be corrected later using the large language model based on the anomaly information.

[0085] S5162. Based on the data duplication verification rules, verify whether the values ​​of each field in the data flow record are duplicates of other generated data flow records to avoid duplicate aggregation. If the values ​​of each field in the data flow record are not duplicates of other generated data flow records, continue to execute the subsequent verification in S5163; if the values ​​of each field in the data flow record are duplicates of other generated data flow records, mark the data flow record as failing and record the duplicate data flow record identifier and other abnormal information so that the subsequent large language model can re-parse the data flow file with duplicate data flow records based on the abnormal information.

[0086] S5163. Based on the data continuity verification rules, verify whether the data balance of the data flow record meets the flow continuity condition to prevent computational logic errors in the data flow records used for aggregation. The flow continuity condition is: for any data flow record, the data balance of its predecessor data flow record and the data flow volume of the current data flow record, after being processed by a preset operation method (addition or subtraction), equal the data balance of the current data flow record, and the preset operation method matches the data flow direction of the current data flow record. If the data balance of the data flow record meets the flow continuity condition, the data flow record is marked as verified as passed; if the data balance of the data flow record does not meet the flow continuity condition, the data flow record is marked as verified as failed, and the identifier of the data flow record and its predecessor data flow record, etc., are recorded so that the subsequent large language model can re-parse the data flow records that do not meet the flow continuity condition based on the abnormal information.

[0087] In this embodiment, the data aggregation server performs consistency checks, duplicate checks, and continuity checks on the parsed data flow records sequentially before data aggregation. This enables it to comprehensively identify and intercept inconsistent errors, duplicate records, and logical anomalies that occur during the parsing process, thereby improving the accuracy and reliability of data aggregation.

[0088] S518: Determine if there are any abnormal data transfer records that fail verification. If yes, proceed to S520; otherwise, proceed to S528.

[0089] Among them, abnormal data flow records refer to data flow records that are marked as failing the verification process. There can be one or more abnormal data flow records.

[0090] Understandably, if the data aggregation server determines that there is an abnormal data flow record that fails verification in at least one data flow record output by the large language model, it will further determine whether the preset number of iterations has been reached to confirm whether to start the iterative correction process; if the data aggregation server determines that all at least one data flow record output by the large language model has passed verification, it will generate a data aggregation file based on each data flow record that has passed verification.

[0091] S520: Determine if the preset number of iterations has been reached. If yes, proceed to S528; otherwise, proceed to S522.

[0092] The preset iteration count is the maximum number of correction attempts set in advance. It is used to prevent infinite loops caused by defects in the data flow details themselves. The preset iteration count can be pre-configured according to the actual application scenario, for example, set to 3 times, 5 times, etc.

[0093] Optionally, the data aggregation server records the number of corrections currently performed. After each correction, the number is incremented by one and compared with the preset number of iterations. If the number of corrections currently performed reaches the preset number of iterations, the data aggregation server stops correcting the abnormal data flow records and marks the currently failed data flow records as data flow records to be reviewed. If the number of corrections currently performed does not reach the preset number of iterations, the data aggregation server continues to correct the abnormal data flow records.

[0094] S522: Determine the abnormal information associated with the abnormal data flow record.

[0095] The abnormal information associated with abnormal data transfer records includes, but is not limited to: the file identifier of the data transfer file associated with the abnormal data transfer record, the abnormal type of the abnormal data transfer record, and the abnormal description of the abnormal data transfer record. Specifically, the above-mentioned abnormal types may include, but are not limited to, data consistency abnormalities, data duplication abnormalities, and data continuity abnormalities. In the case of a data consistency abnormality, the above-mentioned abnormal description includes, but is not limited to: the name of the abnormal field and the location of the abnormal field. In the case of a data duplication abnormality, the above-mentioned abnormal description includes, but is not limited to: the identifier of the duplicate data transfer record. In the case of a data continuity abnormality, the above-mentioned abnormal description includes, but is not limited to: the identifier of the data transfer record that does not meet the continuity condition and the identifier of its predecessor.

[0096] Understandably, by using the anomaly information, the data aggregation server can quickly locate the original data transfer file corresponding to the abnormal data transfer record and determine the specific cause of the anomaly, thereby providing targeted guidance information for the subsequent correction of the large language model.

[0097] S524: Correct the abnormal data flow records based on the abnormal information using a large language model to obtain corrected data flow records.

[0098] Optionally, the data aggregation server retrieves the data transfer file associated with the abnormal data transfer record based on the data transfer file identifier in the anomaly information, and re-parses the format of the data transfer file to obtain its text content. Then, the anomaly type and description in the anomaly information, along with the aforementioned text content, are input into the large language model as parsing guidance information. The large language model re-extracts and parses the text content based on the parsing guidance information. For example, if the anomaly information indicates a data consistency anomaly, the large language model re-identifies and extracts the corresponding field values ​​from the text content based on the name and position of the anomaly field. If the anomaly information indicates a data duplication anomaly, the large language model re-extracts and parses the text content based on the duplicate data transfer identifier. If the anomaly information indicates a data continuity anomaly, the large language model, considering the data remaining amount of the previous data transfer record, re-infers the data transfer direction of the abnormal data transfer record. Finally, the large language model outputs the corrected data transfer record obtained after the above targeted corrections.

[0099] S526: Modify the data flow record as the new data flow record, and execute the steps in S516 again.

[0100] Optionally, the data aggregation server verifies each corrected data flow record based on preset data verification rules, thereby continuously correcting and verifying abnormal data through iterative loops.

[0101] S528: Generate a data aggregation file based on each data flow record that has passed verification.

[0102] Specifically, S528 is identical to S408, and will not be repeated here.

[0103] In the aforementioned data aggregation method, the data aggregation server identifies file format types and employs corresponding data parsing methods such as optical character recognition and table parsing to uniformly convert heterogeneous format files into standardized text input suitable for subsequent semantic analysis by a large language model. This effectively improves the system's compatibility with different file formats and the efficiency of data aggregation. The large language model extracts data flow elements from unstructured data flow details, determines the data flow direction based on these elements, identifies the data flow participants based on the flow direction, and generates unified structured data flow records. This achieves adaptive conversion from unstructured details to structured records, improving the accuracy of data aggregation. For each data flow record, data consistency, repeatability, and continuity checks are performed sequentially. For identified abnormal data flow records, the large language model is used for correction, improving the reliability of data aggregation. The entire data aggregation process achieves efficient and accurate aggregation of multi-source heterogeneous data flow files, effectively reducing the frequency and complexity of manual intervention and ensuring the reliability of data aggregation through multi-dimensional verification and closed-loop correction.

[0104] In one embodiment, such as Figure 6 As shown, another data aggregation method is provided. Taking the application of this method to a data aggregation server as an example, the following steps are included: S602: Acquire data stream files in various formats.

[0105] Specifically, S602 is the same as S402, and will not be repeated here.

[0106] S604: Extract the data flow elements of each data flow detail in each data flow file through a large language model.

[0107] Specifically, S604 is the same as S508, and will not be repeated here.

[0108] S606: Sort each data flow detail in ascending order according to its data flow time.

[0109] Among them, data transfer time is used to characterize the timestamp of a single data transfer behavior.

[0110] Understandably, the data aggregation server arranges each data flow detail in chronological order according to the data flow time, providing a basis for determining the data flow direction based on the data reserve relationship between adjacent data flow details.

[0111] S608: Iterate through each sorted data flow detail in sequence, and for the current data flow detail, obtain the first reference data balance, the first data flow volume, and the first data balance.

[0112] The first reference data balance is the data balance of the previous data flow detail in the current data flow detail, serving as a benchmark value for inferring the direction of data flow. It's worth noting that if the current data flow detail is the first data flow detail in the current data flow file, then there is no previous data flow detail in that file. In this case, the previous data flow detail is the last data flow detail of the preceding data flow file, meaning the first reference data balance is taken from the data balance of the last data flow detail of that preceding data flow file. The first data flow quantity characterizes the amount of data transferred in a single data flow action; for example, it can be the data transfer value recorded in the current data flow detail. The first data balance characterizes the total amount of data remaining held by the data owner after a single data flow action is completed; for example, it can be the data balance value recorded in the current data flow detail.

[0113] Optionally, after the data aggregation server completes the sorting of each data flow detail, it reads the data flow volume field value and data surplus field value of each data flow detail one by one, starting from the first data flow detail, and uses them as the first data flow volume and the first data surplus of each data flow detail, respectively. Based on the sorting result, it obtains the data surplus field value of the previous data flow detail of the current data flow detail, and uses it as the first reference data surplus of the current data flow detail.

[0114] S610: Determine whether the first reference data margin, the first data flow volume, and the first data margin satisfy the preset operation relationship. If yes, execute S612; if no, execute S614.

[0115] The preset operation relationships include addition and subtraction.

[0116] Optionally, the data aggregation server calculates the sum of the first reference data reserve and the first data flow rate, and the difference between the first reference data reserve and the first data flow rate, respectively, and compares the two calculation results with the first data reserve. If either of them is equal to the first data reserve, it is determined that the preset operation relationship is satisfied; if neither of them is equal to the first data reserve, it is determined that the preset operation relationship is not satisfied.

[0117] S612: Determine the data flow direction corresponding to the current data flow details based on the preset operation relationship.

[0118] The direction of data flow includes data inflow and data outflow. Data inflow indicates that data flows from an external entity to the current data flow owner, while data outflow indicates that data flows from the current data flow owner to an external entity.

[0119] Optionally, the data aggregation server determines the data flow direction corresponding to the current data flow detail based on the calculation result between the first reference data reserve and the first data flow volume and the comparison result of the first data reserve.

[0120] In one possible implementation, the data aggregation server determines the data flow direction corresponding to the current data flow detail as data inflow when the sum of the first reference data reserve and the first data flow volume equals the first data reserve; and determines the data flow direction corresponding to the current data flow detail as data outflow when the difference between the first reference data reserve and the first data flow volume equals the first data reserve.

[0121] Specifically, when the sum of the first reference data balance (i.e., the data balance of the previous data flow detail) and the first data flow volume (i.e., the data flow volume of the current data flow detail) equals the first data balance (i.e., the data balance of the current data flow detail), it indicates that the current data flow behavior increases the data balance, and therefore the data flow direction is determined to be data inflow. When the difference between the first reference data balance and the first data flow volume equals the first data balance, it indicates that the current data flow behavior decreases the data balance, and therefore the data flow direction is determined to be data outflow.

[0122] In this embodiment, the data aggregation server determines whether the first reference data surplus, the first data flow volume, and the first data surplus satisfy an addition or subtraction operation relationship. If the addition operation relationship is satisfied, the data flow direction is determined to be data inflow; if the subtraction operation relationship is satisfied, the data flow direction is determined to be data outflow. This can quickly and accurately determine the data flow direction, improving the efficiency and reliability of the data flow direction determination.

[0123] S614: Determine whether the data flow summary contains data flow direction semantics. If yes, execute S616; otherwise, execute S618.

[0124] The data flow summary provides additional descriptive information to characterize a single data flow transaction, such as a description of the flow's purpose, like "payment for goods" or "deduction of handling fees." The data flow direction semantics refers to the semantic information identified from the text content of the data flow summary that indicates whether data is transferred in or out.

[0125] Optionally, the data aggregation server performs semantic analysis on the data flow summary using a large language model to determine whether the data flow summary contains key semantic information that can indicate the direction of data flow. If it does, the data aggregation server further determines the data flow direction corresponding to the current data flow detail based on the data flow summary using the large language model; if it does not, the data aggregation server further determines the data flow direction corresponding to the current data flow detail based on the data flow identifier using the large language model.

[0126] S616: Determine the data flow direction corresponding to the current data flow detail based on the data flow summary.

[0127] Optionally, when the data flow summary contains any keyword from a preset first keyword set, the large language model determines the data flow direction as data inflow; wherein, the first keyword set includes, but is not limited to: receive, receive, deposit, transfer in, collect payment, credit to account, etc. When the data flow summary contains any keyword from a preset second keyword set, the large language model determines the data flow direction as data outflow; wherein, the second keyword set includes, but is not limited to: send, send out, retrieve, transfer out, pay, deduct payment, pay, credit to account, etc.

[0128] It should be noted that the keyword matching described above is only one specific implementation of the semantic analysis capabilities of the large language model. The large language model can also handle semantic variations beyond the keyword set based on contextual semantic understanding. For example, when the data flow summary is expressed as "deduction and payment of account management fees", the large language model can accurately identify its implicit fund outflow attribute without relying on literal matching of preset keywords. It can automatically identify the semantic direction of data flow contained therein through its semantic generalization capability.

[0129] S618: Determine the data flow direction corresponding to the current data flow detail based on the data flow identifier.

[0130] Optionally, when the data flow identifier is the first identifier, the large language model determines the data flow direction as data inflow; wherein, the first identifier may be, but is not limited to, an identifier value used to represent the perspective of the data receiver, for example, it may be a marker symbol (such as "borrow") or conventional code (such as "IN") used to represent the data receiver in the original data flow details. When the data flow identifier is the second identifier, the large language model determines the data flow direction as data outflow; wherein, the second identifier may be, but is not limited to, an identifier value used to represent the perspective of the data sender, for example, it may be a marker symbol (such as "credit") or conventional code (such as "OUT") used to represent the data sender in the original data flow details.

[0131] It should be noted that the data flow identifier is only used as an auxiliary basis for judgment when the aforementioned state value inference method fails and the data flow summary cannot provide clear directional information. Its judgment priority is lower than that of the state value inference method and the semantic analysis method of the data flow summary.

[0132] In this embodiment, the data aggregation server employs a data flow direction determination strategy that prioritizes state value deduction, followed by summary analysis, and uses identifier judgment as a fallback. This strategy ensures accuracy while effectively addressing boundary situations such as the first detail entry, missing data, and semantically ambiguous summaries. It enables rapid and accurate determination of data flow direction, thereby improving the accuracy and robustness of data aggregation.

[0133] S620: Determine the data flow participants in each data flow detail based on the data flow direction corresponding to each data flow detail.

[0134] The participants in data flow include the data flow owner and the data flow related parties. The data flow owner is the data holder recorded in the data flow details, that is, the entity to which the remaining data belongs after the data flow occurs; the data flow related parties are the counterparty that interacts with the data flow owner, that is, other parties different from the data flow owner in the data flow process.

[0135] In one possible implementation, when the data flow direction corresponding to any data flow detail is data inflow, the data aggregation server determines the data inflow recipient in any data flow detail as the data flow owner and the data outflow recipient in any data flow detail as the data flow associated party; when the data flow direction corresponding to any data flow detail is data outflow, the data outflow recipient in any data flow detail is determined as the data flow owner and the data inflow recipient in any data flow detail is determined as the data flow associated party.

[0136] In this context, the data recipient refers to the party receiving data in the data flow process, while the data sender refers to the party sending data in the data flow process. In the data flow details, the data recipient and data sender can be represented by account name, account number, or system code, etc.

[0137] When the data flow direction is "data in," it indicates that data is flowing from an external entity to the data owner. In this case, the party receiving the data is the data owner, and the party sending the data is the data related party. Therefore, when the data flow direction is "data in," the data aggregation server determines the data recipient in the data flow details as the data owner and the data sender in the data flow details as the data related party.

[0138] When the data flow direction is outward, it indicates that the data is flowing from the data owner to an external entity. In this case, the party sending the data is the data owner, and the party receiving the data is the data related party. Therefore, when the data flow direction is outward, the data aggregation server determines the data sender in the data flow details as the data owner and the data receiver in the data flow details as the data related party.

[0139] In this embodiment, the data aggregation server can automatically determine the data transfer owner and data transfer associate in each data transfer detail without manual intervention, based on the determined data transfer direction. This helps to improve the accuracy and efficiency of data transfer record generation, thereby improving the accuracy and efficiency of data aggregation.

[0140] S622: Using data flow elements, data flow direction, and data flow participants as target fields, generate at least one data flow record corresponding to each data flow file.

[0141] Among them, at least one data flow record refers to a formatted data entry with a unified data structure generated after semantic parsing of the data flow details. Each data flow record is used to represent the detailed information of a single data flow event.

[0142] Optionally, the large language model organizes data flow elements, data flow directions, and data flow participants according to a preset unified format (such as JSON format), generating structured data flow records corresponding to each data flow detail. The structured data flow records generated from each data flow detail corresponding to the same data flow file constitute at least one data flow record corresponding to that data flow file.

[0143] In this embodiment, during the record generation stage, the data aggregation server extracts data flow elements from unstructured data flow details using a large language model. Based on these data flow elements, it employs a priority-based strategy of state value deduction, semantic association deduction, and flow identifier deduction to obtain the data flow direction. Based on the data flow direction, it automatically determines the data flow participants. Finally, it uses the data flow elements, data flow direction, and data flow participants as target fields to generate a unified structured data flow record. This achieves an adaptive conversion from unstructured details to structured records, effectively improving the efficiency and accuracy of data aggregation.

[0144] S624: Based on the data consistency verification rules, verify whether the values ​​of each field in any data flow record match the text content in the corresponding data flow file.

[0145] Among them, the data consistency verification rules are used to verify whether the values ​​of each field in the data flow record are consistent with the text content in the original data flow file, so as to prevent incorrect lines, missing items or misaligned fields.

[0146] Optionally, the data aggregation server first checks whether the values ​​of each field in any data flow record match the text content in the corresponding data flow file based on data consistency verification rules. If they do not match, the data aggregation server marks the data flow record as unacceptable and records the inconsistent abnormal fields as exception information. If they match, the data aggregation server executes S626 to check whether any data flow record is duplicated with another generated data flow record based on data duplication verification rules.

[0147] S626: Based on the data duplication verification rules, verify whether any data flow record is duplicated with another data flow record that has already been generated.

[0148] Among them, the data duplication verification rule is used to verify whether the current data flow record is duplicated with other data flow records that have been generated, so as to avoid duplicate aggregation.

[0149] Optionally, the data aggregation server checks whether any data flow record is duplicated with another already generated data flow record based on data duplication verification rules. If duplicated, the data aggregation server marks the data flow record as unacceptable and records the duplicate data flow record identifier as an anomaly. If not duplicated, the data aggregation server checks whether the data remaining amount of the data flow record meets the flow continuity conditions based on data continuity verification rules.

[0150] S628: Based on the data continuity verification rules, verify whether the data balance of any data flow record meets the flow continuity conditions.

[0151] Among them, the data continuity verification rule is used to verify whether the data balance between adjacent data flow records meets the flow continuity condition, so as to prevent data flow records from having missing records, disordered timing, or incorrect balance calculation logic.

[0152] Optionally, the data aggregation server verifies whether the data balance of the data flow record meets the flow continuity conditions based on data continuity verification rules. If it does not meet the conditions, the data aggregation server marks the data flow record as failing the verification and records the identifier of the data flow record as an anomaly. If the conditions are met, the data aggregation server marks the data flow record as passing the verification.

[0153] In one possible implementation, the data aggregation server performs the following steps when verifying whether the data balance of any data flow record meets the flow continuity condition based on data continuity verification rules: obtaining the second reference data balance, the second data flow volume, and the second data balance of any data flow record; the second reference data balance is the data balance of the previous data flow record of any data flow record; the previous data flow record of any data flow record is determined based on the ascending order of data flow time of each data flow record; if the sum of the second reference data balance and the second data flow volume equals the second data balance, or the difference between the second reference data balance and the second data flow volume equals the second data balance, verifying whether the data balance of any data flow record meets the flow continuity condition according to the data flow direction of any data flow record; if the sum of the second reference data balance and the data flow volume is not equal to the second data balance, and the difference between the second reference data balance and the second data flow volume is not equal to the second data balance, determining that the data balance of any data flow record does not meet the flow continuity condition.

[0154] Optionally, the data aggregation server first sorts the data flow records in ascending order according to their data flow time, ensuring that the data flow records are arranged in chronological order. After sorting, for the current data flow record to be verified, the data aggregation server determines its predecessor data flow record based on the sorting result and obtains the data surplus of that predecessor data flow record as the second reference data surplus; simultaneously, it obtains the data flow volume from the current data flow record as the second data flow volume, and obtains its data surplus as the second data surplus. Subsequently, the data aggregation server calculates the sum of the second reference data surplus and the second data flow volume, as well as the difference between the second reference data surplus and the second data flow volume, and compares the two calculation results with the second data surplus respectively.

[0155] If the sum of the second reference data surplus and the second data flow volume equals the second data surplus, it indicates that the surplus of the current data flow record satisfies the addition operation relationship. At this time, the data aggregation server further determines whether the data flow direction of the current data flow record is data transfer in. If the data flow direction is data transfer in, the data aggregation server determines that the data surplus of the current data flow record meets the flow continuity condition; if the data flow direction is data transfer out and not data transfer in, the data aggregation server determines that the data surplus of the current data flow record does not meet the flow continuity condition.

[0156] If the difference between the second reference data surplus and the second data flow volume equals the second data surplus, it indicates that the data surplus of the current data flow record satisfies the subtraction operation relationship. At this time, the data aggregation server further determines whether the data flow direction of the current data flow record is data out. If the data flow direction is data out, the data aggregation server determines that the data surplus of the current data flow record meets the flow continuity condition; if the data flow direction is data in and not data out, the data aggregation server determines that the data surplus of the current data flow record does not meet the flow continuity condition.

[0157] If the sum of the second reference data balance and the second data flow volume, and the difference between the second reference data balance and the second data flow volume are not equal to the second data balance, it indicates that the balance of the current data flow record does not satisfy either the addition operation relationship or the subtraction operation relationship. At this time, the data aggregation server directly determines that the data balance of the current data flow record does not meet the flow continuity condition, and there is no need to further verify the data flow direction.

[0158] Through the above verification steps, the data aggregation server can automatically verify the logical relationship of data balance between adjacent data flow records, ensuring the temporal integrity and operational consistency of data flow records, thereby effectively identifying and intercepting data flow records with missing records, disordered temporal sequences, or incorrect balance calculation logic.

[0159] S630: Generate a data aggregation file based on each data flow record that has passed verification.

[0160] Specifically, the S630 is identical to the S408, which will not be repeated here.

[0161] In the aforementioned data aggregation method, the data aggregation server first sorts the data flow details according to the data flow time. Then, it iterates through each detail sequentially, using a priority-based strategy of state value inference first, summary semantic analysis second, and identifier judgment as a fallback to determine the data flow direction. Based on the judgment results, it automatically determines the data flow owner and related parties, generating a unified structured data flow record. Subsequently, it performs multi-dimensional automated verification of the data flow record by sequentially executing data consistency verification, duplicate verification, and continuity verification. The continuity verification checks whether the data balance of adjacent records satisfies the addition and subtraction relationship and matches the data flow direction, ensuring the temporal integrity and operational consistency of the data flow. The entire process requires no manual intervention, achieving adaptive inference and transformation from unstructured details to structured records and multi-dimensional automated verification, effectively improving the accuracy, reliability, and efficiency of data aggregation.

[0162] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0163] Based on the above-mentioned data aggregation method, such as Figure 7 As shown in the embodiments of this specification, a data aggregation apparatus 700 for implementing the data aggregation method described above is also provided. The data aggregation apparatus 700 includes: Module 701 is used to acquire data stream files in various formats. The parsing module 702 is used to perform semantic parsing on the text content of each data flow file through a large language model to obtain at least one data flow record corresponding to each data flow file. The verification module 703 is used to verify each data flow record based on preset data verification rules; the data verification rules include data consistency verification rules, data repeatability verification rules, and data continuity verification rules. The generation module 704 is used to generate a data aggregation file based on each data flow record that has passed verification.

[0164] In one possible implementation, the parsing module 702 is further configured to: identify the file format type of each data stream file; if the file format type of any data stream file is identified as an image type or a portable document type, perform optical character recognition on any data stream file to obtain the text content of any data stream file; if the file format type of any data stream file is identified as a spreadsheet type, extract cell text from any data stream file to obtain the text content of any data stream file.

[0165] In one possible implementation, the text content includes at least one data flow detail; the parsing module 702 is specifically used to: extract the data flow elements of each data flow detail in each data flow file through a large language model; obtain the data flow direction corresponding to each data flow detail based on the data flow elements of each data flow detail; determine the data flow participants in each data flow detail based on the data flow direction corresponding to each data flow detail; and generate at least one data flow record corresponding to each data flow file by using the data flow elements, data flow direction, and data flow participants as target fields.

[0166] In one possible implementation, the data flow elements include data flow time, data flow volume, data remaining amount, data flow summary, and data flow identifier; the parsing module 702 is specifically used to: sort each data flow detail in ascending order according to the data flow time of each data flow detail; sequentially traverse each sorted data flow detail, and for the current data flow detail, obtain the first reference data remaining amount, the first data flow volume, and the first data remaining amount of the current data flow detail; the first reference data remaining amount is the data remaining amount of the previous data flow detail of the current data flow detail; the first reference data remaining amount and the first data flow volume... If the first data surplus satisfies a preset operation relationship, the data flow direction corresponding to the current data flow detail is determined according to the preset operation relationship; if the first reference data surplus, the first data flow volume, and the first data surplus do not satisfy the preset operation relationship, it is determined whether the data flow summary contains data flow direction semantics; if the data flow summary contains data flow direction semantics, the data flow direction corresponding to the current data flow detail is determined according to the data flow summary; if the data flow summary does not contain data flow direction semantics, the data flow direction corresponding to the current data flow detail is determined according to the data flow identifier.

[0167] In one possible implementation, the parsing module 702 is specifically used to: determine the data flow direction corresponding to the current data flow detail as data transfer-in when the sum of the first reference data reserve and the first data flow volume is equal to the first data reserve; and determine the data flow direction corresponding to the current data flow detail as data transfer-out when the difference between the first reference data reserve and the first data flow volume is equal to the first data reserve.

[0168] In one possible implementation, the data flow participants include the data flow owner and the data flow related party; the parsing module 702 is specifically used to: when the data flow direction corresponding to any data flow detail is data transfer in, determine the data transfer-in party in any data flow detail as the data flow owner, and determine the data transfer-out party in any data flow detail as the data flow related party; when the data flow direction corresponding to any data flow detail is data transfer out, determine the data transfer-out party in any data flow detail as the data flow owner, and determine the data transfer-in party in any data flow detail as the data flow related party.

[0169] In one possible implementation, the verification module 703 is specifically used to: verify whether the values ​​of each field in any data flow record match the text content in the corresponding data flow file based on data consistency verification rules; verify whether any data flow record is duplicated with another data flow record that has been generated based on data duplication verification rules; and verify whether the data reserve of any data flow record meets the flow continuity condition based on data continuity verification rules.

[0170] In one possible implementation, the verification module 703 is specifically used to: obtain the second reference data balance, the second data flow volume, and the second data balance of any data flow record; the second reference data balance is the data balance of the previous data flow record of any data flow record; the previous data flow record of any data flow record is determined based on the ascending order of data flow time of each data flow record; if the sum of the second reference data balance and the second data flow volume is equal to the second data balance, or the difference between the second reference data balance and the second data flow volume is equal to the second data balance, verify whether the data balance of any data flow record meets the flow continuity condition according to the data flow direction of any data flow record; if the sum of the second reference data balance and the data flow volume is not equal to the second data balance, and the difference between the second reference data balance and the second data flow volume is not equal to the second data balance, determine that the data balance of any data flow record does not meet the flow continuity condition.

[0171] In one possible implementation, the verification module 703 is specifically used to: when the sum of the second reference data reserve and the second data flow amount equals the second data reserve, determine whether the data flow direction of any data flow record is data transfer in; if yes, determine that the data reserve of any data flow record meets the flow continuity condition; if no, determine that the data reserve of any data flow record does not meet the flow continuity condition; when the difference between the second reference data reserve and the second data flow amount equals the second data reserve, determine whether the data flow direction of any data flow record is data transfer out; if yes, determine that the data reserve of any data flow record meets the flow continuity condition; if no, determine that the data reserve of any data flow record does not meet the flow continuity condition.

[0172] In one possible implementation, the data aggregation device 700 further includes a correction module, used to: in response to the existence of an abnormal data flow record that fails verification, determine the abnormal information associated with the abnormal data flow record; correct the abnormal data flow record based on the abnormal information using a large language model to obtain a corrected data flow record; use the corrected data flow record as a new data flow record, and execute the steps of verifying each data flow record based on a preset data verification rule again, until each data flow record passes verification or reaches a preset number of iterations.

[0173] Each module in the aforementioned data aggregation device 700 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0174] This specification also provides an electronic device, which may be a server, and its internal structure diagram may be as follows: Figure 8As shown, this electronic device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The application database stores data such as data transfer files and generated data aggregation files. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. The processor executes computer programs to implement a data aggregation method.

[0175] Those skilled in the art will understand that Figure 8 The structures shown are merely block diagrams of some structures related to the solutions in this specification and do not constitute a limitation on the electronic devices to which the solutions in this specification are applied. Specific electronic devices may include more or fewer components than those shown in the figures, or may combine certain components, or may have different component arrangements.

[0176] In one possible implementation, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0177] This specification also provides a computer storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps in the above embodiments. If the constituent modules of the above-described electronic device are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer storage medium.

[0178] In one possible implementation, a computer storage medium is provided that stores a computer program, which, when executed by a processor, implements the steps in the above method embodiments.

[0179] This specification also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0180] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0181] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and its implementation can be combined arbitrarily.

[0182] The embodiments described above are merely preferred embodiments of this specification and are not intended to limit the scope of this specification. Any modifications and improvements made by those skilled in the art to the technical solutions of this specification without departing from the spirit of this specification should fall within the protection scope defined by the claims.

[0183] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

Claims

1. A data aggregation method, characterized in that, The method includes: Acquire data stream files in various formats; Semantic parsing of the text content of each data transfer file is performed using a large language model to obtain at least one data transfer record corresponding to each data transfer file; Based on preset data verification rules, each data flow record is verified; the data verification rules include data consistency verification rules, data repeatability verification rules, and data continuity verification rules. A data aggregation file is generated based on each verified data flow record.

2. The method as described in claim 1, characterized in that, Before performing semantic parsing of the text content of each data transfer file using a large language model to obtain at least one data transfer record corresponding to each data transfer file, the method further includes: Identify the file format type of each of the data stream files; If the file format type of any of the data stream files is identified as image type or portable document type, optical character recognition is performed on any of the data stream files to obtain the text content of any of the data stream files; If any of the data transfer files is identified as having a spreadsheet format, cell text extraction is performed on any of the data transfer files to obtain the text content of any of the data transfer files.

3. The method as described in claim 1, characterized in that, The text content includes at least one data flow detail; The step involves semantically parsing the text content of each data transfer file using a large language model to obtain at least one data transfer record corresponding to each data transfer file, including: Using a large language model, extract the data flow elements of each data flow detail in each of the data flow files; Based on the data flow elements of each data flow detail, obtain the data flow direction corresponding to each data flow detail; Based on the data flow direction corresponding to each of the data flow details, determine the data flow participants in each of the data flow details; Using the data flow elements, the data flow direction, and the data flow participants as target fields, at least one data flow record is generated corresponding to each data flow file.

4. The method as described in claim 3, characterized in that, The data flow elements include data flow time, data flow volume, data remaining volume, data flow summary, and data flow identifier; The step of obtaining the data flow direction corresponding to each data flow detail based on the data flow elements of each data flow detail includes: Sort the data flow details in ascending order according to the data flow time of each data flow detail; The sorted data flow details are traversed sequentially. For the current data flow detail, the first reference data balance, the first data flow volume, and the first data balance are obtained. The first reference data balance is the data balance of the previous data flow detail of the current data flow detail. When the first reference data margin, the first data flow volume, and the first data margin satisfy a preset operation relationship, the data flow direction corresponding to the current data flow detail is determined according to the preset operation relationship. If the first reference data margin, the first data flow volume, and the first data margin do not satisfy a preset operation relationship, determine whether the data flow summary contains data flow direction semantics. If the data flow summary contains data flow direction semantics, the data flow direction corresponding to the current data flow detail is determined based on the data flow summary. If the data flow summary does not contain data flow direction semantics, the data flow direction corresponding to the current data flow detail is determined based on the data flow identifier.

5. The method as described in claim 4, characterized in that, The step of determining the data flow direction corresponding to the current data flow detail according to the preset operation relationship when the first reference data reserve, the first data flow volume, and the first data reserve satisfy a preset operation relationship includes: If the sum of the first reference data balance and the first data flow volume equals the first data balance, the data flow direction corresponding to the current data flow detail is determined to be data transfer in. If the difference between the first reference data reserve and the first data flow volume is equal to the first data reserve, the data flow direction corresponding to the current data flow detail is determined to be data outflow.

6. The method as described in claim 3, characterized in that, The data flow participants include the data flow owner and the data flow related parties; The step of determining the data flow participants for each data flow detail based on the data flow direction corresponding to each data flow detail includes: If the data flow direction corresponding to any of the data flow details is data transfer in, the data transfer-in party in any of the data flow details shall be determined as the data flow owner, and the data transfer-out party in any of the data flow details shall be determined as the data flow related party. If the data flow direction corresponding to any of the data flow details is data transfer out, the data transferor in any of the data flow details shall be determined as the data flow owner, and the data transferee in any of the data flow details shall be determined as the data flow related party.

7. The method as described in claim 1, characterized in that, The verification of each data flow record based on preset data verification rules includes: Based on the data consistency verification rules, verify whether the values ​​of each field in any of the data flow records match the text content in the corresponding data flow file; Based on the data duplication verification rules, verify whether any of the data flow records is duplicated with another data flow record that has already been generated; Based on the data continuity verification rules, verify whether the data balance of any data flow record meets the flow continuity condition.

8. The method as described in claim 7, characterized in that, The data continuity verification rule, which verifies whether the data remaining amount of any data flow record meets the flow continuity condition, includes: Obtain the second reference data balance, the second data flow volume, and the second data balance for any one of the data flow records; the second reference data balance is the data balance of the previous data flow record of any one of the data flow records; the previous data flow record of any one of the data flow records is determined based on the ascending order of the data flow records according to the data flow time. If the sum of the second reference data reserve and the second data flow volume is equal to the second data reserve, or the difference between the second reference data reserve and the second data flow volume is equal to the second data reserve, the data reserve of any data flow record is checked according to the data flow direction of any data flow record to see if it meets the flow continuity condition. If the sum of the second reference data reserve and the data flow volume is not equal to the second data reserve, and the difference between the second reference data reserve and the second data flow volume is not equal to the second data reserve, then it is determined that the data reserve of any data flow record does not meet the flow continuity condition.

9. The method as described in claim 8, characterized in that, The step of verifying whether the data balance of any one of the data flow records meets the flow continuity condition based on the data flow direction of any one of the data flow records when the sum of the second reference data balance and the second data flow volume equals the second data balance, or the difference between the second reference data balance and the second data flow volume equals the second data balance, includes: If the sum of the second reference data reserve and the second data flow volume equals the second data reserve, determine whether the data flow direction of any data flow record is data transfer in; if yes, determine that the data reserve of any data flow record meets the flow continuity condition; if no, determine that the data reserve of any data flow record does not meet the flow continuity condition. If the difference between the second reference data reserve and the second data flow volume is equal to the second data reserve, determine whether the data flow direction of any data flow record is data out; if yes, determine that the data reserve of any data flow record meets the flow continuity condition; if no, determine that the data reserve of any data flow record does not meet the flow continuity condition.

10. The method as described in claim 1, characterized in that, After verifying each data flow record based on preset data verification rules, the method further includes: In response to the existence of an abnormal data transfer record that fails verification, determine the abnormal information associated with the abnormal data transfer record; The abnormal data flow record is corrected based on the abnormal information using the large language model to obtain the corrected data flow record; The corrected data flow record is used as a new data flow record, and the step of verifying each data flow record based on the preset data verification rules is executed again until each data flow record passes the verification or reaches the preset number of iterations.

11. A data aggregation device, characterized in that, The device includes: The acquisition module is used to acquire data stream files in various formats. The parsing module is used to perform semantic parsing on the text content of each of the data transfer files using a large language model, so as to obtain at least one data transfer record corresponding to each of the data transfer files. The verification module is used to verify each of the data flow records based on preset data verification rules; the data verification rules include data consistency verification rules, data repeatability verification rules, and data continuity verification rules. The generation module is used to generate a data aggregation file based on each of the verified data flow records.

12. An electronic device, characterized in that, include: A processor and a memory; the memory stores a computer program, and the processor executes the computer program to implement the method steps of any one of claims 1-10.

13. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the method steps as claimed in any one of claims 1-10.

14. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-10.

Images