Data submission method, system and device and storage medium

Abstract

The invention provides a data submission method, system and device and a storage medium, and the data submission method can effectively improve the data compression efficiency and facilitates the reduction of the data transmission and storage cost by carrying out column type compression processing on data needing to be submitted under data interfaces of different agent systems at the data level; on the data interface management level, comprehensive and systematic rule carding and integration can be carried out on different data interfaces through the established hot plug supervision rule base, online updating and real-time effectiveness of interface rules can be achieved, and shutdown adaptation of an entrusted party system is avoided; in the aspect of data storage, the row-level version number is configured for the data row in each submitted data, so that transmission type row-level version management of the historical submitted data is realized, and the data row of a specific version can be quickly positioned and processed when the historical submitted data is objection.

Description

Technical Field

[0001] This application relates to the field of financial data management technology, and more specifically, to a data reporting method, system, device, and storage medium. Background Technology

[0002] In areas of strong financial regulation, particularly in the management and operation of funds such as occupational annuities, the trustee, as a legal entity responsible for supervising and managing the fund's assets, has a legal obligation to regularly submit detailed data on the fund's investment operations to the agent (such as the social insurance agency).

[0003] Currently, the entrusted party's system typically follows a standardized data processing and transmission workflow to submit data to the agent's system. Specifically, firstly, the entrusted party's system uses an ETL (Extract, Transform, Load) program to clean and summarize the raw fund operation data obtained from various management parties (such as investment managers and custodians), and generates corresponding structured data according to the specific interface specifications provided by the agent, storing it in the corresponding interface table in the database. Secondly, in the data output stage, the entrusted party's system exports the required data from the interface table and assembles it into the file format required by the agent, such as XML or TXT files. Subsequently, the entrusted party's system encrypts these data files using a fixed key, packages them into compressed formats such as ZIP, and finally uploads them to the FTP server or API interface specified by the agent via a secure HTTPS channel or TSL protocol.

[0004] Well-known technologies that support the above process include general lossless data compression algorithms (such as GZIP, ZIP), JSON / XML / CSV processing technologies for heterogeneous data parsing, and tools for code or file version control (such as Git, SVN).

[0005] However, in practical applications, it has been found that the existing technical solutions have several inherent defects, making it difficult to meet the growing demands for data reporting efficiency, flexibility, and refined management under the environment of stringent financial regulation. Specifically: First, in terms of compression efficiency, general compression algorithms typically achieve low compression ratios (usually less than 3:1) when processing highly structured, dense detailed data with numerous columns, leading to significant waste of network bandwidth and storage resources. Second, regarding rule adaptation, when the agent updates the interface specification, the entrusted system must shut down to modify and redeploy the corresponding ETL code, resulting in a long adaptation cycle and poor system flexibility. Third, in terms of version management, existing solutions can only achieve coarse-grained version control at the file level, failing to trace specific changes at the row level within the data, making incremental reporting or historical data tracing extremely difficult. Summary of the Invention

[0006] In view of this, this application provides a data reporting method, system, device, and storage medium. At the data level, columnar compression is performed on the data to be reported under the data interfaces of different agent systems, which can effectively improve data compression efficiency and reduce data transmission and storage costs. At the data interface management level, a hot-swappable regulatory rule base is established to comprehensively and systematically sort out and integrate rules for different data interfaces, which is conducive to online updates and real-time effectiveness of interface rules and avoids downtime adaptation of the entrusted system. In terms of data storage, by configuring row-level version numbers for data rows in each reported data, a penetrating row-level version management of historical reported data is realized, which facilitates the quick location and processing of specific version data rows when there are discrepancies in historical reported data.

[0007] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings.

[0008] In a first aspect, embodiments of this application provide a data reporting method, which is applied to a trusted system, and the data reporting method includes: From a pre-established hot-swap regulatory rule base, the target data processing rules corresponding to the target data interface of the target agent system are determined; wherein, the hot-swap regulatory rule base stores data processing rules corresponding to the data interfaces of multiple agent systems respectively; According to the target data processing rules, the fund operation data associated with the target portfolio is subjected to columnar compression processing to obtain target compressed data that conforms to the target data processing rules; wherein, the target portfolio refers to the portfolio managed on behalf of the target agent system; Assign a row-level version number to each data row in the target compressed data to uniquely identify the data row, and obtain the target compressed data with the added row-level version number as the data to be reported; The data to be reported is sent to the target agent system via encrypted transmission.

[0009] Secondly, embodiments of this application provide a data reporting system, which includes at least a trusted party system and multiple agent systems, wherein the trusted party system is used for: From a pre-established hot-swap regulatory rule base, the target data processing rules corresponding to the target data interface of the target agent system are determined; wherein, the hot-swap regulatory rule base stores data processing rules corresponding to the data interfaces of multiple agent systems respectively; According to the target data processing rules, the fund operation data associated with the target portfolio is subjected to columnar compression processing to obtain target compressed data that conforms to the target data processing rules; wherein, the target portfolio refers to the portfolio managed on behalf of the target agent system; Assign a row-level version number to each data row in the target compressed data to uniquely identify the data row, and obtain the target compressed data with the added row-level version number as the data to be reported; The data to be reported is sent to the target agent system via encrypted transmission.

[0010] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described data reporting method.

[0011] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the data reporting method described above.

[0012] The technical solutions provided by the embodiments of this application may include the following beneficial effects: This application provides a data reporting method, system, device, and storage medium. At the data level, columnar compression is performed on the data to be reported under the data interfaces of different agent systems, which effectively improves data compression efficiency and helps reduce data transmission and storage costs. At the data interface management level, a hot-swappable regulatory rule base is established to comprehensively and systematically sort and integrate rules for different data interfaces, which facilitates online updates and real-time effectiveness of interface rules and avoids downtime adaptation for the entrusted system. In terms of data storage, by configuring row-level version numbers for each data row in the reported data, a penetrating row-level version management of historical reported data is achieved, which facilitates quick location and processing of specific version data rows when there are discrepancies in historical reported data. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This paper shows a schematic diagram of the structure of a data reporting system provided in an embodiment of this application; Figure 2 A flowchart illustrating a data reporting method provided in an embodiment of this application is shown; Figure 3 This is a schematic diagram of the structure of an electronic device 300 provided in an embodiment of this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0016] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0017] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0018] Here, the data reporting method provided in this application embodiment is applied to the entrusted system in the data reporting system; specifically, Figure 1 This application provides a schematic diagram of the structure of a data reporting system according to an embodiment of the present application. Figure 1 As shown, the data reporting system includes: a trustee system, multiple management systems (the target management system refers to the management system in the multiple management systems that matches the investment portfolio entrusted by the target agent), multiple agent systems (the target agent system refers to any one of the multiple agent systems), and other business systems belonging to the same trustee as the trustee system.

[0019] Here, the trustee refers to the legal entity responsible for supervising and managing the fund assets (e.g., the trustee may be a professional pension insurance company), the manager refers to the professional institution entrusted by the trustee to invest the fund (e.g., the manager may be a fund company), and the agent refers to the relevant functional department or institution that acts on behalf of the principal (government agencies and public institutions and their staff, who are the contributors and owners of rights of occupational annuities) to centrally exercise the entrusted duties and is responsible for the management of occupational annuity fund accounts (e.g., the agent may be a social insurance agency).

[0020] Specifically, in this embodiment, the management system is the system used by the management party to send relevant fund operation data to the trustee, the agent system is the system used by the agent to receive relevant reporting data on fund operation status from the trustee, the trustee system is the system used by the trustee to interface with the management system and the agent system, and the other business systems refer to other business systems on the trustee side besides the trustee system.

[0021] The following is combined Figure 1 The data reporting system shown herein provides a detailed description of the data reporting method provided in the embodiments of this application.

[0022] here, Figure 2 This paper illustrates a flowchart of a data reporting method provided in an embodiment of this application, wherein, as shown... Figure 2 As shown, the data reporting method is applied to Figure 1 In the entrusted system shown, the data reporting method includes steps S201-S204, specifically: S201, determine the target data processing rules corresponding to the target data interface of the target agent system from the pre-established hot-swappable regulatory rule base.

[0023] Here, the target agent system can refer to any one of multiple agent systems, while the target data interface refers to the data interface under the target agent system used to interface with the entrusted system; in practical applications, the target agent system can also refer to the agent system that currently has data reporting requirements.

[0024] For example, if the trustee system receives a data reporting instruction a (e.g., a request to report the fund operation status of the portfolio "Fund B1 + Fund B2") sent by the agent system A through its data interface A1, the trustee system can identify the agent system A as the target agent system, identify the data interface A1 as the target data interface, and identify the portfolio "Fund B1 + Fund B2" that is indicated to be reported in the data reporting instruction a as the target portfolio that needs to be reported.

[0025] Here, the pre-established hot-swap regulatory rule base stores data processing rules corresponding to the data interfaces of multiple agent systems. Taking data interface A1 of agent system A as an example, the data processing rules corresponding to data interface A1 are used to indicate the relevant rules that the data sent by the entrusted system must conform to when it sends data to agent system A through data interface A1. Therefore, the hot-swap regulatory rule base can be constructed based on the rule files of the data interfaces provided by multiple agent systems.

[0026] Specifically, as an optional implementation, on the entrusted system side, the hot-swap regulatory rule base can be established using the methods shown in steps a1-a4 below: Step a1: Receive the data processing rules sent by the multiple agent systems respectively.

[0027] Here, the entrusted system can receive rule files for the data interfaces sent by multiple agent systems respectively, and use the received rule files as the data processing rules given by each agent system for its own data interface (that is, the relevant rules that the data sent must conform to when sending data to the agent system through the data interface).

[0028] Step a2: For each of the data processing rules, extract key information related to the first agent and the first data interface from the data processing rule, and merge and integrate the key information to form a structured description file associated with the first agent system.

[0029] In this embodiment of the application, the entrusted party system integrates and processes all received rule files (i.e., data processing rules sent by multiple agent systems respectively) in a unified manner, and generates a rule base file corresponding to the data interface of each agent system. This file serves as the specific data processing rule determined by the entrusted party system for a specific agent system (i.e., the target agent system) when executing step S201.

[0030] It should be noted that, for the sake of clarity in describing the scheme, in this embodiment of the application, the first agent system refers to the agent system corresponding to the data processing rule, the first data interface refers to the data interface of the first agent system, and the first agent refers to the agent corresponding to the first agent system. That is, the above steps a2-a3 are the specific processing procedures performed by the entrusted system for the data processing rule sent by an agent system (i.e., the first agent system). The processing method performed by the entrusted system for the data processing rule sent by each agent system is the same, and the repeated parts will not be described again here.

[0031] Specifically, taking the first agent system as an example, in step a2, the key information extracted by the entrusted system from the data processing rules sent by the first agent system may include, but is not limited to: agent name (e.g., the agent name of the first agent mentioned above), agent code (e.g., the agent code of the first agent mentioned above), data interface number required by the agent (e.g., the data interface number of the first data interface mentioned above), and detailed information of the first data interface (including but not limited to: interface name, interface code, interface naming rules, interface reporting frequency, interface header information, English name of interface field, Chinese name of interface field, data length of interface field, data type of interface field, interface field generation rules, interface field enumeration value, whether the interface contains attachments, attachment naming rules, specific path of attachment, attachment length, attachment suffix, etc.).

[0032] Specifically, after extracting the aforementioned key information, the entrusted system can merge and integrate the key information to form a structured description file (e.g., an XML file) associated with the first agent system; wherein, the structured description file contains various key information related to the first agent and the first data interface (refer to the aforementioned description of key information).

[0033] Step a3: Based on the structured description file and the pre-stored enumeration information table, convert the structured description file into a rule base file with a nested key-value pair structure.

[0034] Here, the trustee system stores an enumeration information table containing the relationships between different agents and different managers. The enumeration information table includes, but is not limited to, the following information: agent name, agent code, trustee name, trustee code, manager name, manager code, and the plan code (i.e., the plan code representing the business plan to which the agent belongs) and portfolio code (i.e., the portfolio code representing the portfolio managed by the agent).

[0035] Here, combined with the above enumerated information table, various key information in the above structured description file can be converted into a rule base file with a nested key-value pair structure; wherein, the nested key-value pair structure includes multiple key-value pairs, which are determined according to multiple different interface fields associated with the first data interface and the field mapping rules corresponding to each interface field.

[0036] Specifically, the entrusted system can perform entity recognition and matching on various key information in the structured description file based on the enumerated information table, and obtain multiple matching key-value pairs; then, according to the nesting rules, it iterates through the multiple matching key-value pairs to generate a rule base file with a nested key-value pair structure.

[0037] It should be noted that in the rule base file, the top-level key (i.e., the key value at the top level in the nested key-value pair structure) can be the data interface number (e.g., the data interface number of the first data interface), and the bottom-level key (i.e., the key value at the bottom level in the nested key-value pair structure) can be the data interface number field code. Between the top-level key and the bottom-level key, there can be multiple key-value pairs. The key in each key-value pair can be the field name of an interface field, and the value in each key-value pair can be the field mapping rule corresponding to the interface field represented by the key.

[0038] For example, the top-level Key can be "BAD01003_A00 A1" (A1 is the data interface number of the first data interface), and the Value can be a nested set of N Key-Value data, such as Key1 = "Collection Code Set", Value1 = "{I54Z0001,I54Z0002,I54Z0003}" (equivalent to the first agent having 3 different investment portfolios), Key2 = "Data Interface Field Set", Value2 = "{key='Appseriono', value='Business Sequence Number', key='Busiseriono', value='Data Sequence Number', key='Price', value='Collection Market Value'}" (equivalent to the field mapping rule corresponding to the data interface field Appseriono being the business sequence number, the field mapping rule corresponding to the data interface field Busiseriono being the data sequence number, and the field mapping rule corresponding to the data interface field Price being the portfolio market value). Storing such nested Key-Value sets in a file forms the rule base file corresponding to the first data interface of the first agent system.

[0039] Step a4: Based on the rule base files corresponding to the data interfaces of the multiple agent systems, establish and generate the hot-swappable regulatory rule base.

[0040] Here, by storing the aforementioned rule base files corresponding to the data interfaces of each agent's system, the aforementioned hot-swappable regulatory rule base, composed of all the rule base files, can be obtained.

[0041] It should be noted that in the aforementioned hot-swappable regulatory rule base, the rule base file can be placed in a folder on shared storage, which can be read by the entrusted system at any time, and can also be added, deleted, modified, and queried online by the entrusted system's operation and maintenance personnel. Furthermore, the generation logic of the aforementioned rule base file is an independent program, so it can be updated according to the data processing rules sent by the agent system, ensuring that the hot-swappable regulatory rule base can be updated in real time according to the latest interface requirements for data reporting of different agent systems.

[0042] S202, according to the target data processing rules, the fund operation data associated with the target portfolio is subjected to columnar compression processing to obtain target compressed data that conforms to the target data processing rules.

[0043] Here, the target portfolio refers to the portfolio managed on behalf of the target agent system. For example, if the portfolio managed on behalf of the target agent system is "Fund B1 + Fund B2", then the above target portfolio can be "Fund B1 + Fund B2".

[0044] In the embodiments of this application, such as Figure 1 As shown, the entrusted party system can receive data from two sources: one is the management party system corresponding to the management party, and the other is other business systems on the entrusted party side besides the aforementioned entrusted party system.

[0045] Based on this, in step S202, the aforementioned fund operation data associated with the target investment portfolio may include: first fund operation data and second fund operation data; wherein, the first fund operation data represents external fund operation data sent from the target management system, and the second fund operation data represents fund operation data from other business systems within the trustee, and the target management system is the management system corresponding to the target investment portfolio (for example, if the target investment portfolio is "fund B1 + fund B2", then the target management system may be the management system corresponding to the management system of the management system responsible for investing in fund B1 and the management system of the management system responsible for investing in fund B2, respectively).

[0046] Here, a low-power dense compression engine is deployed in the trustee system. This engine can perform columnar compression processing on the aforementioned fund operation data according to steps b1-b4 below. Specifically: Step b1: According to the target data processing rules, the fund operation data is split into columns to obtain common column data, business column data and reporting information.

[0047] Here, the target data processing rule is the rule base file corresponding to the target data interface of the target agent system in the hot-swappable supervision rule base mentioned above. As can be seen from the aforementioned description of the rule base file, the rule base file contains multiple key-value pairs. Based on the interface field represented by the key value in the key-value pair, the field mapping rule corresponding to the interface field (i.e., the value corresponding to the key value) can be queried.

[0048] Based on this, according to the data definitions corresponding to the public column data, business column data, and reporting information (i.e., the interface fields contained in the public column data, business column data, and reporting information respectively), by querying multiple key-value pairs contained in the above rule base file, the specific data corresponding to the public column data, business column data, and reporting information can be obtained from the above fund operation data (equivalent to the specific data corresponding to the value of the interface field in the above fund operation data).

[0049] Specifically, public column data may include, but is not limited to: data interface authentication, rule definition, field definition, etc.; business column data may include, but is not limited to: business data contained in the final xml / txt file (i.e. the final data reporting file); reporting information may include, but is not limited to: related file paths, attachment information, etc.

[0050] For example, the entrusted system can first read the rule base file corresponding to the target data processing rule, then expand the KV mapping relationship in the rule base file, and iteratively read the nested Key, Value sets in the rule base file. For example, if Key='BAD01003_A00 A1' is read, the Value is first obtained from the {rule text} corresponding to Key='plan code set', Values={'A1 insurance combination 1', 'A1 insurance combination 2'}. Then, it iterates over the plan code set, using a fixed ETL to read the above fund operation data, and calculates and summarizes the result data E required for the BAD01003 interface (i.e., the target data interface); then, according to the {rule text} Key='data interface field set', Value={Key='Appseriono', Value='business sequence number', Key='Price', The K1 and V1 sets of {Value='Planning Layer Market Value', Key='Busiseriono', Value='Data Sequence Number'} and Key='Data Interface Field Classification', Value={Key='Appseriono', Value='Common Column', Key='Price', Value='Business Column', Key='Busiseriono', Value='Reporting Information Column'} are used to perform columnar splitting on the result data E, dividing it into 3 data: E1 Common Column Data, E2 Business Column Data, and E3 Reporting Information.

[0051] Step b2: According to the target data processing rules, the business column data is cropped and differentially encoded to obtain target business column data whose field length conforms to the target data processing rules.

[0052] It should be noted that, in this embodiment of the application, since the common column data is basic data that is common to different data interfaces, it is not necessary to process it according to the exclusive data processing rules (i.e., target data processing rules) corresponding to the target data interface.

[0053] Here, based on multiple key-value pairs contained in the rule base file (i.e., target data processing rules), the value corresponding to the key of the interface field that belongs to the numeric type (e.g., amount, quantity, etc.) can be used to dynamically identify numeric field data from the business column data. By performing length pruning and differential encoding on the identified numeric field data, the target business column data whose field length conforms to the rule base file (i.e., target data processing rules) can be obtained.

[0054] Step b3: According to the target data processing rules, process the target column data in the reporting information, and perform horizontal compression on the processed reporting information to obtain target reporting information that conforms to the target data processing rules.

[0055] Here, the target column data mentioned above may include, but is not limited to: the first target column data containing strings and the second target column data involving special processing requirements (i.e., the exclusive requirements unique to the target data results corresponding to the target data processing rules).

[0056] Specifically, when executing step b3, the entrusted system can, according to the target data processing rules, truncate the length of the first target column data containing the string in the reported information to obtain the first target column data whose length conforms to the target data processing rules.

[0057] Here, similar to step b2, based on the multiple key-value pairs contained in the rule base file (i.e., the target data processing rules), the entrusted system can dynamically identify the first target column data belonging to the string type from the reported information according to the value corresponding to the key of the interface field belonging to the string type field (such as date, serial number, etc.). By truncating the length of the identified first target column data, the first target column data with a length conforming to the rule base file (i.e., the target data processing rules) can be obtained.

[0058] Specifically, when executing step b3, the entrusted system can perform targeted processing on the second target column data with specific requirements in the reported information according to the target data processing rules, so as to obtain the second target column data that conforms to the target data processing rules.

[0059] It should be noted that the above-mentioned specific requirements can be determined based on the relevant rule files corresponding to the target data interface actually sent by the target agent system. For example, the above-mentioned specific requirements may be "the file path needs to be renamed according to the interface version", etc. This application embodiment does not impose any limitations on this.

[0060] It should be noted that after horizontal compression of the processed reporting information, a dynamic data dictionary can be formed. This dynamic data dictionary can be dynamically updated as the rule base files in the hot-swappable regulatory rule base change, in order to meet the latest data reporting requirements of the agent's system.

[0061] Step b4: Merge the common column data, the target business column data, and the target reporting information to obtain merged data, and then perform column-based compression processing on the merged data to obtain the target compressed data.

[0062] Specifically, the entrusted system can perform columnar compression processing on the merged data according to the following steps c1-c3: Step c1: Before performing the merging, obtain the set of column names of the target business column data in advance from the target business column data.

[0063] Here, taking the target business column data containing N rows and N columns of data as an example, before performing the above merging, we can obtain the column names corresponding to the N columns of data from the target business column data to obtain the above set of column names.

[0064] Step c2: After obtaining the merged data, perform double loop processing on the target business column data and the column name set in the merged data to obtain single-column compressed data corresponding to the target business column data.

[0065] Specifically, based on the set of column names (denoted as Set2) of the target business column data (denoted as E2) obtained before the merger, the entrusted system can perform double loop processing on the entire E2 and the set of column names Set2. When looping through each row in E2, it can loop through Set2, using each column name in Set2 as the key of the dictionary, and using the value in the cell of the row in E2 (i.e. the row corresponding to the current loop processing) and the column (i.e. the column corresponding to the column name being processed in the current loop) as the value of the dictionary.

[0066] Specifically, after each row loop, the entrusted system can put the column names and cell values ​​involved in Set2 into a JSON dictionary. Based on this, after all rows in E2 have been looped, the original N rows and N columns of E2 can be compressed into an N rows and 1 column of JSON dataset (i.e., the single-column compressed data corresponding to the target business column data E2).

[0067] Step c3: Replace the target business column data in the merged data with the single-column compressed data to obtain the target compressed data.

[0068] Here, after compressing the target business column data into the single-column compressed data described above, you only need to replace the target business column data in the merged data with the single-column compressed data to obtain the target compressed data.

[0069] S203, assign a row-level version number to each data row in the target compressed data to uniquely identify the data row, and obtain the target compressed data with the added row-level version number as the data to be reported.

[0070] Here, by configuring row-level version numbers for each data row in the submitted data, this embodiment of the application enables penetrating row-level version management of historical submitted data, which facilitates quick location and processing of specific version data rows when there are discrepancies in historical submitted data.

[0071] Specifically, when executing step S203, the entrusted system can generate the row-level version number corresponding to each data row in the target compressed data according to the method shown in steps d1-d3 below: Step d1: Based on the historical data reported to the target agent system through the target data interface in the previous cycle, obtain the data version number of the historical data reported, and generate an incremental data version number corresponding to the target compressed data based on the data version number.

[0072] Here, for each data row, the row-level version number corresponding to that data row can be composed of the following three parts: an incrementing data version number, a row-level data fingerprint, and a timestamp.

[0073] Specifically, the incremental data version number is determined by the historical data reported by the same data interface under the same reporting period.

[0074] For example, taking a monthly reporting cycle as an example, the entrusted system can obtain the data version number x1 of the historical reporting data of the target data interface (i.e., the same interface) in the same reporting cycle (such as the previous month), in the same reporting interval (e.g., if the current month is February, then the same reporting interval can be January), and in the same investment portfolio (i.e., the target investment portfolio); then, based on the data version number x1, a preset step size (such as adding one) is added to obtain the incremental data version number corresponding to the target compressed data as "x1+1".

[0075] Step d2: Calculate the hash value of each data row in the target compressed data to obtain the row-level data fingerprint corresponding to each data row.

[0076] Here, when calculating the hash value, the MurmurHash3 hash algorithm can be used. In order to ensure that the result is positive, the ABS method can also be used, and the modulo operation %(10 ** m) is performed. Here, m is the original total number of rows to ensure that the hash value is mapped to a fixed range, so that the row-level data fingerprint corresponding to each data row is unique.

[0077] Therefore, if it is found that there are missing or incomplete data in the data that has been reported after the data reporting is completed, and incremental reporting is required, the range of data rows that need to be incrementally reported can be determined according to the above row-level data fingerprint, thereby avoiding data regeneration and helping to improve reporting efficiency.

[0078] Step d3: Generate the row-level version number corresponding to each data row based on the timestamp corresponding to the current time, the incrementing data version number, and the row-level data fingerprint corresponding to each data row.

[0079] Here, the row-level version number corresponding to the data row can be composed of the timestamp, the incremental data version number, and the row-level data fingerprint mentioned above. This ensures that each data row in each data report corresponds to a unique row-level version number, realizing penetrating row-level version management of historical data reports. This facilitates the quick location and processing of specific version data rows when there are discrepancies in historical data reports.

[0080] S204, The data to be reported is sent to the target agent system via encrypted transmission.

[0081] Here, the entrusted system can encrypt the data to be reported according to the data encryption method corresponding to the target data interface, and send the encrypted data to be reported to the target data interface corresponding to the target agent system. The data encryption methods corresponding to the data interfaces under different agent systems may be different. This application embodiment does not limit the specific data encryption method used in actual application.

[0082] Based on the data reporting method provided in this application embodiment, at the data level, columnar compression processing is performed on the data to be reported under the data interfaces of different agent systems, which can effectively improve data compression efficiency and reduce data transmission and storage costs. At the data interface management level, the hot-swappable regulatory rule base can be established to comprehensively and systematically sort out and integrate the rules of different data interfaces, which is conducive to realizing online updates and real-time effectiveness of interface rules and avoiding downtime adaptation of the entrusted system. In terms of data storage, by configuring row-level version numbers for data rows in each reported data, penetrating row-level version management of historical reported data is realized, which makes it easy to quickly locate and process data rows of specific versions when there are disputes in historical reported data.

[0083] Based on the same inventive concept, this application also provides a data reporting system corresponding to the above data reporting method. Since the principle of solving the problem by the data reporting system in the embodiments of this application is similar to that of the above data reporting method in the embodiments of this application, the implementation of the data reporting system can refer to the implementation of the above data reporting method, and the repeated parts will not be described again.

[0084] like Figure 1 As shown, the data reporting system includes: a trustee system, multiple management systems, multiple agent systems, and other business systems belonging to the same trustee as the trustee system; wherein, the trustee system is used for: From a pre-established hot-swap regulatory rule base, the target data processing rules corresponding to the target data interface of the target agent system are determined; wherein, the hot-swap regulatory rule base stores data processing rules corresponding to the data interfaces of multiple agent systems respectively; According to the target data processing rules, the fund operation data associated with the target portfolio is subjected to columnar compression processing to obtain target compressed data that conforms to the target data processing rules; wherein, the target portfolio refers to the portfolio managed on behalf of the target agent system; Assign a row-level version number to each data row in the target compressed data to uniquely identify the data row, and obtain the target compressed data with the added row-level version number as the data to be reported; The data to be reported is sent to the target agent system via encrypted transmission.

[0085] In one alternative implementation, the entrusted system is used to establish the hot-swap regulatory rule base by means of: Receive data processing rules sent by the multiple agent systems respectively; For each data processing rule, key information related to the first agent and the first data interface is extracted from the data processing rule, and the key information is merged and integrated to form a structured description file associated with the first agent system; wherein, the first agent system refers to the agent system corresponding to the data processing rule, the first data interface refers to the data interface of the first agent system, and the first agent refers to the agent corresponding to the first agent system. Based on the structured description file and the pre-stored enumeration information table, the structured description file is converted into a rule base file with a nested key-value pair structure; wherein, the nested key-value pair structure includes multiple key-value pairs, and the multiple key-value pairs are determined according to multiple different interface fields associated with the first data interface and the field mapping rules corresponding to each interface field; Based on the rule base files corresponding to the data interfaces of the multiple agent systems, the hot-swap regulatory rule base is established and generated.

[0086] In one optional implementation, the fund operation data includes: first fund operation data and second fund operation data; wherein, the first fund operation data represents external fund operation data sent from the target management system, the second fund operation data represents fund operation data from other business systems within the trustee, and the target management system represents the management system corresponding to the target investment portfolio.

[0087] In an optional implementation, when performing columnar compression processing on the fund operation data associated with the target portfolio according to the target data processing rules, the trustee system is used to: According to the target data processing rules, the fund operation data is columnarly split to obtain common column data, business column data and reporting information; According to the target data processing rules, the business column data is trimmed and differentially encoded to obtain target business column data whose field length conforms to the target data processing rules; According to the target data processing rules, the target column data in the reporting information is processed, and the processed reporting information is horizontally compressed to obtain target reporting information that conforms to the target data processing rules. The common column data, the target business column data, and the target reporting information are merged to obtain merged data, and the merged data is then subjected to column-based compression to obtain the target compressed data.

[0088] In one optional implementation, when processing the target column data in the reported information according to the target data processing rules, the entrusted system is used to: According to the target data processing rules, the length of the first target column data containing strings in the reported information is truncated to obtain the first target column data whose length conforms to the target data processing rules. According to the target data processing rules, the second target column data with specific requirements in the reported information is processed in a targeted manner to obtain the second target column data that conforms to the target data processing rules.

[0089] In one alternative implementation, when performing columnar compression on the merged data, the entrusted system is configured to: Before performing the merging, the set of column names of the target business column data is obtained in advance from the target business column data; After obtaining the merged data, the target business column data and the column name set in the merged data are processed in a double loop to obtain the single-column compressed data corresponding to the target business column data. The target business column data in the merged data is replaced with the single-column compressed data to obtain the target compressed data.

[0090] In one alternative implementation, when assigning a line-level version number to each data line in the target compressed data to uniquely identify that data line, the entrusted system is configured to: Based on the historical data reported to the target agent system through the target data interface in the previous cycle, obtain the data version number of the historical data reported, and generate an incremental data version number corresponding to the target compressed data based on the data version number; Calculate the hash value for each data row in the target compressed data to obtain the row-level data fingerprint corresponding to each data row; Based on the timestamp corresponding to the current moment, the incrementing data version number, and the row-level data fingerprint corresponding to each data row, the row-level version number corresponding to each data row is generated.

[0091] Based on the same inventive concept, this application also provides an electronic device corresponding to the above-described data reporting method. Since the principle of solving the problem by the electronic device in the embodiments of this application is similar to that of the above-described data reporting method in the embodiments of this application, the implementation of the electronic device can refer to the implementation of the above-described data reporting method, and the repeated parts will not be described again.

[0092] Figure 3 This is a schematic diagram of the structure of an electronic device 300 provided in an embodiment of the present application. It includes a processor 301, a memory 302, and a bus 303. The memory 302 stores machine-readable instructions that can be executed by the processor 301. When the electronic device runs a data reporting method as described in the embodiment, the processor 301 and the memory 302 communicate through the bus 303. The processor 301 executes the machine-readable instructions, wherein the processor 301 implements the above-mentioned data reporting method when executing the machine-readable instructions.

[0093] Specifically, the memory 302 and processor 301 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 301 runs the computer program stored in the memory 302, it can execute the above data reporting method.

[0094] Corresponding to the data reporting method in this application, this application embodiment also provides a computer-readable storage medium storing a computer program, which is executed by a processor to perform the steps of the above-described data reporting method.

[0095] Specifically, the storage medium can be a general-purpose storage medium, such as a removable disk or hard disk, and when the computer program on the storage medium is run, it can execute the above-mentioned data reporting method.

[0096] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0097] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0098] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0099] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0101] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A data reporting method, characterized in that, The data reporting method is applied to the entrusted party's system, and the data reporting method includes: From a pre-established hot-swap regulatory rule base, the target data processing rules corresponding to the target data interface of the target agent system are determined; wherein, the hot-swap regulatory rule base stores data processing rules corresponding to the data interfaces of multiple agent systems respectively; According to the target data processing rules, the fund operation data associated with the target portfolio is subjected to columnar compression processing to obtain target compressed data that conforms to the target data processing rules; wherein, the target portfolio refers to the portfolio managed on behalf of the target agent system; Assign a row-level version number to each data row in the target compressed data to uniquely identify the data row, and obtain the target compressed data with the added row-level version number as the data to be reported; The data to be reported is sent to the target agent system via encrypted transmission.

2. The data reporting method according to claim 1, characterized in that, The hot-plug regulatory rule base is established using the following method: Receive data processing rules sent by the multiple agent systems respectively; For each data processing rule, key information related to the first agent and the first data interface is extracted from the data processing rule, and the key information is merged and integrated to form a structured description file associated with the first agent system; wherein, the first agent system refers to the agent system corresponding to the data processing rule, the first data interface refers to the data interface of the first agent system, and the first agent refers to the agent corresponding to the first agent system. Based on the structured description file and the pre-stored enumeration information table, the structured description file is converted into a rule base file with a nested key-value pair structure; wherein, the nested key-value pair structure includes multiple key-value pairs, which are determined according to multiple different interface fields associated with the first data interface and the field mapping rules corresponding to each interface field; Based on the rule base files corresponding to the data interfaces of the multiple agent systems, the hot-swap regulatory rule base is established and generated.

3. The data reporting method according to claim 1, characterized in that, The fund operation data includes: first fund operation data and second fund operation data; wherein, the first fund operation data represents external fund operation data sent from the target management system, and the second fund operation data represents fund operation data from other business systems within the trustee, and the target management system represents the management system corresponding to the target investment portfolio.

4. The data reporting method according to claim 1, characterized in that, The step of performing columnar compression processing on the fund operation data associated with the target investment portfolio according to the target data processing rules includes: According to the target data processing rules, the fund operation data is columnarly split to obtain common column data, business column data and reporting information; According to the target data processing rules, the business column data is trimmed and differentially encoded to obtain target business column data whose field length conforms to the target data processing rules; According to the target data processing rules, the target column data in the reporting information is processed, and the processed reporting information is horizontally compressed to obtain target reporting information that conforms to the target data processing rules. The common column data, the target business column data, and the target reporting information are merged to obtain merged data, and the merged data is then subjected to column-based compression to obtain the target compressed data.

5. The data reporting method according to claim 4, characterized in that, The step of processing the target column data in the reported information according to the target data processing rules includes: According to the target data processing rules, the length of the first target column data containing strings in the reported information is truncated to obtain the first target column data whose length conforms to the target data processing rules. According to the target data processing rules, the second target column data with specific requirements in the reported information is processed in a targeted manner to obtain the second target column data that conforms to the target data processing rules.

6. The data reporting method according to claim 4, characterized in that, The columnar compression process for the merged data includes: Before performing the merging, the set of column names of the target business column data is obtained in advance from the target business column data; After obtaining the merged data, the target business column data and the column name set in the merged data are processed in a double loop to obtain the single-column compressed data corresponding to the target business column data. The target business column data in the merged data is replaced with the single-column compressed data to obtain the target compressed data.

7. The data reporting method according to claim 1, characterized in that, Assigning a line-level version number to each data line in the target compressed data to uniquely identify that data line includes: Based on the historical data reported to the target agent system through the target data interface in the previous cycle, obtain the data version number of the historical data reported, and generate an incremental data version number corresponding to the target compressed data based on the data version number; Calculate the hash value for each data row in the target compressed data to obtain the row-level data fingerprint corresponding to each data row; Based on the timestamp corresponding to the current moment, the incrementing data version number, and the row-level data fingerprint corresponding to each data row, the row-level version number corresponding to each data row is generated.

8. A data reporting system, characterized in that, The data reporting system includes at least a trustee system and multiple agent systems, wherein the trustee system is used for: From a pre-established hot-swap regulatory rule base, the target data processing rules corresponding to the target data interface of the target agent system are determined; wherein, the hot-swap regulatory rule base stores data processing rules corresponding to the data interfaces of multiple agent systems respectively; According to the target data processing rules, the fund operation data associated with the target portfolio is subjected to columnar compression processing to obtain target compressed data that conforms to the target data processing rules; wherein, the target portfolio refers to the portfolio managed on behalf of the target agent system; Assign a row-level version number to each data row in the target compressed data to uniquely identify the data row, and obtain the target compressed data with the added row-level version number as the data to be reported; The data to be reported is sent to the target agent system via encrypted transmission.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the data reporting method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the data reporting method as described in any one of claims 1 to 7.

Images