A number processing method and apparatus

Abstract

The application discloses a kind of for number processing method and device, it is related to big data data processing technical field.The specific embodiment of the method includes: in response to reach the time point of for number of receiving system, obtain the for number parameter agreed in advance with the receiving system;Wherein, the for number parameter includes table name and whether the parameter of centralized for number;Query the data corresponding to the table name from database, execute the number of data and obtain the for number file, generate the filename of the for number file according to the for number parameter;According to the value of whether the centralized for number parameter, determine the for number mode, construct the transmission channel of the receiving system, to use the for number mode and transmit the for number file to the receiving system through the transmission channel.This embodiment realizes the demand of compatible direct connection for number and centralized for number by whether the for number parameter in new addition is divided into rows and whether the for number parameter is centralized, and for number parameter can be adjusted according to actual, with flexible compatibility.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, and in particular to a data processing method and apparatus. Background Technology

[0002] In enterprise systems, based on different business needs, each physical subsystem typically needs to perform data analysis and processing through file interaction with upstream and downstream physical subsystems. There are two main methods of file interaction: direct data supply (point-to-point file interaction) and centralized data supply, which uses a data warehouse as an intermediary. The rules for file interaction between the upstream and downstream physical subsystems and the data warehouse differ between these two methods.

[0003] The number of direct data supply systems will continue to increase as the business expands. When the number of direct data supply systems or data supply files increases, code development is required based on the needs of the new direct data supply systems. When the new data supply files need to be centrally supplied, further development is required based on the data warehouse rules. This results in a long development and testing cycle, low efficiency, and poor compatibility.

[0004] When data warehouse rules change, centralized data supply requires a global overhaul based on the existing centralized data supply mechanism. When files are supplied through both centralized and direct connections, the modified code must meet the centralized data supply rules and be compatible with the original direct connection data supply rules. This necessitates a direct connection data supply overhaul based on the centralized data supply mechanism, resulting in a large workload for code modifications, long development and testing cycles, low efficiency, and poor compatibility. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a data supply processing method and apparatus, which, while ensuring the existing direct data supply and centralized data supply, develops a flexible approach to merging data supply files to meet different needs for adding direct data supply and modifying centralized data supply.

[0006] To achieve the above objectives, according to one aspect of the present invention, a data processing method is provided, comprising:

[0007] In response to reaching the data supply time point to the receiving system, the data supply parameters agreed upon with the receiving system in advance are obtained; wherein, the data supply parameters include table name and whether centralized data supply is used;

[0008] Query the database for data corresponding to the table name, perform a data unloading operation on the data to obtain a data supply file, and generate the file name of the data supply file according to the data supply parameters;

[0009] The data supply method is determined based on the value of the centralized data supply parameter, and a transmission channel with the receiving system is constructed so that the data supply file can be transmitted to the receiving system through the transmission channel using the data supply method.

[0010] Optionally, the data supply parameters may also include: organization type, whether to use a line parameter, data type, batch, and file type;

[0011] The step of generating the filename of the data supply file based on the data supply parameters includes:

[0012] Query the organization number corresponding to the value of the "whether to split into rows" parameter from the organization mapping relationship;

[0013] The filename of the data supply file is generated by sequentially concatenating the organization type, organization number, data type, table name, batch, and file type.

[0014] Optionally, the mechanism mapping relationship is configured in a first configuration file, and the data supply parameters are configured in a second configuration file;

[0015] The step of querying the organization number corresponding to the value of the "whether to split into rows" parameter from the organization mapping relationship includes:

[0016] In response to the condition that the branch line parameter is a first value, the head office organization number is queried from the organization mapping relationship;

[0017] In response to the condition that the branching parameter is a second value, all branching organization numbers are queried from the organization mapping relationship.

[0018] Optionally, the method further includes:

[0019] Obtain the current date, and generate the filename of the data supply file based on the data supply parameters and the current date.

[0020] Optionally, determining the data supply method based on the value of the centralized data supply parameter includes:

[0021] In response to the "whether to provide centralized data" parameter being "yes", the data supply method is determined to be centralized data supply, and the receiving system is determined to be a data warehouse; or

[0022] In response to the condition that the centralized data supply parameter is no, the data supply method is determined to be direct connection data supply, and the receiving system is determined to be another physical subsystem.

[0023] Optionally, the transmission channel between the construction and the receiving system includes:

[0024] Based on the system number of the receiving system, query the transmission node number of the receiving system to establish a transmission channel with the transmission node number.

[0025] Optionally, the method further includes: querying the security node number of the receiving system, wherein the transmission node number depends on the security node number, and the security node number is used to encrypt the transmission channel.

[0026] To achieve the above objectives, according to another aspect of the present invention, a data processing apparatus is provided, comprising:

[0027] The acquisition module is used to acquire the data supply parameters pre-agreed with the receiving system in response to the arrival of the data supply time point to the receiving system; wherein, the data supply parameters include table name and whether centralized data supply is used.

[0028] The unloading module is used to query data corresponding to the table name from the database, perform unloading operations on the data to obtain a data supply file, and generate the file name of the data supply file according to the data supply parameters.

[0029] The data supply module is used to determine the data supply method based on the value of the centralized data supply parameter, and to construct a transmission channel with the receiving system so as to transmit the data supply file to the receiving system through the transmission channel using the data supply method.

[0030] Optionally, the data supply parameters may also include: organization type, whether to use a line parameter, data type, batch, and file type;

[0031] The unloading module is used for:

[0032] Query the organization number corresponding to the value of the "whether to split into rows" parameter from the organization mapping relationship;

[0033] The filename of the data supply file is generated by sequentially concatenating the organization type, organization number, data type, table name, batch, and file type.

[0034] Optionally, the unloading module is used for:

[0035] Obtain the current date, and generate the filename of the data supply file based on the data supply parameters and the current date.

[0036] Optionally, the mechanism mapping relationship is configured in a first configuration file, and the data supply parameters are configured in a second configuration file;

[0037] The unloading module is used for:

[0038] In response to the condition that the branch line parameter is a first value, the head office organization number is queried from the organization mapping relationship;

[0039] In response to the condition that the branching parameter is a second value, all branching organization numbers are queried from the organization mapping relationship.

[0040] Optionally, the data supply module is used for:

[0041] In response to the "whether to provide centralized data" parameter being "yes", the data supply method is determined to be centralized data supply, and the receiving system is determined to be a data warehouse; or

[0042] In response to the condition that the centralized data supply parameter is no, the data supply method is determined to be direct connection data supply, and the receiving system is determined to be another physical subsystem.

[0043] Optionally, the data supply module is used for:

[0044] Based on the system number of the receiving system, query the transmission node number of the receiving system to establish a transmission channel with the transmission node number.

[0045] Optionally, the data supply module is further configured to: query the security node number of the receiving system, wherein the transmission node number depends on the security node number, and the security node number is used to encrypt the transmission channel.

[0046] To achieve the above objectives, according to another aspect of the present invention, a data processing electronic device is provided.

[0047] The electronic device of this invention includes: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the data processing methods described above.

[0048] To achieve the above objectives, according to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the data processing methods described above.

[0049] To achieve the above objectives, according to another aspect of the present invention, a computing program product is provided. The computing program product of the present invention includes a computer program, which, when executed by a processor, implements the data processing method provided in the present invention.

[0050] According to the solution provided by the present invention, one embodiment of the invention has the following advantages or beneficial effects: while ensuring that the existing direct data supply mechanism and centralized data supply mechanism remain unchanged, a new configuration file is added to the data supply system. The configuration file sets data supply parameters that can be flexibly configured and modified. By using the parameters of whether to supply data in different branches and whether to supply data in a centralized manner, the needs of direct data supply and centralized data supply that are adapted to different characteristic businesses and system rules can be realized.

[0051] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0052] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0053] Figure 1 This is a schematic diagram illustrating the data supply relationships between the existing systems;

[0054] Figure 2 This is a schematic diagram of the data supply configuration and combination in this scheme;

[0055] Figure 3 This is a schematic diagram of the main flow of a data processing method according to an embodiment of the present invention;

[0056] Figure 4 This is a diagram illustrating how the data supply system generates the data supply file name;

[0057] Figure 5 This is a flowchart illustrating a specific data supply processing method according to an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of the main modules of a data processing device according to an embodiment of the present invention;

[0059] Figure 7 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0060] Figure 8 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present invention, such as a mobile device or server. Detailed Implementation

[0061] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0062] It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of this invention can be combined with each other. The acquisition, storage, analysis, use, and processing of data (such as user personal information) in the technical solutions of this application all comply with relevant laws and regulations, are used for legitimate and reasonable purposes, and are not shared, disclosed, or sold outside of these legitimate uses, and are subject to supervision and management by regulatory authorities. Necessary measures should be taken to prevent unauthorized access to such personal information data, ensure that personnel authorized to access personal information data comply with relevant laws and regulations, and ensure the security of user personal information. Once such user personal information data is no longer needed, the risk should be minimized by restricting or even prohibiting data collection and / or deleting the data. When applicable, including in certain relevant applications, user privacy may be protected by de-identifying data, for example, by removing specific identifiers, controlling the amount or specificity of stored data, controlling how data is stored, and / or other de-identification methods.

[0063] The following is an explanation of the terms used in this plan:

[0064] Data supply: In enterprise systems (such as banking systems), when data is exchanged between systems with different business functions or technical functions, the provision of data or data-related files by one system to another is called data supply.

[0065] Data warehouse: In enterprise-level data management, a data warehouse can be used to store, process, forward, and customize data or data-related files.

[0066] Unloading data: This does not mean uninstalling or deleting data. Instead, it involves retrieving data from database tables based on specific field order and conditions using database query statements, generating a data file with a specified filename, and storing it in a specific path on the server.

[0067] In enterprise systems, systems with different business and technical functions may exist as follows: Figure 1 The diagram illustrates one or more data supply relationships, such as physical subsystem 1 directly supplying data to physical subsystem 2; physical subsystems 6 and 7 having a direct data supply relationship; physical subsystem 12 being directly supplied to physical subsystem 11 and centrally supplying data to the data warehouse; physical subsystem 14 centrally supplying data to the data warehouse and directly supplying data to physical subsystem 13; physical subsystem 18 and the data warehouse having a mutually centralized data supply relationship; physical subsystem 8 centrally supplying data to the data warehouse, and the data warehouse centrally supplying data to physical subsystem 10.

[0068] Based on the data supply relationships of each physical subsystem, the data supply methods are mainly divided into two types: direct connection data supply and centralized data supply. The rules for the two are different, so the data supply physical subsystem needs to develop data supply methods that meet different rules.

[0069] 1) Direct data transfer is a point-to-point file exchange that may involve two or more physical subsystems. Files are transferred from one physical subsystem to another via a file transfer medium without intermediaries. The data transfer rules for the same file may differ between different physical subsystems. The two directly connected physical subsystems only need to agree on the data transfer batches, filenames, file quantity, file type, file splitting method, data transfer frequency, time points, and data transfer paths. For example... Figure 1 Physical subsystems 3, 4, and 5 may agree on one set of rules, while physical subsystems 4 and 5 may agree on another set of rules.

[0070] 2) The centralized data supply involves many upstream and downstream physical subsystems. Each physical subsystem supplies data to the data warehouse, which then forwards the corresponding files or customizes the data according to the downstream integration requirements, and finally transmits them to the corresponding directory of the downstream physical subsystem server.

[0071] As an intermediary, the data warehouse needs to coordinate many upstream and downstream physical subsystems. Therefore, in order to ensure that different downstream physical subsystems receive the same data from the upstream physical subsystems, there are clear regulations and restrictions on the upstream physical subsystems that provide data and the downstream physical subsystems that need the data files, such as the batch, file name, number of files, file type, file splitting method, frequency, time point, and path of data provision. This ensures that the upstream and downstream physical subsystems and the data warehouse maintain a unified data provision agreement for the same file.

[0072] Therefore, the rules for centralized data supply may differ from those for direct data supply. The centralized data supply rules themselves may also be adjusted and changed due to business adjustments and technological changes. For example, the unified data supply physical subsystem, unified data supply organization, unified data supply system node number, unified data supply file quantity, unified data supply type, unified data supply file name, and unified data supply path.

[0073] This data supply solution mainly includes a data supply system and a receiving system. The data supply system mainly includes two types of configuration files: Configuration File 1 (i.e., the first configuration file) and Configuration File 2 (i.e., the second configuration file). See [link / reference]. Figure 2 As shown.

[0074] 1. Configuration file 1 is at the system level and is mainly used to configure information related to the data supply system and the receiving system. It is universal and mainly includes: system number, security node number, transmission node number, data supply organization, and file path.

[0075] 1) System number: refers to the number of the physical or logical subsystem for data supply and reception. For example, the physical subsystem number of the new generation personal loan system is: A0202.

[0076] 2) Security Node Number: This refers to the security node number corresponding to the data supply system and the receiving system, used to encrypt the transmission channel of the data supply file. There is a one-to-one correspondence between the system number and the security node number. The security node number of all systems generally has a uniform and fixed length limit. The security node number of different systems is different and unique. For example, the security node number of the new generation personal loan system is 6 digits, with a value of 108080, while other systems cannot have 108080.

[0077] 3) Transmission Node Number: This refers to the transmission node number corresponding to the data supply system and the receiving system. Each system has a unique and distinct transmission node number. There is a fixed mapping relationship between the transmission node number and the security node number. Some projects require the security node number to be consistent with the transmission node number. For example, the security node number of the new generation personal loan system is 6 digits, with a value of 108080, and the transmission node number is also 108080. Transmission node numbers for other systems cannot be 108080. Other projects require inconsistencies. If the transmission node number and security node number for the same system are inconsistent, then the transmission node number must also have a uniform and fixed length limit. For example, the security node number of the new generation personal loan system is 6 digits, with a value of 108080, and the transmission node number is G15Z, meaning there is a one-to-one correspondence between the system number, the security node number, and the transmission node number.

[0078] 4) Data supply institution: refers to the first-level branch institution number or head office institution number that needs to supply data.

[0079] 5) File path: The storage path of the data supply system after the data file is unloaded, and the storage path of the receiving system after the file is received.

[0080] 2. Configuration file 2 is at the database table level, mainly involving the database table name and related characteristics corresponding to the data supply file. That is, the parameter values ​​are derived from the characteristics corresponding to the database table name. Adding configuration file 2 allows for the addition of direct data supply or (and) centralized data supply requirements. The main data supply parameters include: organization type, whether to use row-specific parameters, data type, table name, batch, file type, and whether to use centralized data supply. The order can be adjusted according to the data supply requirements.

[0081] 1) Institution type: refers to domestic institutions, overseas institutions, or institutions of the entire bank. For example, domestic institutions are uniformly represented by CN0000000, and institutions of the entire bank are represented by 000000000.

[0082] 2) Whether it is a branch: refers to the head office or a first-level branch. For example, whether it is a branch: 0, that is, take the head office number 000; whether it is a branch: 1, that is, take the corresponding branch number. This scheme prefers to take all branch numbers. For example, if there are 37 branches in total, then take 37 branch numbers.

[0083] 3) Data type: refers to initialization and full data. In daily use, it is divided into full data, incremental data, and deletion data. For example, initialization is INIT, full data is ALL, incremental data is ADD, and deletion data is DEL.

[0084] 4) Table name: refers to the name of the database table corresponding to the data in the data file.

[0085] 5) Batch: This refers to the fact that when a file is too large, it can be divided into the first, second, or third batch of data. Usually, there is only the first batch of data: 0001.

[0086] 6) File types: Data supply systems typically need to provide three types of files: data files (.dat files), data information files (.xml files), and data structures (.ddl files).

[0087] 7) Centralized data supply: This refers to whether centralized data supply is required (i.e., supplying data to the data warehouse) or direct connection data supply is required. Furthermore, centralized and direct connection data supply may have different file naming requirements.

[0088] See Figure 3 The diagram shows the main flowchart of a data processing method provided by an embodiment of the present invention, which includes the following steps:

[0089] S301: In response to reaching the data supply time point to the receiving system, obtain the data supply parameters pre-agreed with the receiving system; wherein, the data supply parameters include table name and whether to supply data centrally;

[0090] S302: Query the data corresponding to the table name from the database, perform a data unloading operation on the data to obtain a data supply file, and generate the file name of the data supply file according to the data supply parameters;

[0091] S303: Determine the data supply method based on the value of the centralized data supply parameter, construct a transmission channel with the receiving system, and use the data supply method to transmit the data supply file to the receiving system through the transmission channel.

[0092] In the above embodiments, for step S301, both the physical subsystem and the data warehouse can act as either a data supplier or a data receiver. Therefore, in this scheme, both the data supplier system and the receiving system can be physical subsystems, such as upstream and downstream physical subsystems, see [link to relevant documentation]. Figure 1 The physical subsystem 11 shown is the data supply system, and the physical subsystem 12 is the receiving system. The two are in an upstream-downstream relationship, or either one can be a data warehouse, such as... Figure 1 The physical subsystem 12 shown is the data supply system, and the data warehouse is the receiving system.

[0093] The data supply system and the receiving system pre-set the data supply time, such as agreeing to supply data at 22:00 every day. However, a tolerance is actually set, such as 2 hours, meaning the data supply system can supply data at any time between 22:00 and 24:00. If the receiving system does not receive the data supply file from the data supply system by 24:00, it will send a message similar to "data supply file not received" to the data supply system, causing the data supply system to check its data supply logic.

[0094] The data supply system and the receiving system pre-agree on data supply parameters, which are configured in... Figure 2 In configuration file 2 shown, the same data supply system may need to supply data to multiple receiving systems. Considering the individual needs of different receiving systems, the data supply parameters need to be set separately. See [link / reference] Figure 1 As shown, physical subsystem 14 directly supplies data to physical subsystem 13 and centrally supplies data to the data warehouse. Therefore, physical subsystem 14 acts as a data supply system, agreeing on data supply parameter A with physical subsystem 13 and pre-determining data supply parameter B with the data warehouse. Both data supply parameter A and data supply parameter B are stored in configuration file 2 of physical subsystem 14.

[0095] For step S302, according to Figure 2 As described, the data supply parameters include institution type, whether it is a branch, data type, table name, batch, file type, and whether it is centralized data supply. For example, based on business development needs, the new generation personal loan system needs to add a centralized data file for domestic institutions, the entire bank, the full volume, customer information table (CUSTINFO), and batch 0001. Then, configure the data supply parameters "Institution type: CN0000000, Branch: 0, Data type: ALL, Table name: CUSTINFO, Batch: 0001, File type: dat, Centralized data supply: Yes" and add the data supply parameters to configuration file 2.

[0096] When querying data to be supplied, the primary reliance is on table names. Table names refer to the database table names corresponding to the data in the supply file. For example, querying the database using the table name "Course Schedule" retrieves the corresponding course schedule data. Similarly, querying the database using the aforementioned "CUSTINFO" retrieves the corresponding data, and then extracting this data from the database yields the supply file.

[0097] In practice, there will be many data supply files. Therefore, to distinguish between different data supply files, it is necessary to name them. For example, the file name of the data supply file can be generated by concatenating the organization type, organization number, data type, table name, batch, and file type in sequence. In order to distinguish the data supply time point, the date element is also included when supplying data, such as 2022-11-03 and 2022-11-04.

[0098] The organization number here depends on the organization mapping relationship, which is configured in configuration file 1. This mapping relationship stores the organization numbers of multiple organizations, including the head office organization number (e.g., 000) and the organization numbers of each branch (e.g., 110, 310). The organization mapping relationship is used to compose the data supply file name. Each data supply file must meet the naming rules, see [link to relevant documentation]. Figure 4 As shown.

[0099] Therefore, when the "whether to split into branches" parameter is 0 (i.e., the first value), the head office organization number 000 is retrieved from the organization mapping relationship, and the organization type, organization number, data type, table name, batch, and file type are concatenated in sequence, resulting in a filename named CN0000000_000_ALL_CUSTINFO_0001.dat. When the "whether to split into branches" parameter is 1 (i.e., the second value), all branch organization numbers are retrieved from the organization mapping relationship. If there are 37 branches, then 37 branch numbers are taken, generating 37 data filenames, such as CN0000000_110_ALL_CUSTINFO_0001.dat, CN0000000_120_ALL_CUSTINFO_0001.dat, ..., CN0000000_480_ALL_CUSTINFO_0001.dat.

[0100] For step S303, the "Centralized Data Supply" parameter in the data supply parameters, if yes, indicates that the receiving system is a data warehouse and a centralized data supply method is required. Figure 1 The physical subsystem 14 feeds data centrally to the data warehouse. If no, it means that centralized data feeding is not required and data is fed directly, such as... Figure 1 The physical subsystem 14 directly supplies data to the physical subsystem 13.

[0101] Configuration file 1 in the data supply system is universal. It stores the system number, transmission node number, and security node number of the data supply system, as well as the system number, transmission node number, and security node number of the associated receiving system. The system number of each system is unique and fixed, and there is a one-to-one correspondence between the system number, transmission node number, and security node number.

[0102] Another function of configuration file 1 is for the transfer of data supply files. To achieve data supply, files need to be transferred from one system to another. Therefore, the source file path and file name, as well as the target file path and file name, need to be determined. When the data supply file is transferred, it needs to be transferred according to the source file path and name, as well as the target file path and name. Configuration file 1 can be used to satisfy the concatenation of the data supply file transfer path to form a complete source file path and target file path, thereby realizing the transfer of data supply files.

[0103] Different data supply systems and receiving systems have different parameter values ​​in configuration file 1. For the same system, the data supply file transmission node number, receiving system system number, transmission node number, and file path may differ in different projects. The rules for these parameters are pre-defined within the same project. To enable the same system to quickly meet the needs of different projects, parameterized configuration is required to improve system reusability, reduce system modification workload, and achieve productization. Therefore, when data supply files are transmitted to the receiving system, they need to be transmitted according to the data supply system transmission node number and the receiving system system system number, transmission node number, and file path specified in configuration file 1.

[0104] The transmission node number depends on the security node number, which is used to encrypt the transmission channel. Therefore, this scheme constructs a dedicated transmission channel for transmitting data supply files based on the transmission node numbers of the data supply system and the data receiving system.

[0105] Taking step S302 as an example, the data supply parameters are configured as follows: "Organization type: CN0000000, whether to split into rows: 0, data type: ALL, table name: CUSTINFO, batch: 0001, file type: dat, whether to centralize data supply: yes". Therefore, the centralized data supply method is adopted, and the data supply file named CN0000000_000_ALL_CUSTINFO_0001.dat is transmitted to the data warehouse through the constructed transmission channel. There is only one data warehouse in this solution.

[0106] The method provided in the above embodiments, in complex system relationships, achieves compatibility with both direct and centralized data supply by adding data supply parameters such as whether to supply data in separate rows and whether to supply data in a centralized manner. Moreover, the data supply parameters can be configured and adjusted according to actual conditions, and have flexible compatibility.

[0107] See Figure 5 The diagram illustrates a specific data processing method according to an embodiment of the present invention, including the following steps:

[0108] S201: In response to reaching the data supply time point to the receiving system, obtain the data supply parameters pre-agreed with the receiving system; wherein, the data supply parameters include organization type, whether to use a line parameter, data type, batch, file type, table name, and whether to use a centralized data supply parameter;

[0109] S502: Query the data corresponding to the table name from the database, and perform a data unloading operation on the data to obtain the data supply file;

[0110] S503: In response to the condition that the branch line parameter is a first value, query the head office organization number from the organization mapping relationship; wherein, the organization mapping relationship is configured in the first configuration file, and the data supply parameter is configured in the second configuration file;

[0111] S504: Concatenate the organization type, head office organization number, data type, table name, batch, and file type in sequence to generate the file name of the data supply file;

[0112] S505: In response to the condition that the branch parameter is a second value, query all branch organization numbers from the organization mapping relationship;

[0113] S506: Concatenate the organization type, each branch organization number, data type, table name, batch, and file type in sequence to generate the filename of each data supply file;

[0114] S507: In response to the condition that the centralized data supply parameter is yes, determine that the data supply method is centralized data supply, and determine that the receiving system is a data warehouse;

[0115] S508: Based on the system number of the data warehouse, query the transmission node number to establish a transmission channel with the transmission node number, and use a centralized data supply method to transmit the data supply file to the data warehouse through the transmission channel;

[0116] S509: In response to the condition that the centralized data supply parameter is no, determine that the data supply method is direct connection data supply, and determine that the receiving system is another physical subsystem;

[0117] S510: Based on the system number of other physical subsystems, query the transmission node number to establish a transmission channel with the transmission node number, and use the direct data supply method to transmit the data supply file to other physical subsystems through the transmission channel.

[0118] The method provided in the above embodiments offers a flexible approach to data supply that is compatible with both direct and centralized data supply in complex system relationships. This allows the data supply system to meet the data supply mechanism that may be adjusted by the data warehouse, without being constrained by changes in data warehouse rules, and without affecting the data supply needs of multiple point-to-point physical subsystems.

[0119] 1. All data supply rules can be parameterized at the system level and database table level, forming configuration file 1 and configuration file 2, thereby adapting to the needs of direct and centralized data supply for different bank-specific businesses and system rules.

[0120] 2. Add a new configuration file 2 to each data supply system, and the data supply parameters in configuration file 2 can be flexibly configured and modified to achieve customized parameter configuration operations;

[0121] 3. The data supply parameters mainly set two parameters: whether to split into lines and whether to supply data centrally. Whether to split into lines determines the file name of the data supply file, and whether to supply data centrally determines the data supply method. These two parameters can meet the needs of both direct data supply and centralized data supply.

[0122] 4. The data supply parameters also include setting the table name, which is done by adding a new data supply file, corresponding to the database table name, and configuring related traits.

[0123] See Figure 6 The diagram shows a schematic of the main modules of a data processing device 600 provided in an embodiment of the present invention, including:

[0124] The acquisition module 601 is used to acquire the data supply parameters pre-agreed with the receiving system in response to the arrival of the data supply time point to the receiving system; wherein, the data supply parameters include table name and whether to supply data centrally;

[0125] The unloading module 602 is used to query data corresponding to the table name from the database, perform unloading operations on the data to obtain a data supply file, and generate a file name of the data supply file according to the data supply parameters.

[0126] The data supply module 603 is used to determine the data supply method based on the value of the centralized data supply parameter, construct a transmission channel with the receiving system, and use the data supply method to transmit the data supply file to the receiving system through the transmission channel.

[0127] In the apparatus of this invention, the data supply parameters further include: organization type, whether to split into rows, data type, batch, and file type;

[0128] The unloading module 602 is used for:

[0129] Query the organization number corresponding to the value of the "whether to split into rows" parameter from the organization mapping relationship;

[0130] The filename of the data supply file is generated by sequentially concatenating the organization type, organization number, data type, table name, batch, and file type.

[0131] In the apparatus of this invention, the unloading module 602 is used for:

[0132] Obtain the current date, and generate the filename of the data supply file based on the data supply parameters and the current date.

[0133] In the device of this invention, the mechanism mapping relationship is configured in a first configuration file, and the data supply parameters are configured in a second configuration file;

[0134] The unloading module 602 is used for:

[0135] In response to the condition that the branch line parameter is a first value, the head office organization number is queried from the organization mapping relationship;

[0136] In response to the condition that the branching parameter is a second value, all branching organization numbers are queried from the organization mapping relationship.

[0137] In the apparatus of this invention, the data supply module 603 is used for:

[0138] In response to the "whether to provide centralized data" parameter being "yes", the data supply method is determined to be centralized data supply, and the receiving system is determined to be a data warehouse; or

[0139] In response to the condition that the centralized data supply parameter is no, the data supply method is determined to be direct connection data supply, and the receiving system is determined to be another physical subsystem.

[0140] In the apparatus of this invention, the data supply module 603 is used for:

[0141] Based on the system number of the receiving system, query the transmission node number of the receiving system to establish a transmission channel with the transmission node number.

[0142] In the apparatus of the present invention, the data supply module 603 is further configured to: query the security node number of the receiving system, wherein the transmission node number depends on the security node number and the security node number is used to encrypt the transmission channel.

[0143] Furthermore, the specific implementation details of the device described in the embodiments of the present invention have been described in detail in the above-described method, so the details will not be repeated here.

[0144] Figure 7 An exemplary system architecture 700 to which embodiments of the present invention can be applied is shown, including terminal devices 701, 702, 703, network 704, and server 705 (this is merely an example).

[0145] Terminal devices 701, 702, and 703 can be various electronic devices with displays and support for web browsing, and have various communication client applications installed. Users can use terminal devices 701, 702, and 703 to interact with server 705 through network 704 to receive or send messages, etc.

[0146] Network 704 is a medium used to provide a communication link between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0147] Server 705 can be a server that provides various services. It should be noted that the methods provided in the embodiments of the present invention are generally executed by server 705, and correspondingly, the devices are generally set in server 705.

[0148] It should be understood that Figure 7 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0149] The following is for reference. Figure 8 It shows a schematic diagram of the structure of a computer system 800 suitable for implementing a terminal device of the present invention. Figure 8 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0150] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the system 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0151] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0152] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs the functions defined above in the system of this invention.

[0153] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0155] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including an acquisition module, a data unloading module, and a data supply module. The names of these modules do not necessarily limit the module itself; for example, the data supply module can also be described as a "channel data supply module."

[0156] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to perform any of the data processing methods described above.

[0157] The computer program product of the present invention includes a computer program that, when executed by a processor, implements the data processing method in the embodiments of the present invention.

[0158] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data supply processing method, characterized in that, include: In response to reaching the data supply time point to the receiving system, the data supply parameters agreed upon with the receiving system in advance are obtained; wherein, the data supply parameters include table name and whether centralized data supply is used; Query the database for the data corresponding to the table name, and perform a data unloading operation on the data to obtain the data supply file; Generating the filename of the data supply file based on the data supply parameters includes: in response to reaching the data supply time point to the receiving system, obtaining the data supply parameters pre-agreed with the receiving system; wherein, the data supply parameters include organization type, whether it is a branch, data type, batch, file type, table name, and whether it is a centralized data supply parameter; querying data corresponding to the table name from the database, and performing a data unloading operation on the data to obtain the data supply file; in response to the whether it is a branch being a first value, querying the head office organization number from the organization mapping relationship; wherein, the organization mapping relationship is configured in a first configuration file, and the data supply parameters are configured in a second configuration file; concatenating the organization type, head office organization number, data type, table name, batch, and file type in sequence to generate the filename of the data supply file; in response to the whether it is a branch being a second value, querying all branch organization numbers from the organization mapping relationship; concatenating the organization type, each branch organization number, data type, table name, batch, and file type in sequence to generate the filename of each data supply file; The data supply method is determined based on the value of the centralized data supply parameter, and a transmission channel with the receiving system is constructed, including: in response to the centralized data supply parameter being yes, determining the data supply method as centralized data supply and determining the receiving system as a data warehouse; or in response to the centralized data supply parameter being no, determining the data supply method as direct data supply and determining the receiving system as another physical subsystem; and querying the transmission node number of the receiving system based on the system number of the receiving system to establish a transmission channel with the transmission node number. The data supply file is transmitted to the receiving system via the transmission channel using the aforementioned data supply method.

2. The method according to claim 1, characterized in that, The step of generating the filename of the data supply file based on the data supply parameters includes: Query the organization number corresponding to the value of the "whether to split into rows" parameter from the organization mapping relationship; The filename of the data supply file is generated by sequentially concatenating the organization type, organization number, data type, table name, batch, and file type.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Obtain the current date, and generate the filename of the data supply file based on the data supply parameters and the current date.

4. The method according to claim 1, characterized in that, The method further includes: querying the security node number of the receiving system, wherein the transmission node number depends on the security node number, and the security node number is used to encrypt the transmission channel.

5. A data processing device, characterized in that, include: The acquisition module is used to acquire the data supply parameters pre-agreed with the receiving system in response to the arrival of the data supply time point to the receiving system; wherein, the data supply parameters include table name and whether centralized data supply is used. The data unloading module is used to query data corresponding to the table name from the database, perform data unloading operations on the data to obtain a data supply file, and generate the file name of the data supply file according to the data supply parameters. This includes: in response to reaching the data supply time point to the receiving system, obtaining the data supply parameters pre-agreed with the receiving system; wherein the data supply parameters include organization type, whether it is a branch parameter, data type, batch, file type, table name, and whether it is a centralized data supply parameter; querying data corresponding to the table name from the database, and performing data unloading operations on the data to obtain a data supply file; in response to the whether it is a branch parameter being a first value, querying the head office organization number from the organization mapping relationship; wherein the organization mapping relationship is configured in a first configuration file, and the data supply parameters are configured in a second configuration file; sequentially concatenating the organization type, head office organization number, data type, table name, batch, and file type to generate the file name of the data supply file; in response to the whether it is a branch parameter being a second value, querying all branch organization numbers from the organization mapping relationship; sequentially concatenating the organization type, each branch organization number, data type, table name, batch, and file type to generate the file name of each data supply file. The data supply module is used to determine the data supply method based on the value of the centralized data supply parameter, and to construct a transmission channel with the receiving system. This includes: in response to the centralized data supply parameter being "yes", determining the data supply method as centralized data supply and identifying the receiving system as a data warehouse; or in response to the centralized data supply parameter being "no", determining the data supply method as direct data supply and identifying the receiving system as another physical subsystem; querying the transmission node number of the receiving system based on its system number to establish a transmission channel with that transmission node number; and transmitting the data supply file to the receiving system through the transmission channel using the data supply method.

6. The apparatus according to claim 5, characterized in that, The unloading module is used for: Query the organization number corresponding to the value of the "whether to split into rows" parameter from the organization mapping relationship; The filename of the data supply file is generated by sequentially concatenating the organization type, organization number, data type, table name, batch, and file type.

7. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-4.

8. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-4.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-4.

Images