Data conversion method, device, computer equipment and storage medium

By using concurrent thread pools and preset sharding rules in the data conversion method of financial markets, the problem of inefficient processing and poor scalability of data file conversion schemes is solved, and efficient data conversion and system scalability is achieved. It is suitable for Complex transaction system data interaction.

CN118708642BActive Publication Date: 2024-12-06GUOTAI JUNAN SECURITIES CO LTD +1
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411204301.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-12-06
Estimated Expiration
2044-08-30

AI Technical Summary

Technical Problem

The existing data file conversion schemes are inefficient in processing and are not very scalable, especially when the data interaction between the peripheral operation and maintenance operation systems and the trading systems in the financial market is complicated, resulting in complex loading and poor scalability of the trading system.

Method used

By using a concurrent thread pool in the data conversion method, the configuration name is allocated to the concurrent thread pool according to the task concurrent processing instructions, and the source data is processed according to the preset sharding rules and the conversion configuration, generating the converted data and writing to a new file , realize concurrent conversion processing, improve data conversion efficiency, and realize data changes by modifying the data model to enhance scalability.

Benefits of technology

It improves the efficiency of data conversion and the scalability of the system, making the data processing of the transaction system more efficient and flexible, and can cope with complex data interaction needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118708642B_ABST
    Figure CN118708642B_ABST
Patent Text Reader

Abstract

The present application provides a data conversion method, device, computer equipment and storage medium, which call the concurrent threads corresponding to each configuration name in the concurrent thread pool, obtain the first empty file corresponding to each configuration name according to each concurrent thread, find the preset conversion configuration corresponding to each configuration name and read the first source data corresponding to each configuration name; process the first source data corresponding to each configuration name according to the preset sharding rule and the corresponding preset conversion configuration to obtain the first conversion data corresponding to each first source data; write the first conversion data corresponding to each first source data into the corresponding first empty file, and generate the first conversion file corresponding to each first source data. Thus, concurrent conversion processing is performed on each configuration name in the target configuration name list, improving the data conversion efficiency; and the change of the first source data can be realized by modifying the data model, making the scalability stronger.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data conversion method, device, computer equipment and storage medium. Background Art

[0002] With the rapid development of the financial market, the data interaction between the peripheral operation and maintenance system and the trading system has become increasingly complex. Among them, the trading system relies on a wide variety of data, including account data, parameter data, transaction data, business data, etc., and the peripheral operation and maintenance system data is generally stored in traditional relational databases, which makes it very complicated to load the trading system directly, and once the peripheral operation and maintenance system is changed, it will also affect the trading system.

[0003] At present, traditional data file conversion solutions mostly use hard coding to implement conversion logic, read files item by item, and parse file contents. In which, data is processed and converted line by line, and appended to the new format file until the data processing is completed. It can be seen that the processing efficiency of this traditional data file conversion solution is low and the scalability is not strong. Summary of the invention

[0004] The embodiments of the present application provide a data conversion method, apparatus, computer device and storage medium, aiming to solve the problems of low processing efficiency and poor scalability of existing data file conversion solutions.

[0005] In a first aspect, an embodiment of the present application provides a data conversion method, which is applied to a securities business system, and the method includes:

[0006] When receiving a configuration name list read instruction, read the target configuration name list from the data model;

[0007] If a task concurrent processing instruction is detected, each configuration name in the target configuration name list is assigned to a concurrent thread pool according to the task concurrent processing instruction;

[0008] Calling the concurrent threads corresponding to each configuration name in the concurrent thread pool, and obtaining the first empty file corresponding to each configuration name according to each concurrent thread, searching for the preset conversion configuration corresponding to each configuration name, and reading the first source data corresponding to each configuration name;

[0009] According to each concurrent thread, the first source data corresponding to each configuration name is processed according to a preset sharding rule and a preset conversion configuration corresponding to each configuration name to obtain first conversion data corresponding to each first source data;

[0010] According to each concurrent thread, the first conversion data corresponding to each first source data is written into the corresponding first empty file, so as to generate a first conversion file corresponding to each first source data.

[0011] In a second aspect, an embodiment of the present application provides a data conversion device, the device comprising:

[0012] A reading unit, configured to read a target configuration name list from a data model when receiving a configuration name list reading instruction;

[0013] an allocating unit, configured to allocate each configuration name in the target configuration name list to a concurrent thread pool according to the task concurrent processing instruction if a task concurrent processing instruction is detected;

[0014] A calling unit, configured to call the concurrent threads corresponding to each configuration name in the concurrent thread pool, and obtain the first empty file corresponding to each configuration name according to each concurrent thread, search for the preset conversion configuration corresponding to each configuration name, and read the first source data corresponding to each configuration name;

[0015] A data processing unit, configured to process the first source data corresponding to each configuration name according to the preset sharding rule and the preset conversion configuration corresponding to each configuration name according to each concurrent thread, to obtain first conversion data corresponding to each first source data;

[0016] The generating unit is used for writing the first conversion data corresponding to each first source data into the corresponding first empty file according to each concurrent thread, so as to generate the first conversion file corresponding to each first source data.

[0017] In a third aspect, an embodiment of the present application further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data conversion method described in the first aspect when executing the computer program.

[0018] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the data conversion method described in the first aspect above.

[0019] The embodiment of the present application provides a data conversion method, device, computer equipment and storage medium, by calling the concurrent thread corresponding to each configuration name in the concurrent thread pool, obtaining the first empty file corresponding to each configuration name according to each concurrent thread, searching for the preset conversion configuration corresponding to each configuration name and reading the first source data corresponding to each configuration name; processing the first source data corresponding to each configuration name according to the preset sharding rule and the corresponding preset conversion configuration, obtaining the first conversion data corresponding to each first source data; writing the first conversion data corresponding to each first source data into the corresponding first empty file, generating the first conversion file corresponding to each first source data. Thus, concurrent conversion processing is performed on each configuration name in the target configuration name list, improving the data conversion efficiency; and the change of the first source data can be realized by modifying the data model, making the scalability stronger. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A flowchart of a data conversion method provided in an embodiment of the present application;

[0022] Figure 2 A schematic block diagram of a data conversion device provided in an embodiment of the present application;

[0023] Figure 3 A schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0026] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0027] It should be further understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0028] See also Figure 1 , Figure 1 A flow chart of a data conversion method provided in an embodiment of the present application, wherein the data conversion method is applied in a securities business system.

[0029] like Figure 1 As shown, the method includes steps S101 to S105.

[0030] S101. When a configuration name list read instruction is received, a target configuration name list is read from a data model.

[0031] In this embodiment, the configuration name list reading instruction can be triggered by relevant staff; when the transaction system receives the configuration name list reading instruction triggered by the relevant staff, the target configuration name list is read from the data model. The data model is an abstraction of data features, which describes the static features, dynamic behaviors and constraints of the system at an abstract level, and provides an abstract framework for information representation and operation of the database system; the target configuration name list includes multiple configuration names, each of which corresponds to a file name; thus, by obtaining the target configuration name list, it is convenient to subsequently obtain the first empty file corresponding to each configuration name, find the preset conversion configuration corresponding to each configuration name, and read the first source data corresponding to each configuration name.

[0032] In one embodiment, reading the target configuration name list from the data model includes:

[0033] Reading a first list of configuration names from the data model;

[0034] Filtering the first configuration name list to obtain a second configuration name list;

[0035] Each second configuration name in the second configuration name list is sorted according to a preset priority to obtain the target configuration name list.

[0036] In this embodiment, when a configuration name list reading instruction is received, a first configuration name list consisting of first configuration names is first read from the data model; and a preprocessing operation of screening the first configuration names in the first configuration name list is performed according to user requirements to remove redundant first configuration names, thereby obtaining a second configuration name list that meets user requirements. In addition, in order to facilitate the subsequent execution of serial threads, serial tasks can be executed in sequence based on each configuration name, and the transaction system will also sort each second configuration name in the second configuration name list according to a preset priority to obtain a sorted target configuration name list; wherein the preset priority includes the urgency and importance of the task; therefore, this embodiment uses a sorted target configuration name list to facilitate subsequent task scheduling.

[0037] In one embodiment, the securities business system further includes a core trading system and a database, and before reading the target configuration name list from the data model, it also includes:

[0038] Responsive to the configuration building instruction, acquiring business data of the core transaction system;

[0039] Reading a target database list from the database according to the business data;

[0040] A preset conversion configuration set is generated according to the target database list; wherein the preset conversion configuration set includes at least one preset conversion configuration; each preset conversion configuration includes a file category, a file name, a file format and a conversion query logic.

[0041] In this embodiment, the securities business system includes an operation and maintenance system, a core trading system and a database. Since the core trading system relies on complex data, such as account data, parameter data, transaction data and business data, and the operation and maintenance system is maintained by a relational database, it will be very complicated to load the core trading system directly, and once the operation and maintenance system is changed, the core trading system will be affected. Therefore, in order to decouple, before reading the target configuration name list from the data model, a preset conversion configuration set of the data will be defined first; specifically, first respond to the configuration construction instruction, obtain the business data of the core trading system; wherein the configuration construction instruction can be triggered by the business personnel; then read the target database list from the database according to the business data supported by the core trading system; in addition, generate a preset conversion configuration set according to the read target database list. Among them, the preset conversion configuration set includes at least one of the preset conversion configurations; wherein each preset conversion configuration includes configurations such as file category, file name, file format and conversion query logic to ensure the accuracy and consistency of subsequent data conversion; and the file category includes category files such as account class and transaction class; the file format can be .bin, .xml, .Dbf, etc.; the conversion query logic includes the file field name to be written and the relevant attributes of the field; wherein the relevant attributes of the field include database query field name, field type (for example, the securities code is a string, the securities market is a number, etc.) and field precision, etc.; in addition, for the conversion query logic, the relevant conversion configuration of the field can also be defined in json format to facilitate subsequent data transmission and interaction.

[0042] In one embodiment, generating a preset conversion configuration set according to the target database list includes:

[0043] Analyze the table data content format and table design field meanings in the target database list to obtain a conversion rule document;

[0044] The conversion rule document is determined as the preset conversion configuration set.

[0045] In this embodiment, generating a preset conversion configuration set according to the target database list specifically includes: analyzing and mapping the table data content format and the meaning of the table design fields in the target database list to obtain the meaning and conversion rule document of each field in the target database list; wherein, the table data content format can be character type, numeric type, etc. And the rule configuration is performed based on the conversion rule document to determine the preset conversion configuration set, and then the preset conversion configuration set is used to facilitate subsequent field conversion. In addition, after obtaining the preset conversion configuration set, when the operation and maintenance system is started, the preset conversion configuration set is loaded and parsed, and the preset conversion configuration set is parsed into multiple conversion objects according to the file dimension, each object contains the file name, query logic, write file field name and optional configuration (such as batch query), and these objects are cached in the application memory in the form of a list array.

[0046] In one embodiment, after reading the target configuration name list from the data model, the method further includes:

[0047] If a task serial processing instruction is detected, a serial thread corresponding to the target configuration name list is generated; wherein the serial thread executes serial tasks in sequence based on each configuration name;

[0048] For each serial task corresponding to a configuration name, the serial thread is called, and according to the current serial task of the serial thread, a current configuration name and a target empty file corresponding to the current configuration name are obtained, a target preset conversion configuration corresponding to the current configuration name is searched, and target source data corresponding to the current configuration name is read;

[0049] According to the current serial task of the serial thread, the target source data is processed according to the preset slicing rule and the target preset conversion configuration to obtain target conversion data;

[0050] According to the current serial task of the serial thread, the target conversion data is written into the target empty file to obtain a target conversion file.

[0051] In this embodiment, while realizing the concurrent processing conversion of the source data, the serial processing conversion of the source data can also be realized, so that the user can set it according to his own needs. Among them, the serial processing conversion process for the source data is specifically as follows: when the securities trading system detects a task serial processing instruction, a serial thread corresponding to the target configuration name list is generated; wherein the serial thread executes serial tasks in sequence based on each configuration name in the target configuration name list; and for the serial task corresponding to each configuration name, by calling the serial thread, and obtaining the current configuration name and the target empty file corresponding to the current configuration name according to the current serial task of the serial thread, searching for the target preset conversion configuration corresponding to the current configuration name, and reading the target source data corresponding to the current configuration name; wherein the target source data is transmitted from the operation and maintenance system to the core trading system, and cached in the memory of the core trading system, so as to realize the rapid access to the target source data later. And according to the current serial task of the serial thread, the target source data is processed according to the preset sharding rules and the target preset conversion configuration to obtain the target conversion data; in addition, according to the current serial task of the serial thread, the target conversion data is written into the target empty file, so as to obtain the target conversion file to complete the serial tasks executed sequentially in the serial thread. In addition, after obtaining the target conversion file, in order to confirm the state of the serial thread, the conversion status of each target source data is detected; wherein, if it is detected that each target source data has generated a corresponding target conversion file, the serial thread is stopped from being called.

[0052] S102: If a concurrent task processing instruction is detected, each configuration name in the target configuration name list is assigned to a concurrent thread pool according to the concurrent task processing instruction.

[0053] In this embodiment, when a task concurrent processing instruction is detected, the conversion logic is concurrently executed according to the task concurrent processing instruction, specifically by assigning each configuration name in the target configuration name list to a concurrent thread pool; wherein the concurrent thread pool includes multiple concurrent threads; and each concurrent thread corresponds one-to-one to each configuration name.

[0054] S103, calling the concurrent threads corresponding to each configuration name in the concurrent thread pool, and obtaining the first empty file corresponding to each configuration name according to each concurrent thread, searching for the preset conversion configuration corresponding to each configuration name, and reading the first source data corresponding to each configuration name.

[0055] In this embodiment, in order to realize concurrent conversion, firstly, by calling the concurrent thread corresponding to each configuration name in the concurrent thread pool, and according to the tasks in each concurrent thread, executing the job pre-expansion method, a first empty file corresponding to each configuration name is generated, so that the converted source data can be written into the first empty file later to realize the data conversion operation. And, according to the tasks in each concurrent thread, the preset conversion configuration corresponding to each configuration name is searched, so as to realize the field conversion of the source data through the preset conversion configuration later; wherein, when there is a batch configuration, the corresponding query conditions are supplemented, for example, the brokerage customer data is basically distributed according to the business department dimension data, then parallel query can be performed according to the business department dimension. And according to the tasks in each concurrent thread, the first source data corresponding to each configuration name is read and searched; wherein, the first source data corresponding to each configuration name is the source data to be converted.

[0056] S104. According to each concurrent thread, the first source data corresponding to each configuration name is processed according to a preset sharding rule and a preset conversion configuration corresponding to each configuration name to obtain first conversion data corresponding to each first source data.

[0057] In this embodiment, in order to realize concurrent conversion of the first source data corresponding to each configuration name, this embodiment processes the first source data corresponding to each configuration name according to each concurrent thread, according to the preset sharding rule and the preset conversion configuration corresponding to each configuration name, so as to obtain the first conversion data corresponding to each first source data.

[0058] Specifically, the first source data corresponding to each configuration name is processed according to the preset sharding rule and the preset conversion configuration corresponding to each configuration name to obtain the first conversion data corresponding to each first source data, including:

[0059] Slice the first source data corresponding to each configuration name based on the preset slicing rule to obtain at least one piece of sub-data corresponding to each first source data;

[0060] Wherein, the first source data corresponding to each configuration name includes the total amount of source data;

[0061] For example, if the total amount of data is 10,000 and the amount of data in each slice in the preset sharding rule is 2,000, then 5 slices of data are obtained, and the amount of data in each slice is 2,000.

[0062] For at least one piece of sub-data corresponding to each first source data, field conversion is performed on each piece of sub-data in the at least one piece of sub-data according to the preset conversion configuration corresponding to each configuration name to obtain first conversion data corresponding to each first source data; wherein the first conversion data contains conversion sub-data corresponding to each piece of sub-data.

[0063] More specifically, for each piece of sub-data in the at least one piece of sub-data, each piece of data in each piece of sub-data is traversed to perform field conversion, thereby obtaining conversion sub-data composed of each piece of data in each piece of sub-data, and then combining the conversion sub-data corresponding to each piece of sub-data to generate the first conversion data.

[0064] For example, taking the cash_fund.xml file as an example, for the first piece of sub-data (2000 entries) in the at least one piece of sub-data, one piece of data in the first piece of sub-data is:

[0065] fund_id=001,branch_id=1001,partition=1,balance=2000.22000;

[0066] After the fields are converted according to the preset conversion configuration corresponding to each configuration name, the conversion results are:

[0067] <cash_fund>

[0068] <row>

[0069] <fundid> 001< / fundid>

[0070] <partition> 1001< / partition>

[0071] <branchid> 1< / branchid>

[0072] <fundbalance> 2000.2200< / fundbalance>

[0073] <row>

[0074] < / cash_fund>

[0075] FundID:{column:fund id,type:string,length:1,precision:0,order:0},

[0076] Among them, FundID rule: take the fund_id value in the result, string, length 1, precision 0, sort 0;

[0077] Partition:{column:partition,type:String,length:1,precision:0,order:0},

[0078] Among them, the Partition rule is: take the partition value in the result, a string with a length of 1, a precision of 0, and a sorting of 0;

[0079] BranchID:{column:branch_id,type:String,length:1,precision:0,order:0},

[0080] Among them, the BranchID rule is: take the branch_id value in the result, a string with a length of 1, a precision of 0, and a sorting of 0;

[0081] FundBalance: {column:balance,type:BigDecimal,length:1,precision:4,order:0}

[0082] Among them, FundBalance rule: take the balance value in the result, BigDecimal, length 1, precision 4 decimal places, sort 0; BigDecimal: Java provides the API class BigDecimal in the java.math package, which is used to perform precise calculations on numbers with more than 16 significant digits.

[0083] Therefore, the above embodiment performs sharding processing on the first source data corresponding to each configuration name, so as to disperse each piece of sub-data in at least one piece of sub-data corresponding to each first source data to multiple nodes for processing, thereby improving the performance and throughput of the securities business system while enhancing the scalability of the securities business system.

[0084] In one embodiment, the method further comprises:

[0085] The batch processing framework is used to concurrently perform field transformation on each piece of data in each piece of sub-data.

[0086] In this embodiment, in order to further improve the performance of the securities business system, the method also uses a batch processing framework to concurrently perform field conversion on each piece of data in each piece of sub-data; wherein the batch processing framework can be a springbatch framework.

[0087] S105 . According to each concurrent thread, write the first conversion data corresponding to each first source data into the corresponding first empty file to generate a first conversion file corresponding to each first source data.

[0088] In this embodiment, after obtaining the first conversion data corresponding to each first source data, in order to further realize the file conversion of each first source data, the first conversion data corresponding to each first source data is written into the corresponding first empty file according to each concurrent thread to generate the first conversion file corresponding to each first source data. Wherein, the first conversion data corresponding to each first source data is written into the corresponding first empty file can be specifically executed by writers such as xmlWriter and BindWirter; in addition, after generating the first conversion file corresponding to each first source data, in order to confirm the status of each parallel thread, the conversion status of each first source data is detected; wherein, if it is detected that each first source data has generated the corresponding first target conversion file, then the calling of each concurrent thread is stopped. In addition, the securities business system also provides an http interface to convert the first conversion file corresponding to each first source data into a visual file for the securities dealer to view; wherein, according to the difference in usage scenarios, the generation time of the first conversion file corresponding to each first source data into a visual file is different. For example, in the securities industry, some exchange files arrive at the brokerage firm first, while others arrive at the brokerage firm later. Therefore, the http interface supports batch conversion as well as conversion of a single file.

[0089] It can be seen from the above technical solutions that the present invention realizes concurrent conversion processing of each configuration name in the target configuration name list, thereby improving data conversion efficiency; and the first source data can be changed by modifying the data model, making it more scalable.

[0090] The present application also provides a data conversion device, which is used to execute any embodiment of the aforementioned data conversion method. Figure 2 , Figure 2 It is a schematic block diagram of a data conversion device provided in an embodiment of the present application.

[0091] Among them, Figure 2 As shown, the data conversion device 100 includes a reading unit 101 , an allocating unit 102 , a calling unit 103 , a data processing unit 104 and a generating unit 105 .

[0092] The reading unit 101 is used to read the target configuration name list from the data model when receiving a configuration name list reading instruction.

[0093] In this embodiment, the configuration name list reading instruction can be triggered by relevant staff; when the transaction system receives the configuration name list reading instruction triggered by the relevant staff, the target configuration name list is read from the data model. The data model is an abstraction of data features, which describes the static features, dynamic behaviors and constraints of the system at an abstract level, and provides an abstract framework for information representation and operation of the database system; the target configuration name list includes multiple configuration names, each of which corresponds to a file name; thus, by obtaining the target configuration name list, it is convenient to subsequently obtain the first empty file corresponding to each configuration name, find the preset conversion configuration corresponding to each configuration name, and read the first source data corresponding to each configuration name.

[0094] In one embodiment, reading the target configuration name list from the data model includes:

[0095] Reading a first list of configuration names from the data model;

[0096] Filtering the first configuration name list to obtain a second configuration name list;

[0097] Each second configuration name in the second configuration name list is sorted according to a preset priority to obtain the target configuration name list.

[0098] In this embodiment, when a configuration name list reading instruction is received, a first configuration name list consisting of first configuration names is first read from the data model; and a preprocessing operation of screening the first configuration names in the first configuration name list is performed according to user requirements to remove redundant first configuration names, thereby obtaining a second configuration name list that meets user requirements. In addition, in order to facilitate the subsequent execution of serial threads, serial tasks can be executed in sequence based on each configuration name, and the transaction system will also sort each second configuration name in the second configuration name list according to a preset priority to obtain a sorted target configuration name list; wherein the preset priority includes the urgency and importance of the task; therefore, this embodiment uses a sorted target configuration name list to facilitate subsequent task scheduling.

[0099] In one embodiment, the securities business system further includes a core trading system and a database, and before reading the target configuration name list from the data model, it also includes:

[0100] Responsive to the configuration building instruction, acquiring business data of the core transaction system;

[0101] Reading a target database list from the database according to the business data;

[0102] A preset conversion configuration set is generated according to the target database list; wherein the preset conversion configuration set includes at least one preset conversion configuration; each preset conversion configuration includes a file category, a file name, a file format and a conversion query logic.

[0103] In this embodiment, the securities business system includes an operation and maintenance system, a core trading system and a database. Since the core trading system relies on complex data, such as account data, parameter data, transaction data and business data, and the operation and maintenance system is maintained by a relational database, it will be very complicated to load the core trading system directly, and once the operation and maintenance system is changed, the core trading system will be affected. Therefore, in order to decouple, before reading the target configuration name list from the data model, a preset conversion configuration set of the data will be defined first; specifically, first respond to the configuration construction instruction, obtain the business data of the core trading system; wherein the configuration construction instruction can be triggered by the business personnel; then read the target database list from the database according to the business data supported by the core trading system; in addition, generate a preset conversion configuration set according to the read target database list. Among them, the preset conversion configuration set includes at least one of the preset conversion configurations; wherein each preset conversion configuration includes configurations such as file category, file name, file format and conversion query logic to ensure the accuracy and consistency of subsequent data conversion; and the file category includes category files such as account class and transaction class; the file format can be .bin, .xml, .Dbf, etc.; the conversion query logic includes the file field name to be written and the relevant attributes of the field; wherein the relevant attributes of the field include database query field name, field type (for example, the securities code is a string, the securities market is a number, etc.) and field precision, etc.; in addition, for the conversion query logic, the relevant conversion configuration of the field can also be defined in json format to facilitate subsequent data transmission and interaction.

[0104] In one embodiment, generating a preset conversion configuration set according to the target database list includes:

[0105] Analyze the table data content format and table design field meanings in the target database list to obtain a conversion rule document;

[0106] The conversion rule document is determined as the preset conversion configuration set.

[0107] In this embodiment, generating a preset conversion configuration set according to the target database list specifically includes: analyzing and mapping the table data content format and the meaning of the table design fields in the target database list to obtain the meaning and conversion rule document of each field in the target database list; wherein, the table data content format can be character type, numeric type, etc. And the rule configuration is performed based on the conversion rule document to determine the preset conversion configuration set, and then the preset conversion configuration set is used to facilitate subsequent field conversion. In addition, after obtaining the preset conversion configuration set, when the operation and maintenance system is started, the preset conversion configuration set is loaded and parsed, and the preset conversion configuration set is parsed into multiple conversion objects according to the file dimension, each object contains the file name, query logic, write file field name and optional configuration (such as batch query), and these objects are cached in the application memory in the form of a list array.

[0108] In one embodiment, after reading the target configuration name list from the data model, the method further includes:

[0109] If a task serial processing instruction is detected, a serial thread corresponding to the target configuration name list is generated; wherein the serial thread executes serial tasks in sequence based on each configuration name;

[0110] For each serial task corresponding to a configuration name, the serial thread is called, and according to the current serial task of the serial thread, a current configuration name and a target empty file corresponding to the current configuration name are obtained, a target preset conversion configuration corresponding to the current configuration name is searched, and target source data corresponding to the current configuration name is read;

[0111] According to the current serial task of the serial thread, the target source data is processed according to the preset slicing rule and the target preset conversion configuration to obtain target conversion data;

[0112] According to the current serial task of the serial thread, the target conversion data is written into the target empty file to obtain a target conversion file.

[0113] In this embodiment, while realizing the concurrent processing conversion of the source data, the serial processing conversion of the source data can also be realized, so that the user can set it according to his own needs. Among them, the serial processing conversion process for the source data is specifically as follows: when the securities trading system detects a task serial processing instruction, a serial thread corresponding to the target configuration name list is generated; wherein the serial thread executes serial tasks in sequence based on each configuration name in the target configuration name list; and for the serial task corresponding to each configuration name, by calling the serial thread, and obtaining the current configuration name and the target empty file corresponding to the current configuration name according to the current serial task of the serial thread, searching for the target preset conversion configuration corresponding to the current configuration name, and reading the target source data corresponding to the current configuration name; wherein the target source data is transmitted from the operation and maintenance system to the core trading system, and cached in the memory of the core trading system, so as to realize the rapid access to the target source data later. And according to the current serial task of the serial thread, the target source data is processed according to the preset sharding rules and the target preset conversion configuration to obtain the target conversion data; in addition, according to the current serial task of the serial thread, the target conversion data is written into the target empty file, so as to obtain the target conversion file to complete the serial tasks executed sequentially in the serial thread. In addition, after obtaining the target conversion file, in order to confirm the state of the serial thread, the conversion status of each target source data is detected; wherein, if it is detected that each target source data has generated a corresponding target conversion file, the serial thread is stopped from being called.

[0114] The allocating unit 102 is configured to allocate each configuration name in the target configuration name list to a concurrent thread pool according to the task concurrent processing instruction if a task concurrent processing instruction is detected.

[0115] In this embodiment, when a task concurrent processing instruction is detected, the conversion logic is concurrently executed according to the task concurrent processing instruction, specifically by assigning each configuration name in the target configuration name list to a concurrent thread pool; wherein the concurrent thread pool includes multiple concurrent threads; and each concurrent thread corresponds one-to-one to each configuration name.

[0116] The calling unit 103 is used to call the concurrent threads corresponding to each configuration name in the concurrent thread pool, and obtain the first empty file corresponding to each configuration name according to each concurrent thread, search for the preset conversion configuration corresponding to each configuration name, and read the first source data corresponding to each configuration name.

[0117] In this embodiment, in order to realize concurrent conversion, firstly, by calling the concurrent thread corresponding to each configuration name in the concurrent thread pool, and according to the tasks in each concurrent thread, executing the job pre-expansion method, a first empty file corresponding to each configuration name is generated, so that the converted source data can be written into the first empty file later to realize the data conversion operation. And, according to the tasks in each concurrent thread, the preset conversion configuration corresponding to each configuration name is searched, so as to realize the field conversion of the source data through the preset conversion configuration later; wherein, when there is a batch configuration, the corresponding query conditions are supplemented, for example, the brokerage customer data is basically distributed according to the business department dimension data, then parallel query can be performed according to the business department dimension. And according to the tasks in each concurrent thread, the first source data corresponding to each configuration name is read and searched; wherein, the first source data corresponding to each configuration name is the source data to be converted.

[0118] The data processing unit 104 is used to process the first source data corresponding to each configuration name according to each concurrent thread, according to the preset sharding rule and the preset conversion configuration corresponding to each configuration name, to obtain the first conversion data corresponding to each first source data.

[0119] In this embodiment, in order to realize concurrent conversion of the first source data corresponding to each configuration name, this embodiment processes the first source data corresponding to each configuration name according to each concurrent thread, according to the preset sharding rule and the preset conversion configuration corresponding to each configuration name, so as to obtain the first conversion data corresponding to each first source data.

[0120] Specifically, the data processing unit 104 is specifically used for:

[0121] Slice the first source data corresponding to each configuration name based on the preset slicing rule to obtain at least one piece of sub-data corresponding to each first source data;

[0122] Wherein, the first source data corresponding to each configuration name includes the total amount of source data;

[0123] For example, if the total amount of data is 10,000 and the amount of data in each slice in the preset sharding rule is 2,000, then 5 slices of data are obtained, and the amount of data in each slice is 2,000.

[0124] For at least one piece of sub-data corresponding to each first source data, field conversion is performed on each piece of sub-data in the at least one piece of sub-data according to the preset conversion configuration corresponding to each configuration name to obtain first conversion data corresponding to each first source data; wherein the first conversion data contains conversion sub-data corresponding to each piece of sub-data.

[0125] More specifically, for each piece of sub-data in the at least one piece of sub-data, each piece of data in each piece of sub-data is traversed to perform field conversion, thereby obtaining conversion sub-data composed of each piece of data in each piece of sub-data, and then combining the conversion sub-data corresponding to each piece of sub-data to generate the first conversion data.

[0126] For example, taking the cash_fund.xml file as an example, for the first piece of sub-data (2000 entries) in the at least one piece of sub-data, one piece of data in the first piece of sub-data is:

[0127] fund_id=001, branch_id=1001, partition=1, balance=2000.22000;

[0128] After the fields are converted according to the preset conversion configuration corresponding to each configuration name, the conversion results are:

[0129] <cash_fund>

[0130] <row>

[0131] <fundid> 001< / fundid>

[0132] <partition> 1001< / partition>

[0133] <branchid> 1< / branchid>

[0134] <fundbalance> 2000.2200< / fundbalance>

[0135] <row>

[0136] < / cash_fund>

[0137] FundID:{column:fund id,type:string,length:1,precision:0,order:0},

[0138] Among them, FundID rule: take the fund_id value in the result, string, length 1, precision 0, sort 0;

[0139] Partition:{column:partition,type:String,length:1,precision:0,order:0},

[0140] Among them, the Partition rule is: take the partition value in the result, a string with a length of 1, a precision of 0, and a sorting of 0;

[0141] BranchID:{column:branch_id,type:String,length:1,precision:0,order:0},

[0142] Among them, the BranchID rule is: take the branch_id value in the result, a string with a length of 1, a precision of 0, and a sorting of 0;

[0143] FundBalance: {column:balance,type:BigDecimal,length:1,precision:4,order:0}

[0144] Among them, FundBalance rule: take the balance value in the result, BigDecimal, length 1, precision 4 decimal places, sort 0; BigDecimal: Java provides the API class BigDecimal in the java.math package, which is used to perform precise calculations on numbers with more than 16 significant digits.

[0145] Therefore, the above embodiment performs sharding processing on the first source data corresponding to each configuration name, so as to disperse each piece of sub-data in at least one piece of sub-data corresponding to each first source data to multiple nodes for processing, thereby improving the performance and throughput of the securities business system while enhancing the scalability of the securities business system.

[0146] In one embodiment, the device further comprises:

[0147] The batch processing framework is used to concurrently perform field transformation on each piece of data in each piece of sub-data.

[0148] In this embodiment, in order to further improve the performance of the securities business system, the method also uses a batch processing framework to concurrently perform field conversion on each piece of data in each piece of sub-data; wherein the batch processing framework can be a springbatch framework.

[0149] The generating unit 105 is used to write the first conversion data corresponding to each first source data into the corresponding first empty file according to each concurrent thread, so as to generate a first conversion file corresponding to each first source data.

[0150] In this embodiment, after obtaining the first conversion data corresponding to each first source data, in order to further realize the file conversion of each first source data, the first conversion data corresponding to each first source data is written into the corresponding first empty file according to each concurrent thread to generate the first conversion file corresponding to each first source data. Wherein, the first conversion data corresponding to each first source data is written into the corresponding first empty file can be specifically executed by writers such as xmlWriter and BindWirter; in addition, after generating the first conversion file corresponding to each first source data, in order to confirm the status of each parallel thread, the conversion status of each first source data is detected; wherein, if it is detected that each first source data has generated the corresponding first target conversion file, then the calling of each concurrent thread is stopped. In addition, the securities business system also provides an http interface to convert the first conversion file corresponding to each first source data into a visual file for the securities dealer to view; wherein, according to the difference in usage scenarios, the generation time of the first conversion file corresponding to each first source data into a visual file is different. For example, in the securities industry, some exchange files arrive at the brokerage firm first, while others arrive at the brokerage firm later. Therefore, the http interface supports batch conversion as well as conversion of a single file.

[0151] It can be seen from the above technical solutions that the present invention realizes concurrent conversion processing of each configuration name in the target configuration name list, thereby improving data conversion efficiency; and the first source data can be changed by modifying the data model, making it more scalable.

[0152] The data conversion device can be implemented in the form of a computer program. Figure 3 Runs on the computer device shown.

[0153] See also Figure 3 , Figure 3 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 is a server or a server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0154] See also Figure 3 The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a device bus 501 , wherein the memory may include a storage medium 503 and an internal memory 504 .

[0155] The storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 may execute a data conversion method.

[0156] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0157] The internal memory 504 provides an environment for the operation of the computer program 5032 in the storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute the data conversion method.

[0158] The network interface 505 is used for network communication, such as providing data information transmission, etc. Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0159] The processor 502 is used to run a computer program 5032 stored in the memory to implement the data conversion method disclosed in the embodiment of the present application.

[0160] Those skilled in the art will understand that Figure 3 The embodiments of the computer device shown in the figure do not constitute a limitation on the specific composition of the computer device. In other embodiments, the computer device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, in some embodiments, the computer device may only include a memory and a processor. In such embodiments, the structure and function of the memory and the processor are the same as those of the embodiment of the present invention. Figure 3 The embodiments shown are consistent and will not be described again here.

[0161] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0162] In another embodiment of the present application, a computer-readable storage medium is provided. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the data conversion method disclosed in the embodiment of the present application is implemented.

[0163] It will be clear to those skilled in the art that, for the convenience and simplicity of description, the specific working process of the above-described equipment, devices and units can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It will be appreciated by those of ordinary skill in the art that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0164] In the several embodiments provided in the present application, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, and units with the same function may be combined into one unit. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection.

[0165] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0166] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0167] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a background server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a magnetic disk or an optical disk.

[0168] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.< / row> < / row> < / row> < / row>

Claims

1. A data conversion method, applied to securities business system, It is characterized in that The method comprises: When receiving a configuration name list read instruction, read the target configuration name list from the data model; If a task concurrent processing instruction is detected, each configuration name in the target configuration name list is assigned to a concurrent thread pool according to the task concurrent processing instruction; Calling the concurrent threads corresponding to each configuration name in the concurrent thread pool, and obtaining the first empty file corresponding to each configuration name according to each concurrent thread, searching for the preset conversion configuration corresponding to each configuration name, and reading the first source data corresponding to each configuration name; According to each concurrent thread, the first source data corresponding to each configuration name is processed according to a preset sharding rule and a preset conversion configuration corresponding to each configuration name to obtain first conversion data corresponding to each first source data; According to each concurrent thread, the first conversion data corresponding to each first source data is written into the corresponding first empty file to generate a first conversion file corresponding to each first source data; After reading the target configuration name list from the data model, the method further includes: If a task serial processing instruction is detected, a serial thread corresponding to the target configuration name list is generated; wherein the serial thread executes serial tasks in sequence based on each configuration name; For each serial task corresponding to a configuration name, the serial thread is called, and according to the current serial task of the serial thread, a current configuration name and a target empty file corresponding to the current configuration name are obtained, a target preset conversion configuration corresponding to the current configuration name is searched, and target source data corresponding to the current configuration name is read; According to the current serial task of the serial thread, the target source data is processed according to the preset slicing rule and the target preset conversion configuration to obtain target conversion data; According to the current serial task of the serial thread, the target conversion data is written into the target empty file to obtain a target conversion file.

2. The data conversion method according to claim 1, It is characterized in that The first source data corresponding to each configuration name is processed according to the preset sharding rule and the preset conversion configuration corresponding to each configuration name to obtain the first conversion data corresponding to each first source data, including: Slice the first source data corresponding to each configuration name based on the preset slicing rule to obtain at least one piece of sub-data corresponding to each first source data; For at least one piece of sub-data corresponding to each first source data, field conversion is performed on each piece of sub-data in the at least one piece of sub-data according to the preset conversion configuration corresponding to each configuration name to obtain first conversion data corresponding to each first source data; wherein the first conversion data contains conversion sub-data corresponding to each piece of sub-data.

3. The data conversion method according to claim 1, It is characterized in that The securities business system also includes a core trading system and a database. Before reading the target configuration name list from the data model, it also includes: Responsive to the configuration building instruction, acquiring business data of the core transaction system; Reading a target database list from the database according to the business data; A preset conversion configuration set is generated according to the target database list; wherein the preset conversion configuration set includes at least one preset conversion configuration; each preset conversion configuration includes a file category, a file name, a file format and a conversion query logic.

4. The data conversion method according to claim 3, It is characterized in that The generating a preset conversion configuration set according to the target database list includes: Analyze the table data content format and table design field meanings in the target database list to obtain a conversion rule document; The conversion rule document is determined as the preset conversion configuration set.

5. The data conversion method according to claim 1, It is characterized in that The step of reading the target configuration name list from the data model includes: Read a first configuration name list from the data model file; Filtering the first configuration name list to obtain a second configuration name list; Each second configuration name in the second configuration name list is sorted according to a preset priority to obtain the target configuration name list.

6. The data conversion method according to claim 2, It is characterized in that The method further comprises: The batch processing framework is used to concurrently perform field transformation on each piece of data in each piece of sub-data.

7. A data conversion device, It is characterized in that The device comprises: A reading unit, configured to read a target configuration name list from a data model when receiving a configuration name list reading instruction; an allocating unit, configured to allocate each configuration name in the target configuration name list to a concurrent thread pool according to the task concurrent processing instruction if a task concurrent processing instruction is detected; A calling unit, configured to call the concurrent threads corresponding to each configuration name in the concurrent thread pool, and obtain the first empty file corresponding to each configuration name according to each concurrent thread, search for the preset conversion configuration corresponding to each configuration name, and read the first source data corresponding to each configuration name; A data processing unit, configured to process the first source data corresponding to each configuration name according to the preset sharding rule and the preset conversion configuration corresponding to each configuration name according to each concurrent thread, to obtain first conversion data corresponding to each first source data; A generating unit, configured to write the first conversion data corresponding to each first source data into the corresponding first empty file according to each concurrent thread, and generate a first conversion file corresponding to each first source data; After reading the target configuration name list from the data model, the method further includes: If a task serial processing instruction is detected, a serial thread corresponding to the target configuration name list is generated; wherein the serial thread executes serial tasks in sequence based on each configuration name; For each serial task corresponding to a configuration name, the serial thread is called, and according to the current serial task of the serial thread, a current configuration name and a target empty file corresponding to the current configuration name are obtained, a target preset conversion configuration corresponding to the current configuration name is searched, and target source data corresponding to the current configuration name is read; According to the current serial task of the serial thread, the target source data is processed according to the preset slicing rule and the target preset conversion configuration to obtain target conversion data; According to the current serial task of the serial thread, the target conversion data is written into the target empty file to obtain a target conversion file.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the data conversion method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the data conversion method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Data loading method and device, equipment and medium

    CN114860349A

  • Clearing task real-time execution method and device, equipment and medium

    CN115936885A

  • Data return method and device, equipment and medium

    CN118014732A

  • Efficient processing of rule-based computing workflows

    US11256557B1

  • Configurable data transformation between different database systems

    US20230144349A1