Data sorting method and device, electronic equipment and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- MASHANG CONSUMER FINANCE CO LTD
- Filing Date
- 2022-10-27
- Publication Date
- 2026-06-02
Smart Images
Figure CN116150144B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of data processing, and specifically relates to a data sorting method, apparatus, electronic device and storage medium. Background Technology
[0002] The batch deduction system is a system that processes and filters the data in the original integrated files of the accounting system according to certain rules, regenerating data with split codes and identification codes, and then pushing it to the payment system for batch deductions. In the batch deduction system, the original integrated files of the accounting system need to be processed to generate split codes and batch deduction identification codes for batch deduction comparison testing with the payment system. Before generating the batch deduction identification codes, the data records in the accounting integrated files need to be sorted in a multi-dimensional combination; that is, for data records containing multiple fields to be sorted, the data records are sorted according to the priority order and sorting method of different fields.
[0003] The system requires minimizing the total time overhead for sorting millions or tens of millions of data records. However, in related technologies, using MySQL databases to sort parsed data records results in sorting times ranging from minutes to hours due to the millions or even tens of millions of data records, failing to meet SLA (Service Level Agreement) and performance requirements. As the amount of data to be sorted increases to tens of millions, using database sorting can also lead to other unknown risks, such as a decline in overall database performance. Summary of the Invention
[0004] This application proposes a data sorting method, apparatus, electronic device, and storage medium to save sorting time, reduce the occupation of database resources, and improve the efficiency of data sorting when the number of data records to be sorted is large.
[0005] Firstly, this application provides a data sorting method, including:
[0006] Obtain the data records to be sorted, and determine the multiple fields contained in the data records as sorting fields;
[0007] Convert the original values of each sorting field into field information code values;
[0008] According to the field priority order, the field information code values of multiple sorting fields are concatenated in sequence to obtain the sorting key of the data record;
[0009] The data record containing the sort key is obtained by combining the sort key and the data record to be sorted, and then written into the key-value database; the key-value database is used to sort the data records according to the sort key corresponding to them.
[0010] Retrieve data records sorted by a key-value database.
[0011] Secondly, this application provides a data sorting apparatus, comprising:
[0012] The field extraction module is used to obtain the data records to be sorted and determine the multiple fields contained in the data records as sorting fields;
[0013] The conversion module is used to convert the original values of each sorting field into field information code values.
[0014] The generation module is used to concatenate the field information code values of multiple sorting fields in order of field priority to obtain the sorting key of the data record.
[0015] The sorting module is used to combine the sort key and the data record to be sorted to obtain the data record containing the sort key, and write the data record containing the sort key into the key-value database; the key-value database is used to sort the data records according to the sort key corresponding to them.
[0016] The results retrieval module is used to retrieve data records that have been sorted using a key-value database.
[0017] Thirdly, this application provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor;
[0018] The memory stores one or more computer programs that can be executed by at least one processor, which in turn execute the data sorting method described above.
[0019] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data sorting method described above.
[0020] In the data sorting method provided in this application, multiple fields contained in the data records to be sorted are identified as sorting fields. The original value corresponding to each sorting field is then converted into an information code value. Next, the information code values of the multiple sorting fields are concatenated sequentially according to field priority to obtain a sorting key. Finally, the sorting key and the data records to be sorted are combined to obtain a data record containing the sorting key, which is then written into a key-value database. This yields the sorted data record obtained through the key-value database. Therefore, this method can generate a sorting key containing multiple sorting fields by processing multiple fields contained in the data record. Based on different sorting methods of multiple sorting fields, it can achieve multi-dimensional combination sorting of data records, thereby shortening the sorting time, reducing the occupation of database resources, reducing system risks, and improving data sorting performance when the number of data records to be sorted is large.
[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description
[0022] The accompanying drawings are provided to further illustrate the present application and form part of the specification. They are used together with the embodiments of the present application to explain the application and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed example embodiments described with reference to the accompanying drawings, in which:
[0023] Figure 1 A flowchart of a data sorting method provided in Embodiment 1 of this application is shown;
[0024] Figure 2 A flowchart of a data sorting method provided in Embodiment 2 of this application is shown;
[0025] Figure 3 A flowchart of the data sorting method provided in Specific Example 1 of Embodiment 2 of this application is shown;
[0026] Figure 4 A flowchart of the data sorting method provided in Specific Example 2 of Embodiment 2 of this application is shown;
[0027] Figure 5 This paper shows a structural block diagram of a work order data detection device provided in Embodiment 3 of this application;
[0028] Figure 6 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of this application is shown. Detailed Implementation
[0029] To enable those skilled in the art to better understand the technical solutions of this application, exemplary embodiments of this application are described below in conjunction with the accompanying drawings, including various details of the embodiments of this application to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0030] Where there is no conflict, the various embodiments of this application and the features thereof may be combined with each other.
[0031] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.
[0032] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.
[0033] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.
[0034] The work order data detection method according to the embodiments of this application can be executed by electronic devices such as terminal devices or servers. Terminal devices can be in-vehicle devices, user equipment (UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistants (PDAs), handheld devices, computing devices, in-vehicle devices, wearable devices, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Specifically, the method can be implemented by a processor calling a computer program stored in memory.
[0035] In related technologies, for massive datasets of millions or tens of millions of records, using MySQL databases for storage and SQL `ORDER BY` syntax for multi-dimensional sorting results in processing times reaching minutes or even hours due to the sheer volume of data. This fails to meet SLA and performance requirements and can lead to other unknown risks, such as overall database performance degradation. To address these issues, this application provides a data sorting method. This method processes multiple fields within a data record to generate a sorting key containing multiple sorting fields. Based on different sorting methods for these fields, it achieves multi-dimensional combined sorting of the data records. This reduces sorting time, decreases database resource consumption, mitigates system risks, and improves data sorting performance, even with a large number of data records to be sorted.
[0036] Example 1
[0037] Figure 1 A flowchart of a data sorting method provided in Embodiment 1 of this application is shown. (Refer to...) Figure 1 The method includes:
[0038] Step S110: Obtain the data records to be sorted, and determine the multiple fields contained in the data records as sorting fields.
[0039] The data to be sorted consists of multiple records, each containing multiple fields, each representing a different meaning. Multiple fields are manually selected from all fields in the data records to serve as sorting fields. To reflect the characteristics of multi-dimensional combination sorting, the number of sorting fields is two or more.
[0040] Step S120: Convert the original field value corresponding to each sorting field into the field information code value.
[0041] In one optional implementation, each character in the original field value is converted to its corresponding ASCII code value by referring to the ASCII code table. The result is the field information code value. Converting the original field value corresponding to each sorting field into a uniform format field information code value facilitates subsequent sorting of the fields by comparing the magnitudes of the ASCII code values. Optionally, the field information code value can also be other forms of information code, which should be set by those skilled in the art according to specific circumstances when implementing this method, and should not be limited by the definition in this embodiment.
[0042] Step S130: Concatenate the field information code values of multiple sorting fields in order of field priority to obtain the sorting key of the data record.
[0043] The priority of the fields is set manually in advance, and the field information code value of each sorting field is obtained in step S120. The field information code values of each sorting field are concatenated from front to back according to the order of field priority from high to low, and the concatenated result is used as the sorting key of the data record.
[0044] Step S140: Combine the sort key and the data record to be sorted to obtain the data record containing the sort key, and write the data record containing the sort key into the key-value database; wherein, the key-value database is used to sort according to the sort key corresponding to the data record.
[0045] The data records written to the key-value database include the original data content of the data records to be sorted and a sort key generated based on the sorting field in the original data content. The sort key is used as the key of the data record, and the original data content is used as the value of the data record. The data record containing the sort key is then written to the key-value database.
[0046] Step S150: Obtain the data records sorted by the key-value database.
[0047] Specifically, the data to be sorted is processed through steps S110-S140 and written into a key-value database. The key-value database sorts the written data records according to the sorting key contained in the data records. At this time, the data records in the key-value database are traversed and read, and the data records obtained are the data records after being sorted by the key-value database.
[0048] In summary, the data sorting method provided in this embodiment determines multiple fields contained in the data records to be sorted as sorting fields. Then, the original value corresponding to each sorting field is converted into an information code value. Next, according to the field priority, the information code values of multiple sorting fields are concatenated sequentially to obtain a sorting key. Finally, the sorting key and the data records to be sorted are combined to obtain a data record containing the sorting key, which is then written into a key-value database. This yields the sorted data record obtained through the key-value database. Therefore, this method can generate a sorting key containing multiple sorting fields by processing multiple fields contained in the data record. Based on different sorting methods of multiple sorting fields, it achieves multi-dimensional combination sorting of data records, thereby shortening the sorting time, reducing database resource consumption, lowering system risk, and improving data sorting performance when the number of data records to be sorted is large.
[0049] Example 2
[0050] Figure 2 A flowchart of a data sorting method provided in Embodiment 2 of this application is shown. (Refer to...) Figure 2 The method includes:
[0051] Step S210: Obtain the data records to be sorted, and determine the multiple fields contained in the data records as sorting fields.
[0052] The data to be sorted consists of multiple records, each containing multiple fields, each representing a different meaning. Multiple fields from all the fields in each data record are pre-determined to serve as sorting fields. The system receives field configuration instructions input through the sorting field configuration entry in the configuration interface. Based on the sorting field identifiers contained in the field configuration instructions, multiple fields are extracted from the data records as sorting fields; the sorting field identifier is the field name of that sorting field. To reflect the characteristics of multi-dimensional combined sorting scenarios, the number of sorting fields is two or more.
[0053] Step S220: Obtain the sorting method of the sorting field.
[0054] Different sorting fields represent different meanings. The sorting method for each sorting field in the data records to be sorted is predetermined, and the sorting method can be ascending or descending. The system receives sorting configuration commands input through the field sorting configuration entry in the configuration interface and determines the sorting method for each sorting field based on these commands. Additionally, the sorting method for a sorting field can also be determined based on the field's business type.
[0055] Step S230: Convert the original values of the fields corresponding to the sorting fields into field information code values according to the sorting method.
[0056] In step S220, the sorting method of each sorting field in the data records to be sorted is obtained. Then, for different sorting methods, the original value of each sorting field is converted into a field information code value in different ways:
[0057] If the sorting method is ascending, the original field value is converted into an information exchange standard code; specifically, based on the mapping relationship between characters and information exchange standard codes, each character contained in the original field value is converted into an information exchange standard code.
[0058] In one specific implementation, each character in the original field value is converted to its corresponding ASCII code value by referring to the ASCII code table. The result is the field information code value. Converting the original field value of each sorting field into a uniform format field information code value facilitates subsequent sorting of the fields by comparing the magnitudes of the ASCII code values. Optionally, the field information code value can also be other forms of information code, which should be set by those skilled in the art according to specific circumstances when implementing this method, and should not be limited by the definition in this embodiment.
[0059] In traditional field sorting methods, multiple sorting fields are sorted in the same way, such as all using ascending order or all using descending order. Sorting is only convenient when the sorting methods of the sorting fields are the same. In the business scenario of this embodiment, the sorting methods of each sorting field may be different, making it impossible to process them using a uniform method. To solve this problem, in this embodiment, if the sorting method is descending, an inversion operation is performed on the information exchange standard code, which is obtained by converting the original field value. In this embodiment, for sorting fields with descending sorting, the information exchange standard code for the descending sorting field is first determined, and then an inversion operation is performed on the information exchange standard code for the descending sorting field. By using the inversion operation, on the one hand, the descending sorting field can be included in the final sorting key along with the ascending sorting field; on the other hand, it ensures that the sorting processing methods for the descending sorting field and the ascending sorting field are different.
[0060] In one specific implementation, the information exchange standard code contains multiple bytes. Performing an inversion operation on the information exchange standard code includes: performing a byte inversion operation on each byte contained in the information exchange standard code; wherein, the specific implementation of the byte inversion operation is: calculating the difference between the preset maximum byte value and the information exchange standard code value of the byte to obtain the result of the byte inversion operation.
[0061] In one specific implementation, the correspondence in the ASCII code table is used as the mapping relationship between the original value of the field and the information exchange standard code. The information exchange standard code is in bytes. The inversion operation includes: for each byte of the information exchange standard code, subtract the code value of the current byte from the maximum code value to obtain the inversion result of the byte. Since 1 byte contains 8 bits, the maximum value of 8-bit binary ASCII code is 127.
[0062] Step S240: Concatenate the field information code values of multiple sorting fields in order of field priority to obtain the sorting key of the data record.
[0063] Specifically, when concatenating the field information code values of multiple sorting fields, the field information code values of the multiple sorting fields need to be concatenated sequentially to form the sort key of the data record according to the field priority of each sorting field. Since the order of the field information code values in the sort key directly affects the final sorting result of the data record, in this embodiment, a field priority needs to be set for each sorting field so that the order of the field information code values in the sort key can be determined according to the field priority.
[0064] In one optional implementation, field priorities are pre-determined, and priority configuration instructions are received from field priority configuration entry points included in the configuration interface. The priority of each sorting field is determined based on these instructions. The field priorities are set by those skilled in the art according to specific circumstances when implementing this method, and are not limited in this embodiment. This approach primarily provides multiple field priority configuration entry points in the configuration interface, allowing users to flexibly configure the priorities of each sorting field. This approach helps users customize the priorities of each sorting field according to the characteristics of their business scenarios, and the configured field priorities can be flexibly changed during each sorting process, making it adaptable to various scenarios.
[0065] In another optional implementation, the field attribute information of each sorting field is obtained, and the field priority of each sorting field is configured based on this information. The field attribute information for each sorting field includes: field type, field query frequency, and field value storage format. The field type can be categorized based on business type, field function, etc. For example, the storage field corresponding to core projects in the business can be given a higher priority, while the storage field corresponding to peripheral projects can be given a lower priority. The field query frequency is dynamically determined based on the frequency of data query requests containing that field received within a preset time period. The preset time period can be a specified period corresponding to the current system time. For example, assuming the current system time is February 1st, the preset time period can be from January 1st to January 31st (i.e., the period within one month prior to the current system time). Therefore, the field query frequency and the field priority set based on the query frequency vary depending on the trigger time of the sorting operation, allowing for flexible dynamic adjustment of field priorities based on the online business data query situation. Furthermore, the storage format of field values indicates whether the data within the field is stored in key-value pairs, long integers, or other formats. Specifically, the priority of the sorting field corresponding to a specified format can be set to higher or lower, depending on business requirements. In short, this method eliminates the need for manual configuration of field priorities; it automatically determines field priorities based on field attribute information. Moreover, field attribute information (such as query frequency) can be dynamically changed, ensuring that field priority settings are consistent with the latest business system status.
[0066] In step S220, the field information code value of each sorting field is obtained; according to the field priority from high to low, the field information code values of each sorting field are concatenated from front to back, and the concatenated result is used as the sorting key of the data record.
[0067] Step S250: Combine the sort key and the data record to be sorted to obtain the data record containing the sort key, and write the data record containing the sort key into the key-value database; wherein, the key-value database is used to sort according to the sort key corresponding to the data record.
[0068] The data records written to the key-value database include the original data content of the data records to be sorted and a sort key generated based on the sorting field in the original data content. The sort key is used as the key of the data record, and the original data content is used as the value of the data record. The data record containing the sort key is then written to the key-value database.
[0069] Step S260: Obtain the data records sorted by the key-value database.
[0070] Specifically, the data to be sorted is processed through steps S210-S250 and written into a key-value database. The key-value database sorts the written data records according to the sorting key contained in the data records. At this time, the data records in the key-value database are traversed and read, and the data records obtained are the data records after being sorted by the key-value database.
[0071] In one specific implementation, if there are multiple data records to be sorted, the data record containing the sort key is written to a key-value database. The data records obtained after sorting through the key-value database include:
[0072] Create a key-value database instance;
[0073] Multiple data records containing sort keys are written to the memory area of the key-value database through a key-value database instance;
[0074] The refresh operation stores multiple data records containing sorting keys from memory to the disk area of the key-value database, and retrieves the sorted data records from the disk area.
[0075] In one optional implementation, the data records to be sorted are the original accounting data records. The purpose of multi-dimensional combination sorting of the original accounting data records is to process the accounting data records, generate split codes and batch deduction identification codes, and provide them to the payment system for batch deduction comparison testing. Therefore, after obtaining the data records sorted through a key-value database, the process also includes: generating deduction identification codes for the sorted data records; and performing batch deduction operations based on the deduction identification codes.
[0076] In summary, the data sorting method provided in this application identifies multiple fields contained in the data records to be sorted as sorting fields. Then, based on different sorting methods for each sorting field, the original value corresponding to each sorting field is converted into an information code value in different ways. Next, according to the field priority, the information code values of multiple sorting fields are concatenated sequentially to obtain a sort key. Finally, the sort key and the data records to be sorted are combined to obtain a data record containing the sort key, which is then written into a key-value database. This yields the sorted data record obtained through the key-value database. Therefore, this method can generate a sort key containing multiple sorting fields by processing multiple fields contained in the data record. Based on different sorting methods of multiple sorting fields, it achieves multi-dimensional combination sorting of data records, thereby shortening the sorting time, reducing database resource consumption, lowering system risk, and improving data sorting performance when dealing with a large number of data records to be sorted.
[0077] To facilitate understanding, two specific examples will be used to describe in detail the specific implementation of this embodiment.
[0078] In the batch deduction system, it is necessary to process the original integrated file data of the accounting system. The data format is 0dda90d5c7934408a94492c75f806144|110107090401000927|232|3259.12|1911.12|675.52|2022-07-02|. The data meanings, in descending order of vertical lines, are customer number, contract number, overdue CPD (customer's overdue days), remaining total debt, current debt, next debt, and due date. A split code (a code generated after the batch deduction strategy is executed, which the payment system uses to determine whether the batch deduction is executed in the morning or afternoon, code format: split_default_1) and a batch deduction identifier code (which the payment system uses to determine whether to use the primary card or the secondary card for deduction, code format: 0405, 0505) are generated for batch deduction comparison testing with the payment system. Before generating the batch deduction identification code, the accounting integration file data needs to be sorted in multiple dimensions, such as sorting by overdue CPD in ascending order and due date in descending order. The system requires that the total time cost of sorting millions or tens of millions of data should be as small as possible.
[0079] Related technologies use MySQL databases to store and sort data. The data sorting method includes: creating database tables corresponding to the file data format; repeatedly reading file data; batch inserting data into the database; after all data has been inserted, performing multi-dimensional sorting using SQL `ORDER BY` syntax; and reading the sorted data. This method has the following problems: inserting millions or tens of millions of data points into the database is slow; multi-field sorting is time-consuming, consuming database resources for extended periods and causing an overall performance degradation; performing a multi-field `ORDER BY` sort on millions of data points in MySQL can take minutes or even hours; after sorting, querying the sorted data also requires time, resulting in an excessively long process that fails to meet performance requirements.
[0080] To address the aforementioned issues, Specific Example 1 and Specific Example 2 are illustrated using a data sorting method in a batch deduction system as an example, based on a data sorting method provided in Embodiment 2 of this application.
[0081] The key-value database used is RocksDB, an LSM-tree architecture engine developed by Facebook based on Level DB, providing key-value storage and read / write capabilities. User-written key-value pairs are first written to the Write Ahead Log (WAL) on disk, and then to a skip list (also known as a MemTable) in memory. Because the LSM-tree engine transforms random user modifications (insertions) into sequential writes to the WAL file, it has a higher write throughput than B-tree storage engines.
[0082] Figure 3 A flowchart illustrating the data sorting method provided in Specific Example 1 is shown. (Refer to...) Figure 3 This example includes the following detailed steps:
[0083] ① Create a RocksDB instance;
[0084] ②Read accounting file data in a loop;
[0085] ③ Read the sorting fields and sorting rules from the database. The fields and sorting rules to be sorted are manually configured through the interface. The configuration interface will list all field selections in the integration file, the field order selection (ascending or descending), and the field sorting priority selection. Each modification is saved to the database. Example:
[0086]
[0087] ④ After reading each line of data from the integrated file, convert all fields that need to be sorted in the data into bytes according to the sorting rules, and combine them into a KEY in binary byte form;
[0088] ⑤ Write the KEY and data obtained in step ④ into RocksDB in the form of KEY, VALUE;
[0089] ⑥ After all the data has been written, start traversing RocksDB from the beginning to obtain the sorted data. The data reading time is in the order of seconds.
[0090] ⑦ Shut down the RocksDB instance. End.
[0091] Figure 4 A flowchart illustrating the data sorting method provided in Specific Example 2 is shown. (Refer to...) Figure 4 This example includes the following detailed steps:
[0092] ① When sorting and tagging in the batch deduction system, read the integrated file data pushed by the accounting system and sort it;
[0093] ② Read externally defined sorting rules and filter the data;
[0094] ③ According to the sorting rules, convert all fields that need to be sorted for each piece of data that conforms to the sorting rules into bytes, and combine them into a KEY in binary byte form;
[0095] ④ Create a local RocksDB instance;
[0096] ⑤ Write the sorted keys and data into RocksDB in the form of key, value;
[0097] ⑥ Read and write in a loop;
[0098] ⑦ Once writing is complete, flush the RocksDB data in memory to the disk;
[0099] ⑧ Read RocksDB from the beginning in a loop to obtain the sorted data;
[0100] 9. After reading is complete, close the RocksDB instance;
[0101] ⑩ Sorting complete.
[0102] In summary, unlike related technologies that parse the integrated file data and store it in a MySQL database for sorting, this approach is problematic because the file data volume can reach millions or even tens of millions, and database sorting can take minutes or even hours, failing to meet SLA and performance requirements. Furthermore, as the integrated file data volume increases to tens of millions, using database sorting will introduce other unknown risks, such as a decline in overall database performance.
[0103] In the specific example above, by introducing RocksDB's high-performance KV(KEY, VALUE) local storage, and leveraging RocksDB's automatic sorting of the KEY by dictionary (lexicographical order is based on the ASCII code size of the characters, left-aligned, and sorted from left to right, stopping the comparison of subsequent characters once a size is found), all fields that need to be sorted are converted into bytes and then concatenated into a KEY according to predefined sorting rules (for example, sorting by overdue CPD value of 232 in ascending order and due date value of 2022-07-02 in descending order, the generated KEY is 50515077797777827972827977 in binary format), and written to RocksDB. After all the data has been written, RocksDB is read from the beginning, and the read data is the sorted data. The entire process takes only seconds, shortening the total sorting time, reducing database resource usage, reducing system risk, meeting SLA requirements, significantly improving sorting performance, and solving the system's sorting performance bottleneck.
[0104] Example 3
[0105] Figure 5 A structural block diagram of a data sorting apparatus 50 according to Embodiment 3 of this application is shown. (Refer to...) Figure 5 The device 50 includes:
[0106] The field extraction module 51 is used to obtain the data records to be sorted and to determine the multiple fields contained in the data records as sorting fields;
[0107] The conversion module 52 is used to convert the original field value corresponding to each sorting field into the field information code value respectively;
[0108] The generation module 53 is used to concatenate the field information code values of multiple sorting fields in order of field priority to obtain the sorting key of the data record.
[0109] The sorting module 54 is used to combine the sort key and the data record to be sorted to obtain the data record containing the sort key, and write the data record containing the sort key into the key-value database; wherein, the key-value database is used to sort according to the sort key corresponding to the data record;
[0110] The result acquisition module 55 is used to acquire data records sorted by a key-value database.
[0111] Optionally, the conversion module 52 is specifically used for:
[0112] Get the sorting method of the sorting field;
[0113] Convert the original values of the fields corresponding to the sorting fields into field information code values according to the sorting method.
[0114] Optionally, the conversion module 52 is specifically used for:
[0115] If the sorting method is ascending, convert the original field value into the information exchange standard code;
[0116] If the sorting method is descending, the information exchange standard code is inverted, and the information exchange standard code is obtained by converting the original field value.
[0117] Optionally, the conversion module 52 is specifically used for:
[0118] Based on the mapping relationship between characters and standard codes for information exchange, each character contained in the original value of the field is converted into a standard code for information exchange.
[0119] Optionally, the information exchange standard code contains multiple bytes, and the conversion module 52 is specifically used for:
[0120] Perform a byte inversion operation on each byte contained in the information exchange standard code;
[0121] The specific implementation of the byte inversion operation is as follows: calculate the difference between the preset maximum byte value and the byte's information exchange standard code value to obtain the byte inversion result.
[0122] Optionally, module 51 is specifically used for:
[0123] It receives field configuration instructions input through the sorting field configuration entry in the configuration interface, and extracts multiple fields from the data record as sorting fields according to the sorting field identifier contained in the field configuration instructions.
[0124] Optionally, the order of field priorities is determined by receiving a priority configuration instruction input through the field priority configuration entry included in the configuration interface, and determining the field priority of each sorting field according to the priority configuration instruction;
[0125] The sorting method of the sorting field is determined in the following way: receiving the sorting configuration instruction input through the field sorting configuration entry included in the configuration interface, and determining the sorting method of each sorting field according to the sorting configuration instruction.
[0126] Optionally, the data records to be sorted are the original accounting data records, and the result acquisition module 55 is also used for:
[0127] Generate a deduction identification code for the sorted data records;
[0128] Perform batch deduction operations based on the deduction identification code.
[0129] The specific structure and working principle of each of the above modules can be found in the descriptions of the corresponding parts of Method Embodiment 1 and Embodiment 2, and will not be repeated here.
[0130] Example 4
[0131] Figure 6 This is a block diagram of an electronic device 60 provided in Embodiment 4 of this application.
[0132] Reference Figure 6 This application provides an electronic device 60, which includes: at least one processor 601; at least one memory 602; and one or more I / O interfaces 603 connected between the processor 601 and the memory 602; wherein the memory 602 stores one or more computer programs 605 that can be executed by at least one processor 601, and the one or more computer programs 605 are executed by at least one processor 601 using the above-described data sorting method.
[0133] This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor / processor core, implements the aforementioned data sorting method. The computer-readable storage medium can be volatile or non-volatile.
[0134] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).
[0135] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.
[0136] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.
[0137] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.
[0138] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.
[0139] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.
[0140] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0141] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0142] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0143] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for general illustrative purposes only and should not be construed as limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this application as set forth by the appended claims.
Claims
1. A data sorting method, characterized in that, include: Obtain the data records to be sorted, and determine the multiple fields contained in the data records as sorting fields; Each sorting field's original value is converted into a field information code value; wherein, when sorting the sorting fields in descending order, the field information code value is obtained by inverting the information standard exchange code corresponding to the original field value. According to the field priority order, the field information code values of the multiple sorting fields are concatenated sequentially to obtain the sorting key of the data record; By combining the sort key and the data record to be sorted, a data record containing the sort key is obtained, and the data record containing the sort key is written into a key-value database; wherein, the key-value database is used to sort the data records according to the sort key corresponding to them during the writing process; Retrieve the data records sorted by the key-value database.
2. The method according to claim 1, characterized in that, The step of converting the original value of each sorting field into a field information code value includes: Obtain the sorting method of the sorting field; The original values of the fields corresponding to the sorting fields are converted into field information code values according to the sorting method.
3. The method according to claim 2, characterized in that, The step of converting the original value of the field corresponding to the sorting field into the field information code value according to the sorting method includes: If the sorting method is ascending, the original field value is converted into an information exchange standard code; If the sorting method is descending, the information exchange standard code is inverted, and the information exchange standard code is obtained by converting the original field value.
4. The method according to claim 3, characterized in that, The step of converting the original value of the field into the information exchange standard code includes: converting each character contained in the original value of the field into the information exchange standard code according to the mapping relationship between characters and information exchange standard codes.
5. The method according to claim 3, characterized in that, The information exchange standard code contains multiple bytes, and the inversion operation on the information exchange standard code includes: performing a byte inversion operation on each byte contained in the information exchange standard code; The specific implementation of the byte inversion operation is as follows: calculate the difference between the preset maximum byte value and the information exchange standard code value of the byte to obtain the result of the byte inversion operation.
6. The method according to any one of claims 1-5, characterized in that, The step of determining multiple fields contained in the data record as sorting fields includes: receiving a field configuration instruction input through a sorting field configuration entry in the configuration interface, and extracting multiple fields from the data record as sorting fields according to the sorting field identifier contained in the field configuration instruction; The order of the field priorities is determined in the following way: receiving a priority configuration instruction input through the field priority configuration entry included in the configuration interface, and determining the field priority of each sorting field according to the priority configuration instruction; The sorting method of the sorting field is determined by receiving a sorting configuration instruction input through the field sorting configuration entry in the configuration interface, and determining the sorting method of each sorting field according to the sorting configuration instruction.
7. The method according to claim 1, characterized in that, The data records to be sorted are original accounting data records. After obtaining the data records sorted by the key-value database, the process further includes: Generate a deduction identification code for the sorted data records; Perform batch deduction operations based on the deduction identification code.
8. A data sorting device, characterized in that, include: The field extraction module is used to obtain the data records to be sorted and determine the multiple fields contained in the data records as sorting fields; The conversion module is used to convert the original values of each sorting field into field information code values. The generation module is used to concatenate the field information code values of the multiple sorting fields in order of field priority to obtain the sorting key of the data record; wherein, when sorting the sorting fields in descending order, the field information code value is obtained by performing an inversion operation on the information standard exchange code corresponding to the original value of the field; A sorting module is used to combine the sorting key and the data record to be sorted to obtain a data record containing the sorting key, and to write the data record containing the sorting key into a key-value database; wherein, the key-value database is used to sort the data records according to the sorting key corresponding to them during the writing process; The result acquisition module is used to acquire data records sorted by the key-value database.
9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when executed by a processor, implements the method as described in any one of claims 1-7.