Data processing method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a data processing method and device, electronic equipment, a storage medium and a program product. According to the implementation scheme, the method comprises the steps that a to-be-processed statement is determined; obtaining two adjacent grouping items in a data source corresponding to a to-be-processed statement and de-duplication items corresponding to the grouping items; determining whether to add the duplicate removal item corresponding to the top grouping item in two adjacent grouping items to a set function result or not on the basis of a comparison result of the obtained grouping items and the obtained duplicate removal items; continuing to select the next grouping item and the duplicate removal item corresponding to the grouping item from the data source, and returning to continue to determine whether the set function processing is passed or not until the last grouping item in the data source is obtained; and obtaining a set function result of the group corresponding to the group item. The method for determining the processing result of the set function is obtained by comparing the grouping items with the duplicate removal items, the calculation process is simplified, and the function of rapidly calculating the result of the set function is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, device, electronic device, storage medium and program product. Background Art

[0002] When operating a database through a structured query statement, the parameters of the set function in the statement need to be deduplicated. Generally, a data structure is used to store different rows of a data set in different places (for example, if a hash table is used, different rows are stored in different hash slots). Since it is necessary to calculate the deduplication results within a group, another data structure (such as a hash table) is required to store different set function values ​​of the same group. Finally, all the data stored in the data structure storing different set function values ​​are aggregated according to different groups and the final result is calculated.

[0003] As can be seen, the above method is relatively complicated, and it is necessary to create an additional data structure and cache all the data in the table before the grouped set function value can be obtained. The calculation process is relatively complicated and occupies a large amount of memory, and the performance is poor. Summary of the invention

[0004] The present invention provides a data processing method, device, electronic device, storage medium and program product to obtain a method for determining the processing result of a set function, thereby realizing the function of quickly calculating the result of the set function.

[0005] According to one aspect of the present invention, there is provided a data processing method, comprising:

[0006] Determine a statement to be processed, wherein the statement to be processed includes a structured query statement of a set function with a set mark, wherein the set mark includes a mark for removing duplicate values ​​in a parameter of the set function;

[0007] Obtain two adjacent grouping items in the data source corresponding to the statement to be processed and the deduplication items corresponding to each of the grouping items, wherein the data in the data source is ordered, and the data source includes multiple grouping items and the deduplication items corresponding to each of the grouping items;

[0008] Based on the comparison results of the obtained grouping items and the obtained deduplication items, respectively, determine whether to add the deduplication item corresponding to the grouping item with a higher order in the two adjacent grouping items to the determination of the set function result; if so, add the deduplication item corresponding to the grouping item with a higher order in the two adjacent grouping items to the determination of the set function result;

[0009] Continue to select the next grouping item and the deduplication item corresponding to the grouping item from the data source, and use the grouping item with the later order in the two adjacent grouping items and the next grouping item as two new adjacent grouping items, and return to continue to determine whether to pass the set function processing until the last grouping item in the data source is obtained;

[0010] The deduplication item corresponding to the last grouping item is added to the determination of the set function result to obtain the set function result of the grouping corresponding to the grouping item.

[0011] According to another aspect of the present invention, there is provided a data processing device, comprising:

[0012] A first determination module is used to determine a statement to be processed, wherein the statement to be processed includes a structured query statement of a set function with a set mark, and the set mark includes a mark for removing duplicate values ​​in a parameter of the set function;

[0013] An acquisition module, used for acquiring two adjacent grouping items and deduplication items corresponding to each of the grouping items in a data source corresponding to the statement to be processed, wherein the data in the data source is ordered, and the data source includes multiple grouping items and deduplication items corresponding to each grouping item;

[0014] A second determination module is used to determine whether to add the deduplication item corresponding to the grouping item with a higher order in the two adjacent grouping items to the determination of the set function result based on the comparison results of the obtained grouping items and the obtained deduplication items, and if so, add the deduplication item corresponding to the grouping item with a higher order in the two adjacent grouping items to the determination of the set function result;

[0015] A return module is used to continue to select the next grouping item and the deduplication item corresponding to the grouping item from the data source, and take the grouping item with the later order in the two adjacent grouping items and the next grouping item as two new adjacent grouping items, and return to continue to determine whether to pass the set function processing until the last grouping item in the data source is obtained;

[0016] An adding module is used to add the deduplication item corresponding to the last grouping item to the determination of the set function result, so as to obtain the set function result of the grouping corresponding to the grouping item.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the data processing method described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method described in any embodiment of the present invention when executed.

[0022] According to another aspect of the present invention, a computer program product is provided. The computer program product comprises a computer program. When the computer program is executed by a processor, the data processing method according to any embodiment of the present invention is implemented.

[0023] The technical solution of the embodiment of the present invention is to determine the statement to be processed, and obtain the two adjacent grouping items and the deduplication items corresponding to each of the grouping items in the data source corresponding to the statement to be processed. Then, based on the comparison results of the obtained grouping items and the obtained deduplication items, determine whether to add the deduplication items corresponding to the grouping items with a higher order in the two adjacent grouping items to the determination of the set function result. If so, add the deduplication items corresponding to the grouping items with a higher order in the two adjacent grouping items to the determination of the set function result. Continue to select the next grouping item and the deduplication items corresponding to the grouping item from the data source, and use the grouping item with a lower order in the two adjacent grouping items and the next grouping item as the two new adjacent grouping items, and return to continue to determine whether to process through the set function until the last grouping item in the data source is obtained. Finally, add the deduplication items corresponding to the last grouping item to the determination of the set function result to obtain the set function result of the grouping corresponding to the grouping item. By comparing the grouping items and the deduplication items in the data source, a method for determining the processing results of the set function is obtained, the calculation process is simplified, and the function of quickly calculating the results of the set function is realized.

[0024] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 is a flow chart of a data processing method provided according to Embodiment 1 of the present invention;

[0027] Figure 2 is a flow chart of a method for adding and removing duplicate items provided according to Embodiment 2 of the present invention;

[0028] Figure 3 is a structural schematic diagram of a data processing device provided according to Embodiment 3 of the present invention;

[0029] Figure 4 It is a block diagram of an electronic device provided according to a fourth embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment 1

[0033] Figure 1 A flowchart of a data processing method is provided for the first embodiment of the present invention. This embodiment is applicable to the case where the result of set function processing is determined by the data processing method. The method can be executed by a data processing device. The data processing device can be implemented in the form of hardware and / or software. The data processing device can be configured in an electronic device. The electronic device can be a computer with data processing capabilities and a database system. Figure 1 As shown, the method includes:

[0034] S110: Determine statements to be processed, where the statements to be processed include structured query statements of set functions with set marks, where the set marks include marks for removing duplicate values ​​in parameters of the set function.

[0035] In this embodiment, the statement to be processed can be understood as a structured query statement for managing and operating a database, which includes a set function and a set mark, and can realize the statement for removing duplicate values ​​from data in the database and completing the set function operation. Structured query statements are a standardized programming language for managing and operating relational database management systems, and can be widely used to create, query, update and manage data in relational databases. Set functions can be understood as aggregate functions in database languages, which are used to perform calculation operations on a set of data in a database and return calculation results. Set functions can include summing, averaging, maximum and / or minimum values, etc. Setting marks can be understood as marks for performing calculations on a set of data input into a set function, and can perform operations to remove duplicate values ​​from a set of data and return unique different values.

[0036] Specifically, first determine the statement to be processed that performs an operation on a set of data in the database, for example, perform an operation on a table, and the statement to be processed can be a structured query statement containing a set function operation. The set of data contained in the statement to be processed can be used as a parameter of the set function. In order to remove duplicate values ​​when performing a set function operation on the parameters of the set function, a setting mark for removing duplicate values ​​can be added to the set function.

[0037] Exemplarily, set functions may include SUM, AVG, COUNT, MAX, MIN, etc., which can all use the setting mark DISTINCT. The main purpose of DISTINCT is to exclude duplicate values ​​in the set function parameters when performing set function calculations, and only calculate based on unique values. The statement to be processed may be to perform calculations on table T1 (C1, C2). For example, the statement to be processed may be SELECT SUM (DISTINCT C2) FROM T1 GROUP BY C1. In the statement to be processed, the set function is SUM, and the summation operation is performed on the parameters of the set function. The meaning of the statement to be processed is to group the data of the T1 table according to the value of C1, deduplicate the C2 values ​​in each group, and then sum all the deduplicated C2 values.

[0038] S120. Obtain two adjacent grouping items in the data source corresponding to the statement to be processed and the deduplication items corresponding to each of the grouping items, wherein the data in the data source is ordered, and the data source includes multiple grouping items and the deduplication items corresponding to each grouping item.

[0039] In this embodiment, the grouping item can be understood as data used to group the data source, and the grouping item can be a data or a group of data. The deduplication item can be used to remove duplicate values ​​from the data determined by the grouping item. The data source can be understood as a data set that can be retrieved by the set function in the database, and the data set can be an ordered table or a group of ordered data.

[0040] Specifically, the data source corresponding to the statement to be processed is obtained. The data source can be an ordered table in the database, which includes grouping items and deduplication items. Each grouping item has its corresponding deduplication item. When executing the statement to be processed, for the data determined by the grouping item, a flag can be set to perform the operation of removing duplicate values ​​according to the deduplication item.

[0041] For example, if the data source is an ordered table T1(C1,C2), an operation is performed on the table T1(C1,C2) through the pending statement SELECT SUM(DISTINCT C2) FROM T1 GROUP BY C1. The C1 item in the table T1(C1,C2) can be used as a grouping item, and the C2 item can be used as a deduplication item. DISTINCT can perform a deduplication operation on the C2 item after it is grouped by the C1 item.

[0042] S130. Based on the comparison results of the obtained grouping items and the obtained deduplication items, determine whether to add the deduplication items corresponding to the grouping items with a higher order in the two adjacent grouping items to the determination of the set function result. If so, add the deduplication items corresponding to the grouping items with a higher order in the two adjacent grouping items to the determination of the set function result.

[0043] In this embodiment, the set function result can be understood as a unique value output after the set function operation, which can be obtained by performing a set function operation on the data after the duplicate items are removed.

[0044] Specifically, for the data in the data source, first determine whether it is the same group of data based on the grouping items. If so, compare each group of data based on the deduplication items to complete the deduplication operation; otherwise, do not perform the deduplication operation. The deduplication operation can be to determine whether the deduplication items of two adjacent rows of data in a group of data that has been determined to be deduplicated based on the grouping items are the same. If they are the same, the deduplication items with a later order in the two rows of data are added to the determination of the set function result, and the deduplication items with a later order continue to be compared with the next adjacent row of data; if they are not the same, the deduplication items with a earlier order in the two rows of data are added to the determination of the set function result, and the deduplication items with a later order continue to be compared with the next adjacent row of data.

[0045] For example, for an ordered table T1(C1, C2), when executing the SELECT SUM(DISTINCT C2) FROM T1GROUP BY C1 statement. You can first take out two adjacent rows of data in table T1, record them as rows N1 and N2. Then, according to the grouping item C1, compare and determine whether rows N1 and N2 are the same group of data. If the C1 items of rows N1 and N2 are the same, the two rows are determined to be in the same group. Finally, according to the set function to remove duplicate items C2, compare and determine whether the C2 items of rows N1 and N2 are the same. If they are the same, only the C2 value of the data in row N2 is added to the SUM result, and row N2 is used as the N1 row for the next judgment to continue comparison; if they are not the same, the C2 value of the data in row N1 is added to the SUM result, and row N2 is used as the N1 row for the next judgment to continue comparison.

[0046] S140. Continue to select the next grouping item and the deduplication item corresponding to the grouping item from the data source, and use the grouping item with the later order among the two adjacent grouping items and the next grouping item as two new adjacent grouping items, and return to continue to determine whether to pass the set function processing until the last grouping item in the data source is obtained.

[0047] Specifically, the grouping item and its corresponding deduplication item in the two rows of data in S130 are taken as one row of data, and the next grouping item and its corresponding deduplication item adjacent to the row of data in the data source are selected as another row of data. The two rows of data are compared according to the grouping item and the corresponding deduplication item, and the method added to the determination of the set function result is determined until the data in the data source is compared.

[0048] Exemplarily, the next N2 rows of data adjacent to the determined N1 row of data are selected, and the grouping items of the N1 row and the N2 row and their corresponding deduplication items are continuously compared until all the data in the data source T1 (C1, C2) table are compared.

[0049] S150: Add the deduplication item corresponding to the last grouping item to the determination of the set function result, to obtain the set function result of the grouping corresponding to the grouping item.

[0050] Specifically, after the data in the data source is compared, the deduplication item corresponding to the grouping item of the last row of data is added to the set function result corresponding to the grouping item to obtain the set function result of the group corresponding to the last grouping item. By executing the statement to be processed on the data in the data source, the set function result of the group corresponding to each grouping item is obtained, and the processing of the data in the data source is completed.

[0051] Exemplarily, after all the data in the data source table T1 (C1, C2) are compared, the deduplication item C2 corresponding to the grouping item C1 of the last row of data in the table is added to the set function SUM of the grouping corresponding to C1, the operation of the set function SUM is completed, the result of the last group is output, and the data processing operation is completed.

[0052] The technical solution of the embodiment of the present invention is to determine the statement to be processed, and obtain the two adjacent grouping items and the deduplication items corresponding to each of the grouping items in the data source corresponding to the statement to be processed. Then, based on the comparison results of the obtained grouping items and the obtained deduplication items, determine whether to add the deduplication items corresponding to the grouping items with a higher order in the two adjacent grouping items to the determination of the set function result. If so, add the deduplication items corresponding to the grouping items with a higher order in the two adjacent grouping items to the determination of the set function result. Continue to select the next grouping item and the deduplication items corresponding to the grouping item from the data source, and use the grouping item with a lower order in the two adjacent grouping items and the next grouping item as the two new adjacent grouping items, and return to continue to determine whether to process through the set function until the last grouping item in the data source is obtained. Finally, add the deduplication items corresponding to the last grouping item to the determination of the set function result to obtain the set function result of the grouping corresponding to the grouping item. By comparing the grouping items and the deduplication items in the data source, a method for determining the processing results of the set function is obtained, the calculation process is simplified, and the function of quickly calculating the results of the set function is realized.

[0053] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0054] In one embodiment, determining the statement to be processed includes:

[0055] Get the statement to be executed;

[0056] Determine identification information corresponding to the statement to be executed;

[0057] When the identification information is a set value, it is determined that the statement to be executed is a statement to be processed.

[0058] In this embodiment, the statement to be executed can be understood as a structured query statement for managing and operating a database. The statement to be executed can complete the query, insertion, update and deletion of data in the database, and the statement to be processed can be selected from the statements to be executed. The identification information can be understood as the setting information used to determine whether the statement to be executed can be used as the statement to be processed. The identification information can be a mark that satisfies the optimization scenario. The optimization scenario can be understood as the statement to be executed contains a set function, and the set function contains a statement that removes duplicate value marks. At this time, the statement to be executed can be optimized. The set value can be understood as a value set to determine whether the statement to be executed meets the optimization scenario.

[0059] Specifically, the statements to be executed that need to be operated are collected, and the identification information of each statement to be executed is determined by determining the parameters and grouping items of the set function in the statement to be executed. When the identification information of the statement to be executed is a set value, it means that the set function parameters contain the set mark and the grouping items are in order, and the statement to be executed is set as a statement to be processed.

[0060] Exemplarily, when the statement to be executed satisfies the set function parameter containing the setting mark DISTINCT and the grouping items are ordered, the setting mark of the statement to be executed is recorded as the setting value P1. When executing the statement to be executed, first determine whether the setting mark of the statement to be executed is P1, and if so, determine the statement to be executed as a statement to be processed.

[0061] In one embodiment, the data source includes a table, the grouping items and the deduplication items are different columns in the table, and adjacent deduplication items are two adjacent rows of data in the table.

[0062] In this embodiment, the table can be understood as a data set stored in a table format in a database.

[0063] Exemplarily, the data source may be table T1 (C1, C2), wherein column C1 is a grouping item, column C2 is a deduplication item, the data in columns C1 and C2 correspond to each other, and adjacent deduplication items are two adjacent rows of data in table T1 (C1, C2).

[0064] In one embodiment, before determining the statement to be processed, the method further includes:

[0065] Determine all grouping items in the data source;

[0066] Determining whether the parameter of the set function in the statement to be executed has the set mark;

[0067] If present, the grouping term is combined with the parameter of the set function;

[0068] Determine whether the combined data is in order;

[0069] If there is an order, the identification information of the statement to be executed is set to a set value.

[0070] Specifically, for a statement to be executed, determine all the grouping items in the statement to be executed and record them. Determine whether the parameters of the set function in the statement to be executed have a set mark, and combine the parameters of the set function of the statement to be executed with the set mark with the grouping item. Determine whether the combined data is ordered. If the data source is a table, it can be determined whether it is ordered based on the index of the table. If the combined data is ordered, the identification information of the statement to be executed can be set to the set value, indicating that the statement to be executed can be used as a statement to be processed for data processing.

[0071] Exemplarily, all grouping items of a statement to be executed, i.e., GROUP BY items, are collected in order, recorded as X1..Xn. It is determined whether the parameter of the set function in the statement to be executed is marked with DISTINCT. If it is not marked with DISTINCT, the processing is exited, otherwise the parameter Y corresponding to the set function in the statement to be executed is taken out. The parameter Y and the collected grouping items are combined to form a structure in the form of (X1..Xn,Y) sequence. It is determined whether the data corresponding to the (X1..Xn,Y) structure after the combination is ordered. If it is not ordered, the processing is exited, otherwise the identification information of the statement to be executed is set to the set value. When processing the statement to be executed, the statement to be executed whose identification information is the set value is set as the statement to be processed, and the subsequent analysis and processing continues.

[0072] Embodiment 2

[0073] Figure 2 This is a flowchart of a method for adding and determining duplicate removal items provided in the second embodiment of the present invention. This embodiment is based on the method for adding duplicate removal items to the determination of the set function result in the above embodiment. Figure 2 As shown, the method includes:

[0074] S210, determine whether the values ​​of the two obtained grouping items are the same, if so, execute S220; otherwise, execute S230.

[0075] Specifically, for the data in the data source, firstly, two adjacent grouping items and their corresponding deduplication items are selected to determine whether the values ​​of the two grouping items are the same, and then determine whether they are the same group of data based on the grouping item determination result.

[0076] For example, for the data in the data source, two adjacent rows of data are first compared according to the grouping item (X1..Xn) as the key value. For example, row N1 and row N2 are two adjacent rows of data, wherein row N1 is regarded as the current row, and row N2 is the next row of data of the current row, and it is determined whether the grouping item values ​​of row N1 and row N2 are the same.

[0077] S220. If the values ​​of the two obtained grouping items are the same, determine whether the two obtained deduplication items are the same, and determine whether to add the deduplication item corresponding to the grouping item with the earlier order in the two adjacent grouping items to the determination of the set function processing result based on the comparison result of whether the deduplication items are the same.

[0078] Specifically, if the values ​​of the two obtained grouping items are the same, determine whether the two obtained deduplication items are the same. If the two deduplication items are the same, retain the grouping item with the later order in the two grouping items and the corresponding deduplication item, and add the deduplication item to the determination of the set function result; if the two deduplication items are not the same, add the deduplication item corresponding to the grouping item with the earlier order in the two grouping items to the determination of the set function result, and the deduplication item with the later order continues to be compared with the next adjacent row of data.

[0079] Exemplarily, if the grouping items (X1..Xn) of rows N1 and N2 are the same, it means they are the same group, then continue to compare the deduplication item (Y) column: if the deduplication item Y of rows N1 and N2 is the same, then only the data of row N2 is retained, and the data of row N2 is used as the current row; if the deduplication item Y of rows N1 and N2 is different, then the value of the deduplication item (Y) column of the data of row N1 is added to the result calculation according to the set function type, and row N2 is used as the current row.

[0080] Optionally, the step of determining whether to add the deduplication item corresponding to the grouping item with a preceding order in the two adjacent grouping items to the result of the set function processing according to the comparison result of whether the deduplication items are the same includes:

[0081] If the values ​​of the two deduplication items obtained are the same, the grouping item with the later order among the two adjacent grouping items and the corresponding deduplication items are retained; otherwise, it is determined that the deduplication items corresponding to the grouping item with the earlier order among the two adjacent grouping items need to be added to the determination of the set function processing result.

[0082] Specifically, if two deduplication items are the same, the grouping item with a later order among the two grouping items and the corresponding deduplication items are retained, and the deduplication items are added to the set function result determination of the grouping corresponding to the grouping item; if the two deduplication items are different, the deduplication items corresponding to the grouping item with a earlier order among the two grouping items are added to the set function result determination of the grouping corresponding to the grouping item, and the deduplication items with a later order continue to be compared with the next adjacent row of data.

[0083] Exemplarily, if the deduplication item Y of row N1 and row N2 is the same, only the data of row N2 is retained, and the deduplication item Y corresponding to row N2 is added to the result calculation according to the set function type, and row N2 is used as the current row; if the deduplication item Y of row N1 and row N2 is different, the value of the deduplication item (Y) column of row N1 data is added to the result calculation according to the set function type, row N2 is used as the current row, and comparison with the next adjacent row of data continues.

[0084] S230. If the two grouping items obtained are different, it is determined that the deduplication item corresponding to the grouping item with a higher order among the two adjacent grouping items needs to be added to the determination of the set function processing result, and the set function result of the grouping corresponding to the grouping item with a higher order among the two adjacent grouping items is determined.

[0085] Specifically, if the values ​​of the two obtained grouping items are different, the deduplication item corresponding to the grouping item with a higher order in the two adjacent grouping items is added to the determination of the set function result, and the deduplication item with a lower order is compared with the next adjacent row of data. In this way, the set function result of the group corresponding to the grouping item with a higher order in the two adjacent grouping items can be determined.

[0086] For example, if the grouping items (X1..Xn) of row N1 and row N2 are different, it means they are different groups. Then, the value of the duplicate-free item (Y) column of row N1 data is added to the result calculation according to the set function type, and the current calculation result is output, that is, the processing result of the grouping corresponding to row N1 after the set function is processed. After the output, the calculation result is cleared to facilitate the subsequent calculation value, and row N2 is used as the current row for the next stage of judgment.

[0087] The technical solution of the embodiment of the present invention determines whether the values ​​of the two obtained grouping items are the same; if the values ​​of the two obtained grouping items are the same, then determine whether the two obtained deduplication items are the same, and determine whether to add the deduplication items corresponding to the grouping items with a higher order in the two adjacent grouping items to the determination of the set function processing result based on the comparison result of whether the deduplication items are the same; if the two obtained grouping items are different, then determine whether to add the deduplication items corresponding to the grouping items with a higher order in the two adjacent grouping items to the determination of the set function processing result, and determine the set function result of the grouping corresponding to the grouping items with a higher order in the two adjacent grouping items. The method of determining to add the deduplication items corresponding to the two adjacent grouping items to the determination of the set function result based on the comparison results of the grouping items and the deduplication items is refined, and the determination of the set function processing result is completed.

[0088] Embodiment 3

[0089] Figure 3 This is a schematic diagram of the structure of a data processing device provided in Embodiment 3 of the present invention. Figure 3As shown, the device comprises:

[0090] A first determination module 310 is used to determine a statement to be processed, wherein the statement to be processed includes a structured query statement of a set function with a set mark, wherein the set mark includes a mark for removing duplicate values ​​in a parameter of the set function;

[0091] An acquisition module 320 is used to acquire two adjacent grouping items and deduplication items corresponding to each of the grouping items in the data source corresponding to the statement to be processed, wherein the data in the data source is ordered, and the data source includes multiple grouping items and deduplication items corresponding to each of the grouping items;

[0092] The second determination module 330 is used to determine whether to add the deduplication item corresponding to the grouping item with a higher order in the two adjacent grouping items to the determination of the set function result based on the comparison results of the obtained grouping items and the obtained deduplication item, and if so, add the deduplication item corresponding to the grouping item with a higher order in the two adjacent grouping items to the determination of the set function result;

[0093] Return module 340, used to continue to select the next grouping item and the deduplication item corresponding to the grouping item from the data source, and use the grouping item with the later order in the two adjacent grouping items and the next grouping item as two new adjacent grouping items, and return to continue to determine whether to pass the set function processing until the last grouping item in the data source is obtained;

[0094] The adding module 350 is used to add the deduplication item corresponding to the last grouping item to the determination of the set function result, so as to obtain the set function result of the grouping corresponding to the grouping item.

[0095] The data loading device provided by the embodiment of the present invention first determines the statement to be processed through the first determination module, and obtains the two adjacent grouping items and the deduplication items corresponding to each of the grouping items in the data source corresponding to the statement to be processed through the acquisition module. Then, the second determination module determines whether to add the deduplication items corresponding to the grouping items with a higher order in the two adjacent grouping items to the determination of the set function result based on the comparison results of the obtained grouping items and the obtained deduplication items. If so, the deduplication items corresponding to the grouping items with a higher order in the two adjacent grouping items are added to the determination of the set function result. The return module continues to select the next grouping item and the deduplication items corresponding to the grouping item from the data source, and the grouping item with a lower order in the two adjacent grouping items and the next grouping item are used as the new two adjacent grouping items, and returns to continue to determine whether to process through the set function until the last grouping item in the data source is obtained. Finally, the deduplication items corresponding to the last grouping item are added to the determination of the set function result through the adding module, and the set function result of the group corresponding to the grouping item is obtained. Through the cooperation between modules, the grouping items and duplicate removal items in the data source are compared, and a method for determining the processing results of the set function is obtained, which simplifies the calculation process and realizes the function of quickly calculating the results of the set function.

[0096] In one embodiment, the second determination module 330 includes:

[0097] A first determining unit, used to determine whether the values ​​of the two acquired grouping items are the same;

[0098] A second determination unit is used to determine whether the two obtained deduplication items are the same if the values ​​of the two obtained grouping items are the same, and determine whether to add the deduplication item corresponding to the grouping item with a front order in the two adjacent grouping items to the determination of the set function processing result according to the comparison result of whether the deduplication items are the same;

[0099] The third determination unit is used to determine the need to add the deduplication item corresponding to the grouping item with a preceding order among the two adjacent grouping items to the determination of the set function processing result if the two obtained grouping items are different, and determine the set function result of the grouping corresponding to the grouping item with a preceding order among the two adjacent grouping items.

[0100] In one embodiment, the second determining unit is specifically configured to:

[0101] If the values ​​of the two deduplication items obtained are the same, the grouping item with the later order among the two adjacent grouping items and the corresponding deduplication items are retained; otherwise, it is determined that the deduplication items corresponding to the grouping item with the earlier order among the two adjacent grouping items need to be added to the determination of the set function processing result.

[0102] In one embodiment, the first determining module 310 is specifically configured to:

[0103] Get the statement to be executed;

[0104] Determine identification information corresponding to the statement to be executed;

[0105] When the identification information is a set value, it is determined that the statement to be executed is a statement to be processed.

[0106] In one embodiment, the data source includes a table, the grouping items and the deduplication items are different columns in the table, and adjacent deduplication items are two adjacent rows of data in the table.

[0107] In one embodiment, the data processing device further includes a third determining module, specifically configured to:

[0108] Determine all grouping items in the data source;

[0109] Determining whether the parameter of the set function in the statement to be executed has the set mark;

[0110] If present, the grouping term is combined with the parameter of the set function;

[0111] Determine whether the combined data is in order;

[0112] If there is an order, the identification information of the statement to be executed is set to a set value.

[0113] The data processing device provided in the embodiment of the present invention can execute the data processing method provided in any embodiment of the present invention, completes the processing of data through the mutual cooperation and collaboration between the modules, and has the functional modules and beneficial effects corresponding to the execution method.

[0114] Embodiment 4

[0115] According to an embodiment of the present invention, the present invention also provides an electronic device, a computer-readable storage medium and a computer program product.

[0116] Figure 4It is a block diagram of an electronic device provided according to Embodiment 4 of the present invention, and the electronic device can implement the data processing method described in the embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0117] like Figure 4 As shown, the electronic device 410 includes at least one processor 411, and a memory connected to the at least one processor 411 in communication, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or the computer program loaded from the storage unit 418 to the random access memory (RAM) 413. In the RAM 413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, the ROM 412, and the RAM 413 are connected to each other via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0118] A number of components in the electronic device are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, an optical disk, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0119] The processor 411 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 411 performs the various methods and processes described above, such as a data processing method.

[0120] In some embodiments, the data processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded into the RAM 413 and executed by the processor 411, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the processor 411 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).

[0121] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0123] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0125] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0126] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0127] In some embodiments, the computer program product includes a computer program, and when the computer program is executed by a processor, it implements the data processing method provided by the embodiment of the present invention.

[0128] The technical solution of the embodiment of the present invention is to determine the statement to be processed through a data processing method, device, electronic device, storage medium and program product, and obtain two adjacent grouping items and deduplication items corresponding to each grouping item in the data source corresponding to the statement to be processed. Then, based on the comparison results of the obtained grouping items and the obtained deduplication items, it is determined whether to add the deduplication items corresponding to the grouping items with a higher order in the two adjacent grouping items to the determination of the set function result. If so, the deduplication items corresponding to the grouping items with a higher order in the two adjacent grouping items are added to the determination of the set function result. Continue to select the next grouping item and the deduplication items corresponding to the grouping item from the data source, and use the grouping item with a lower order in the two adjacent grouping items and the next grouping item as the new two adjacent grouping items, and return to continue to determine whether to process through the set function until the last grouping item in the data source is obtained. Finally, the deduplication items corresponding to the last grouping item are added to the determination of the set function result to obtain the set function result of the group corresponding to the grouping item. By comparing the grouping items and the deduplication items in the data source, a method for determining the processing results of the set function is obtained, the calculation process is simplified, and the function of quickly calculating the results of the set function is realized.

[0129] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0130] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: Determining a statement to be processed, the statement to be processed comprising a structured query statement of a set function with a set mark, the set mark comprising a mark for removing duplicate values ​​in parameters of the set function; Obtaining two adjacent grouping items and duplicate removal items corresponding to each of the grouping items in a data source corresponding to the statement to be processed, wherein the data in the data source is ordered, and the data source includes multiple grouping items and duplicate removal items corresponding to each of the grouping items; Based on the comparison results of the obtained grouping items and the obtained deduplication items, respectively, determine whether to add the deduplication item corresponding to the grouping item with a higher order in the two adjacent grouping items to the determination of the set function result; if so, add the deduplication item corresponding to the grouping item with a higher order in the two adjacent grouping items to the determination of the set function result; Continue to select the next grouping item and the deduplication item corresponding to the grouping item from the data source, and use the grouping item with the later order among the two adjacent grouping items and the next grouping item as the new two adjacent grouping items, and return to continue to determine whether to pass the set function processing until the last grouping item in the data source is obtained; The deduplication item corresponding to the last grouping item is added to the determination of the set function result to obtain the set function result of the group corresponding to the grouping item.

2. The method according to claim 1, characterized in that The determining whether to add the deduplication item corresponding to the grouping item with a higher order in the order of the two adjacent grouping items to the determination of the set function result based on the comparison results of the obtained grouping items and the obtained deduplication items respectively includes: Determine whether the values ​​of the two obtained group items are the same; If the values ​​of the two obtained grouping items are the same, determine whether the two obtained deduplication items are the same, and determine whether to add the deduplication item corresponding to the grouping item with the earlier order in the two adjacent grouping items to the determination of the set function processing result based on the comparison result of whether the deduplication items are the same; If the two grouping items obtained are different, it is determined that the deduplication items corresponding to the grouping item with the earlier order in the two adjacent grouping items need to be added to the determination of the set function processing result, and the set function result of the group corresponding to the grouping item with the earlier order in the two adjacent grouping items is determined.

3. The method according to claim 2, characterized in that The determining, based on the comparison result of whether the duplicate-free items are the same, whether to add the grouping item with the earlier order in the two adjacent grouping items to the result of the set function processing comprises: If the values ​​of the two obtained deduplication items are the same, the grouping item with the later order among the two adjacent grouping items and the corresponding deduplication item are retained; otherwise, it is determined that the deduplication item corresponding to the grouping item with the earlier order among the two adjacent grouping items needs to be added to the determination of the set function processing result.

4. The method according to claim 1, wherein Determining the statement to be processed includes: Get the statement to be executed; Determining identification information corresponding to the statement to be executed; In a case where the identification information is a set value, it is determined that the statement to be executed is a statement to be processed.

5. The method according to any one of claims 1 to 4, characterized in that: The data source includes a table, the grouping items and the deduplication items are different columns in the table, and adjacent deduplication items are two adjacent rows of data in the table.

6. The method according to claim 1, characterized in that Before determining the statements to be processed, it also includes: Determine all grouping items in the data source; determining whether the parameter of the set function in the statement to be executed has the set mark; If present, then the grouping term and the parameter of the set function are combined; Determine whether the data is in order after combination; If there is an order, the identification information of the statement to be executed is set to a set value.

7. A data processing device, characterized in that: include: A first determining module is configured to determine a statement to be processed, wherein the statement to be processed includes a structured query statement of a set function with a set mark, wherein the set mark includes a mark for removing duplicate values ​​in parameters of the set function; an acquisition module, configured to acquire two adjacent grouping items and a deduplication item corresponding to each of the grouping items from a data source corresponding to the statement to be processed, wherein the data in the data source is ordered and the data source includes a plurality of grouping items and a deduplication item corresponding to each of the grouping items; A second determination module is used to determine whether to add the deduplication item corresponding to the grouping item with a higher order in the two adjacent grouping items to the determination of the set function result based on the comparison results of the obtained grouping items and the obtained deduplication item, and if so, to add the deduplication item corresponding to the grouping item with a higher order in the two adjacent grouping items to the determination of the set function result; a return module for continuing to select the next grouping item and the deduplication item corresponding to the grouping item from the data source, and taking the grouping item with the later order among the two adjacent grouping items and the next grouping item as the new two adjacent grouping items, and returning to continue to determine whether to pass the set function processing until the last grouping item in the data source is obtained; An adding module is used to add the deduplication item corresponding to the last grouping item to the determination of the set function result, so as to obtain the set function result of the group corresponding to the grouping item.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the data processing method according to any one of claims 1 to 6.

Images