A data processing method and device, electronic equipment, storage medium and program product

Abstract

This invention discloses a data processing method, apparatus, electronic device, storage medium, and program product. The implementation includes: determining a statement to be processed; obtaining two adjacent grouping items and their corresponding deduplication items from the data source corresponding to the statement to be processed; determining whether to add the deduplication item corresponding to the earlier grouping item in the adjacent grouping items to the set function result based on the comparison results of the obtained grouping items and deduplication items; continuing to select the next grouping item and its corresponding deduplication item from the data source, returning to continue determining whether to process using the set function, until the last grouping item in the data source is obtained; and obtaining the set function result for the grouping item. This invention, through the comparison of grouping items and deduplication items, provides a method for determining the processing result of the set function, simplifying the calculation process and realizing the function of quickly calculating the set function result.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] When manipulating a database using structured query statements, deduplication is often required for the aggregate function parameters in the statement. Generally, a data structure is used to store different rows of the dataset in different locations (e.g., different rows are stored in different hash slots if using a hash table). Since deduplication results within groups are needed, another data structure (such as a hash table) is required to store the different aggregate function values ​​for the same group. Finally, all the data stored in the data structure containing the different aggregate function values ​​is aggregated according to the different groups, and the final result is calculated.

[0003] As can be seen, the above method is quite complex, requiring the creation of additional data structures and caching all data in the table to calculate the grouped set function values. The calculation process is complex and consumes a large amount of memory, resulting in poor performance. Summary of the Invention

[0004] This invention provides a data processing method, apparatus, electronic device, storage medium, and program product to determine the processing result of a set function, thereby realizing the function of quickly calculating the result of the set function.

[0005] According to one aspect of the present invention, a data processing method is provided, comprising:

[0006] Identify the statement to be processed, which includes a structured query statement with a set function that has a set flag, the set flag including a flag to remove duplicate values ​​from the parameters of the set function;

[0007] Obtain two adjacent grouping items and the deduplication item corresponding to each grouping item from the data source corresponding to the statement to be processed. The data source is ordered and includes multiple grouping items and the deduplication item corresponding to each grouping item.

[0008] Based on the comparison results of the acquired grouping items and the acquired deduplication items, determine whether to add the deduplication item corresponding to the earlier sequential grouping item in the two adjacent grouping items to the determination of the set function result. If so, add the deduplication item corresponding to the earlier sequential grouping item in the two adjacent grouping items to the determination of the set function result.

[0009] Continue selecting the next group item and the deduplication item corresponding to the group item from the data source, and take the group item that comes later in the order of the two adjacent group items and the next group item as the new two adjacent group items, and return to continue to determine whether to process through the set function, until the last group item in the data source is obtained;

[0010] The deduplication item corresponding to the last grouping item is added to the determination of the set function result to obtain the set function result of the grouping item corresponding to the grouping item.

[0011] According to another aspect of the present invention, a data processing apparatus is provided, comprising:

[0012] The first determining module is used to determine the statement to be processed, the statement to be processed including a structured query statement of a set function with a set flag, the set flag including a flag to remove duplicate values ​​in the parameters of the set function;

[0013] The acquisition module is used to acquire two adjacent grouping items and the deduplication item corresponding to each grouping item in the data source corresponding to the statement to be processed. The data source is ordered and includes multiple grouping items and the deduplication item corresponding to each grouping item.

[0014] The second determining module is used to determine whether to add the deduplication item corresponding to the earlier group item in the two adjacent group items to the determination of the set function result based on the comparison results of the acquired group items and the acquired deduplication items respectively. If so, the deduplication item corresponding to the earlier group item in the two adjacent group items is added to the determination of the set function result.

[0015] The return module is used to continue selecting the next group item and the deduplication item corresponding to the group item from the data source, and to take the group item that comes later in the order of the two adjacent group items and the next group item as the new two adjacent group items, and return to continue to determine whether to process through the set function, until the last group item in the data source is obtained;

[0016] An addition module is used to add the deduplication item corresponding to the last grouping item to the determination of the set function result, thereby obtaining the set function result of the grouping item corresponding to the grouping.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] At least one processor; and

[0019] A memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data processing method according to any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the data processing method described in any embodiment of the present invention.

[0022] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the data processing method described in any embodiment of the present invention.

[0023] The technical solution of this invention involves determining a statement to be processed and obtaining two adjacent grouping items and their corresponding deduplication items from the data source corresponding to the statement. Then, based on the comparison results of the obtained grouping items and deduplication items, it is determined whether to add the deduplication item corresponding to the earlier-ordered grouping item in the two adjacent grouping items to the set function result. If so, the deduplication item corresponding to the earlier-ordered grouping item in the two adjacent grouping items is added to the set function result. The process continues by selecting the next grouping item and its corresponding deduplication item from the data source, and using the later-ordered grouping item in the two adjacent grouping items and the next grouping item as new adjacent grouping items. The process returns to determine whether to process using the set function until the last grouping item in the data source is obtained. Finally, the deduplication item corresponding to the last grouping item is added to the set function result to obtain the set function result for the grouping item. By comparing the grouping and deduplication items in the data source, a method for determining the processing result of the set function was obtained, which simplified the calculation process and enabled the function of quickly calculating the set function result.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a data processing method provided according to Embodiment 1 of the present invention;

[0027] Figure 2 This is a flowchart of a method for adding duplicate items according to Embodiment 2 of the present invention;

[0028] Figure 3 This is a schematic diagram of the structure of a data processing device according to Embodiment 3 of the present invention;

[0029] Figure 4 This is a block diagram of an electronic device provided according to Embodiment 4 of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1

[0033] Figure 1 This is a flowchart of a data processing method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where the result of set function processing is determined through a data processing method. This method can be executed by a data processing device, which can be implemented in hardware and / or software. The data processing device can be configured in an electronic device, which can be a computer with data processing capabilities and a database system. Figure 1 As shown, the method includes:

[0034] S110. Determine the statement to be processed, the statement to be processed includes a structured query statement of a set function with a set flag, the set flag including a flag to remove duplicate values ​​in the parameters of the set function.

[0035] In this embodiment, the statement to be processed can be understood as a structured query statement for managing and operating a database, containing aggregate functions and setting flags. It can remove duplicate values ​​from data in the database and perform aggregate function operations. Structured query statements are a standardized programming language used to manage and operate relational database management systems, and can be widely used to create, query, update, and manage data in relational databases. Aggregate functions can be understood as aggregate functions in database languages, used to perform calculations on a set of data in the database and return the results. Aggregate functions can include summation, averaging, finding the maximum value, and / or finding the minimum value, etc. Setting flags can be understood as flags used to perform calculations on a set of data input to the aggregate function, enabling the removal of duplicate values ​​from a set of data and returning a unique distinct value.

[0036] Specifically, first, determine the statement to be processed that performs operations on a set of data in the database, such as operations on a table. The statement to be processed can be a structured query statement containing aggregate function operations. The set of data contained in the statement to be processed can be used as parameters of the aggregate function. To implement the operation of removing duplicate values ​​when performing aggregate function operations on the parameters of the aggregate function, a setting flag for implementing the duplicate value removal operation can be added to the aggregate function.

[0037] For example, aggregate functions can include SUM, AVG, COUNT, MAX, MIN, etc., all of which can use the DISTINCT flag. The main purpose of DISTINCT is to exclude duplicate values ​​in the aggregate function parameters during calculation, performing calculations only based on unique values. The statement to be processed can be a calculation performed on table T1(C1,C2). For example, the statement to be processed can be SELECT SUM(DISTINCT C2) FROM T1 GROUP BY C1. In this statement, the aggregate function is SUM, which performs a summation operation on the parameters of the aggregate function. The meaning of this statement is to group the data in table T1 according to the values ​​of C1, then deduplicate the values ​​of C2 in each group, and finally sum all the deduplicated values ​​of C2.

[0038] S120. Obtain two adjacent grouping items and the deduplication item corresponding to each grouping item from the data source corresponding to the statement to be processed. The data source is ordered and includes multiple grouping items and the deduplication item corresponding to each grouping item.

[0039] In this embodiment, a grouping item can be understood as data used to group data sources; a grouping item can be a single data item or a group of data items. A deduplication item can be used to remove duplicate values ​​from the data determined by the grouping items. A data source can be understood as a dataset that can be retrieved from a database using set functions; a dataset can be an ordered table or a set of ordered data items.

[0040] Specifically, the data source corresponding to the statement to be processed is obtained. The data source can be an ordered table in the database, which includes grouping items and deduplication items. Each grouping item has its corresponding deduplication item. When executing the statement to be processed, for the data determined by the grouping items, a flag is set to perform the operation of removing duplicate values ​​according to the deduplication item.

[0041] For example, if the data source is an ordered table T1(C1,C2), the statement `SELECT SUM(DISTINCT C2) FROM T1 GROUP BY C1` is used to perform operations on table T1(C1,C2). In table T1(C1,C2), item C1 can be used as a grouping item, and item C2 can be used as a deduplication item. `DISTINCT` can perform deduplication on items C2 grouped based on item C1.

[0042] S130. Based on the comparison results of the acquired grouping items and the acquired deduplication items, determine whether to add the deduplication item corresponding to the earlier grouping item in the two adjacent grouping items to the determination of the set function result. If so, add the deduplication item corresponding to the earlier grouping item in the two adjacent grouping items to the determination of the set function result.

[0043] In this embodiment, the result of the set function can be understood as the unique value output after the set function operation, which can be obtained by performing set function operation on the data after deduplication.

[0044] Specifically, for the data in the data source, the process first determines whether the data belongs to the same group based on the grouping items. If so, each group is compared based on the deduplication items to perform deduplication; otherwise, no deduplication is performed. The deduplication operation can be performed by taking two adjacent rows of data from a group determined by the grouping items and determining if their deduplication items are the same. If they are the same, the later deduplication item from these two rows is added to the result of the set function, and the later deduplication item continues to be compared with the next adjacent row. If they are different, the earlier deduplication item from these two rows is added to the result of the set function, and the later deduplication item continues to be compared with the next adjacent row.

[0045] For example, when executing the statement `SELECT SUM(DISTINCT C2) FROM T1GROUP BY C1` on an ordered table T1(C1, C2), two adjacent rows of data can be retrieved first, denoted as rows N1 and N2. Then, based on the grouping item C1, rows N1 and N2 are compared to determine if they belong to the same group. If the C1 items of rows N1 and N2 are the same, they are considered to belong to the same group. Finally, based on the deduplication item C2, rows N1 and N2 are compared to determine if their C2 items are the same. If they are the same, only the C2 value of row N2 is added to the SUM result, and row N2 is used as the next row to be compared. If they are different, the C2 value of row N1 is added to the SUM result, and row N2 is used as the next row to be compared.

[0046] S140. Continue to select the next group item and the deduplication item corresponding to the group item from the data source, and take the group item that comes later in the order of the two adjacent group items and the next group item as the two new adjacent group items, and return to continue to determine whether to process through the set function until the last group item in the data source is obtained.

[0047] Specifically, the later grouping item and its corresponding deduplicated item in the two rows of data in S130 are taken as one row of data. The next grouping item adjacent to this row of data in the data source and its corresponding deduplicated item are selected as the other row of data. These two rows of data are compared according to the grouping item and its corresponding deduplicated item, and the method for adding them to the set function result is determined, until all the data in the data source has been compared.

[0048] For example, select the next N2 row of data adjacent to the already determined N1 row of data, and continue to compare the grouping items of N1 and N2 rows and their corresponding deduplication items until all data in the data source table T1(C1,C2) has been compared.

[0049] S150. Add the deduplication item corresponding to the last grouping item to the determination of the set function result to obtain the set function result of the grouping item corresponding to the grouping.

[0050] Specifically, after comparing the data in the data source, the deduplication item corresponding to the grouping item of the last row of data is added to the set function result corresponding to that grouping item, thus obtaining the set function result of the group corresponding to the last grouping item. By executing the processing statement on the data in the data source, the set function result of the group corresponding to each grouping item is obtained, completing the processing of the data in the data source.

[0051] For example, after comparing all the data in the data source table T1(C1,C2), the duplicate item C2 corresponding to the grouping item C1 of the last row of data in the table is added to the set function SUM corresponding to C1, the set function SUM is performed, the result of the last group is output, and the data processing operation is completed.

[0052] The technical solution of this invention involves determining a statement to be processed and obtaining two adjacent grouping items and their corresponding deduplication items from the data source corresponding to the statement. Then, based on the comparison results of the obtained grouping items and deduplication items, it is determined whether to add the deduplication item corresponding to the earlier-ordered grouping item in the two adjacent grouping items to the set function result. If so, the deduplication item corresponding to the earlier-ordered grouping item in the two adjacent grouping items is added to the set function result. The process continues by selecting the next grouping item and its corresponding deduplication item from the data source, and using the later-ordered grouping item in the two adjacent grouping items and the next grouping item as new adjacent grouping items. The process returns to determine whether to process using the set function until the last grouping item in the data source is obtained. Finally, the deduplication item corresponding to the last grouping item is added to the set function result to obtain the set function result for the grouping item. By comparing the grouping and deduplication items in the data source, a method for determining the processing result of the set function was obtained, which simplified the calculation process and enabled the function of quickly calculating the set function result.

[0053] Based on the above embodiments, modified embodiments of the above embodiments are proposed. It should be noted that, in order to keep the description brief, only the differences from the above embodiments are described in the modified embodiments.

[0054] In one embodiment, determining the statement to be processed includes:

[0055] Get the statement to be executed;

[0056] Determine the identification information corresponding to the statement to be executed;

[0057] If the identification information is a set value, the statement to be executed is determined to be a statement to be processed.

[0058] In this embodiment, the statement to be executed can be understood as a structured query statement for managing and operating the database. The statement to be executed can perform queries, insertions, updates, and deletions of data in the database. A statement to be processed can be selected from the list of statements to be executed. The identification information can be understood as setting information used to determine whether a statement to be executed can be used as a statement to be processed. The identification information can be a marker that satisfies an optimization scenario. An optimization scenario can be understood as the statement to be executed containing a set function, and the set function containing a statement that removes duplicate value markers. In this case, the statement to be executed can be optimized for execution. The set value can be understood as a value set to determine whether the statement to be executed satisfies the optimization scenario.

[0059] Specifically, the system collects the statements to be executed that require processing. By determining the parameters and grouping items of the set function in each statement, the system identifies the statement's identifier. When the identifier of a statement is a set value, it indicates that the set function's parameters contain a set flag and that the grouping items are ordered; therefore, the statement is designated as a statement to be processed.

[0060] For example, if the parameters of the set function contain the setting flag DISTINCT and the grouping items are ordered, then the setting flag of the statement to be executed is recorded as the setting value P1. When executing the statement to be executed, it is first determined whether the setting flag of the statement to be executed is P1. If so, the statement to be executed is determined as a statement to be processed.

[0061] In one embodiment, the data source includes a table, the grouping items and the deduplication items are different columns in the table, and adjacent deduplication items are two adjacent rows of data in the table.

[0062] In this embodiment, a table can be understood as a dataset stored in a table format in a database.

[0063] For example, the data source can be table T1(C1,C2), where column C1 is the grouping item, column C2 is the deduplication item, the data in column C1 and column C2 correspond to each other, and adjacent deduplication items are the two adjacent rows of data in table T1(C1,C2).

[0064] In one embodiment, before determining the statement to be processed, the following is also included:

[0065] Identify all grouping items in the data source;

[0066] Determine whether the parameters of the set functions in the statement to be executed have the set flag;

[0067] If present, then combine the grouping item and the parameters of the set function;

[0068] Determine whether the combined data is ordered;

[0069] If ordered, the identifier information of the statement to be executed is set to a set value.

[0070] Specifically, for a statement to be executed, all grouping items in the statement are identified and recorded. It is then determined whether the parameters of the set functions in the statement to be executed have set flags; the parameters of the set functions of statements with set flags are combined with the grouping items. The combined data is then checked for order; if the data source is a table, the order can be determined based on the table's index. If the combined data is ordered, the identifier information of the statement to be executed can be set to a set value, indicating that the statement to be executed can be processed as a data processing statement.

[0071] For example, all grouping items (GROUP BY items) of a statement to be executed are collected sequentially, denoted as X1..Xn. It is determined whether the parameters of the aggregate functions in the statement to be executed have the DISTINCT flag. If they do not, processing is terminated; otherwise, the parameter Y corresponding to the aggregate function in the statement to be executed is retrieved. Parameter Y and the collected grouping items are combined to form a structure in the form (X1..Xn,Y). It is determined whether the data in the combined (X1..Xn,Y) structure is ordered. If it is unordered, processing is terminated; otherwise, the identifier information of the statement to be executed is set to a predetermined value. When processing the statements to be executed, those with the predetermined identifier value are designated as pending processing statements and continue with subsequent analysis and processing.

[0072] Example 2

[0073] Figure 2 This is a flowchart of a method for adding and determining deduplicated items according to Embodiment 2 of the present invention. This embodiment focuses on the method of adding deduplicated items to the determination of set function results in the above embodiments. Figure 2 As shown, the method includes:

[0074] S210. Determine whether the values ​​of the two grouping items are the same. If yes, execute S220; otherwise, execute S230.

[0075] Specifically, for the data in the data source, first select two adjacent grouping items and their corresponding deduplication items, determine whether the values ​​of the two grouping items are the same, and determine whether they are the same group of data based on the grouping item judgment results.

[0076] For example, for data from the data source, adjacent rows of data are first compared according to the grouping item (X1..Xn) as the key value. For example, rows N1 and N2 are two adjacent rows of data, where row N1 is regarded as the current row and row N2 is the next row of data of the current row. It is determined whether the grouping item values ​​of rows N1 and N2 are the same.

[0077] S220. If the values ​​of the two obtained grouping items are the same, determine whether the two obtained deduplication items are the same, and determine whether to add the deduplication item corresponding to the grouping item that appears earlier in the order of the two adjacent grouping items to the determination of the set function processing result based on the comparison result of whether the deduplication items are the same.

[0078] Specifically, if the values ​​of the two grouping items are the same, then it is determined whether the two deduplicated items are the same. If the two deduplicated items are the same, the later-ordered grouping item and its corresponding deduplicated item are retained, and the deduplicated item is added to the determination of the set function result; if the two deduplicated items are different, the deduplicated item corresponding to the earlier-ordered grouping item is added to the determination of the set function result, and the later-ordered deduplicated item continues to be compared with the next adjacent row of data.

[0079] For example, if the grouping items (X1..Xn) of rows N1 and N2 are the same, it means they are in the same group. Then, continue to compare the deduplication item (Y) column: if the deduplication item Y of rows N1 and N2 is the same, then only the data of row N2 is kept, and row N2 is used as the current row; if the deduplication item Y of rows N1 and N2 is different, then the value of the deduplication item (Y) column of row N1 is added to the result calculation according to the set function type, and row N2 is used as the current row.

[0080] Optionally, the step of determining whether to add the deduplicated item corresponding to the earlier grouping item in the order of the two adjacent grouping items to the set function processing result based on the comparison result of whether the deduplicated items are the same includes:

[0081] If the values ​​of the two deduplication items are the same, the later group item and its corresponding deduplication item in the two adjacent group items are retained; otherwise, it is determined that the deduplication item corresponding to the earlier group item in the two adjacent group items needs to be added to the determination of the set function processing result.

[0082] Specifically, if two deduplication items are the same, the later-ordered group item and its corresponding deduplication item are retained, and the deduplication item is added to the set function result determination of the corresponding group. If two deduplication items are different, the deduplication item corresponding to the earlier-ordered group item is added to the set function result determination of the corresponding group, and the later-ordered deduplication item continues to be compared with the next adjacent row of data.

[0083] For example, if the deduplication item Y in rows N1 and N2 is the same, then only the data in row N2 is retained, and the deduplication item Y corresponding to row N2 is added to the result calculation according to the set function type, with row N2 as the current row; if the deduplication item Y in rows N1 and N2 is different, then the value of the deduplication item (Y) column of row N1 is added to the result calculation according to the set function type, with row N2 as the current row, and the comparison continues with the next adjacent row of data.

[0084] S230. If the two grouping items obtained are different, it is determined that the deduplication item corresponding to the first grouping item in the two adjacent grouping items needs to be added to the determination of the set function processing result, and the set function result of the group corresponding to the first grouping item in the two adjacent grouping items is determined.

[0085] Specifically, if the values ​​of two grouping items are different, the duplicate item corresponding to the earlier grouping item in the order of these two adjacent grouping items is added to the determination of the set function result, and the duplicate item corresponding to the later grouping item is compared with the next adjacent row of data. In this way, the set function result corresponding to the grouping item corresponding to the earlier grouping item in the order of these two adjacent grouping items can be determined.

[0086] For example, if the grouping items (X1..Xn) of rows N1 and N2 are different, it means they are different groups. In this case, the value of the deduplication item (Y) column of row N1 is added to the result calculation according to the set function type, and the current calculation result is output, which is the processing result of the group corresponding to row N1 after the set function processing. After outputting, the calculation result is cleared to zero for subsequent calculations, and row N2 is used as the current row for the next stage of judgment.

[0087] The technical solution of this invention involves determining whether the values ​​of two acquired grouping items are the same. If the values ​​of the two acquired grouping items are the same, it is determined whether the two acquired deduplication items are the same, and based on the comparison result of whether the deduplication items are the same, it is determined whether to add the deduplication item corresponding to the earlier sequential grouping item among the two adjacent grouping items to the determination of the set function processing result. If the two acquired grouping items are different, it is determined that the deduplication item corresponding to the earlier sequential grouping item among the two adjacent grouping items needs to be added to the determination of the set function processing result, and the set function result of the group corresponding to the earlier sequential grouping item among the two adjacent grouping items is determined. This refines the method for determining whether to add the deduplication item corresponding to the two adjacent grouping items to the determination of the set function result based on the comparison result of the grouping items and the deduplication items, thus completing the determination of the set function processing result.

[0088] Example 3

[0089] Figure 3 This is a schematic diagram of the structure of a data processing device provided in Embodiment 3 of the present invention. Figure 3As shown, the device includes:

[0090] The first determining module 310 is used to determine the statement to be processed, the statement to be processed including a structured query statement of a set function with a set flag, the set flag including a flag to remove duplicate values ​​in the parameters of the set function;

[0091] The acquisition module 320 is used to acquire two adjacent grouping items and the deduplication item corresponding to each grouping item in the data source corresponding to the statement to be processed. The data source is ordered and includes multiple grouping items and the deduplication item corresponding to each grouping item.

[0092] The second determining module 330 is used to determine whether to add the deduplication item corresponding to the first sequential group item in the two adjacent group items to the determination of the set function result based on the comparison results of the acquired group items and the acquired deduplication items respectively. If so, the deduplication item corresponding to the first sequential group item in the two adjacent group items is added to the determination of the set function result.

[0093] Return module 340 is used to continue selecting the next group item and the deduplication item corresponding to the group item from the data source, and take the group item that comes later in the order of the two adjacent group items and the next group item as the new two adjacent group items, and return to continue to determine whether to process through the set function until the last group item in the data source is obtained;

[0094] Add module 350 is used to add the deduplication item corresponding to the last grouping item to the determination of the set function result, so as to obtain the set function result of the grouping item corresponding to the grouping item.

[0095] The data loading apparatus provided in this embodiment of the invention first determines the statement to be processed through a first determining module, and obtains two adjacent grouping items and the deduplication items corresponding to each grouping item from the data source corresponding to the statement to be processed through an obtaining module. Next, a second determining module determines, based on the comparison results of the obtained grouping items and the obtained deduplication items, whether to add the deduplication item corresponding to the earlier-ordered grouping item in the two adjacent grouping items to the determination of the set function result. If so, the deduplication item corresponding to the earlier-ordered grouping item in the two adjacent grouping items is added to the determination of the set function result. A return module continues to select the next grouping item and the deduplication item corresponding to the grouping item from the data source, and takes the later-ordered grouping item in the two adjacent grouping items and the next grouping item as new adjacent grouping items, returning to continue determining whether to process through the set function, until the last grouping item in the data source is obtained. Finally, an adding module adds the deduplication item corresponding to the last grouping item to the determination of the set function result, obtaining the set function result of the grouping item. By coordinating the various modules and comparing the grouping and deduplication items in the data source, a method for determining the processing result of the set function was obtained, simplifying the calculation process and realizing the function of quickly calculating the set function result.

[0096] In one embodiment, the second determining module 330 includes:

[0097] The first determining unit is used to determine whether the values ​​of the two grouping items obtained are the same;

[0098] The second determining unit is used to determine whether the two deduplication items are the same if the values ​​of the two obtained grouping items are the same, and to determine whether to add the deduplication item corresponding to the grouping item that appears earlier in the order of the two adjacent grouping items to the determination of the set function processing result based on the comparison result of whether the deduplication items are the same.

[0099] The third determining unit is used to determine, if the two obtained grouping items are different, that the deduplication item corresponding to the earlier grouping item in the adjacent two grouping items needs to be added to the set function processing result, and to determine the set function result of the group corresponding to the earlier grouping item in the adjacent two grouping items.

[0100] In one embodiment, the second determining unit is specifically used for:

[0101] If the values ​​of the two deduplication items are the same, the later group item and its corresponding deduplication item in the two adjacent group items are retained; otherwise, it is determined that the deduplication item corresponding to the earlier group item in the two adjacent group items needs to be added to the determination of the set function processing result.

[0102] In one embodiment, the first determining module 310 is specifically used for:

[0103] Get the statement to be executed;

[0104] Determine the identification information corresponding to the statement to be executed;

[0105] If the identification information is a set value, the statement to be executed is determined to be a statement to be processed.

[0106] In one embodiment, the data source includes a table, the grouping items and the deduplication items are different columns in the table, and adjacent deduplication items are two adjacent rows of data in the table.

[0107] In one embodiment, the data processing apparatus further includes a third determining module, specifically used for:

[0108] Identify all grouping items in the data source;

[0109] Determine whether the parameters of the set functions in the statement to be executed have the set flag;

[0110] If present, then combine the grouping item and the parameters of the set function;

[0111] Determine whether the combined data is ordered;

[0112] If ordered, the identifier information of the statement to be executed is set to a set value.

[0113] The data processing device provided in the embodiments of the present invention can execute the data processing method provided in any embodiment of the present invention. Through the mutual cooperation and collaborative work between the modules, the data processing is completed, and it has the corresponding functional modules and beneficial effects of the execution method.

[0114] Example 4

[0115] According to embodiments of the present invention, the present invention also provides an electronic device, a computer-readable storage medium, and a computer program product.

[0116] Figure 4This is a block diagram of an electronic device according to Embodiment 4 of the present invention, which implements the data processing method described in the embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0117] like Figure 4 As shown, the electronic device 410 includes at least one processor 411 and a memory, such as a read-only memory (ROM) 412 or a random access memory (RAM) 413, communicatively connected to the at least one processor 411. The memory stores computer programs executable by the at least one processor. The processor 411 can perform various appropriate actions and processes based on the computer program stored in the ROM 412 or loaded from storage unit 418 into the RAM 413. The RAM 413 can also store various programs and data required for the operation of the electronic device 410. The processor 411, ROM 412, and RAM 413 are interconnected via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0118] Multiple components in the electronic device are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, optical disk, etc.; and a communication unit 419, such as a network card, modem, wireless transceiver, etc. The communication unit 419 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0119] Processor 411 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 411 performs the various methods and processes described above, such as data processing methods.

[0120] In some embodiments, the data processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 410 via ROM 412 and / or communication unit 419. When the computer program is loaded into RAM 413 and executed by processor 411, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, processor 411 may be configured to perform the data processing method by any other suitable means (e.g., by means of firmware).

[0121] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0123] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0125] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0126] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0127] In some embodiments, the computer program product includes a computer program that, when executed by a processor, implements the data processing method provided in the embodiments of the present invention.

[0128] The technical solution of this invention, through a data processing method, apparatus, electronic device, storage medium, and program product, determines a statement to be processed and obtains two adjacent grouping items and the deduplication items corresponding to each grouping item from the data source corresponding to the statement to be processed. Then, based on the comparison results of the obtained grouping items and the obtained deduplication items, it is determined whether to add the deduplication item corresponding to the earlier-ordered grouping item in the two adjacent grouping items to the determination of the set function result. If so, the deduplication item corresponding to the earlier-ordered grouping item in the two adjacent grouping items is added to the determination of the set function result. The process continues by selecting the next grouping item and the deduplication item corresponding to the grouping item from the data source, and taking the later-ordered grouping item in the two adjacent grouping items and the next grouping item as new adjacent grouping items, returning to continue determining whether to process through the set function, until the last grouping item in the data source is obtained. Finally, the deduplication item corresponding to the last grouping item is added to the determination of the set function result to obtain the set function result of the grouping item. By comparing the grouping and deduplication items in the data source, a method for determining the processing result of the set function was obtained, which simplified the calculation process and enabled the function of quickly calculating the set function result.

[0129] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0130] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data processing method, characterized in that, include: Identify the statement to be processed, which includes a structured query statement with a set function that has a set flag, the set flag including a flag to remove duplicate values ​​from the parameters of the set function; Obtain two adjacent grouping items and the deduplication item corresponding to each grouping item from the data source corresponding to the statement to be processed. The data source is ordered and includes multiple grouping items and the deduplication item corresponding to each grouping item. Based on the comparison results of the acquired grouping items and the acquired deduplication items, determine whether to add the deduplication item corresponding to the earlier sequential grouping item in the two adjacent grouping items to the determination of the set function result. If so, add the deduplication item corresponding to the earlier sequential grouping item in the two adjacent grouping items to the determination of the set function result. Continue selecting the next group item and the deduplication item corresponding to the group item from the data source, and take the group item that comes later in the order of the two adjacent group items and the next group item as the new two adjacent group items, and return to continue to determine whether to process through the set function, until the last group item in the data source is obtained; The deduplication item corresponding to the last grouping item is added to the determination of the set function result to obtain the set function result of the grouping item corresponding to the grouping item.

2. The method according to claim 1, characterized in that, The determination of whether to add the deduplication item corresponding to the earlier grouping item in the order of the two adjacent grouping items to the set function result, based on the comparison results of the acquired grouping items and the acquired deduplication items, includes: Determine whether the values ​​of the two grouping items obtained are the same; If the values ​​of the two obtained grouping items are the same, it is determined whether the two obtained deduplication items are the same, and based on the comparison result of whether the deduplication items are the same, it is determined whether to add the deduplication item corresponding to the grouping item that appears earlier in the order of the two adjacent grouping items to the determination of the set function processing result; If the two grouping items obtained are different, it is determined that the deduplication item corresponding to the earlier grouping item in the two adjacent grouping items needs to be added to the determination of the set function processing result, and the set function result of the group corresponding to the earlier grouping item in the two adjacent grouping items is determined.

3. The method according to claim 2, characterized in that, The step of determining whether to add the earlier grouping item from two adjacent grouping items to the set function processing result based on the comparison result of whether the deduplication items are the same includes: If the values ​​of the two deduplication items are the same, the later group item and its corresponding deduplication item in the two adjacent group items are retained; otherwise, it is determined that the deduplication item corresponding to the earlier group item in the two adjacent group items needs to be added to the determination of the set function processing result.

4. The method according to claim 1, characterized in that, The determination of the statement to be processed includes: Get the statement to be executed; Determine the identification information corresponding to the statement to be executed; If the identification information is a set value, the statement to be executed is determined to be a statement to be processed.

5. The method according to any one of claims 1-4, characterized in that, The data source includes a table, and the grouping items and the deduplication items are different columns in the table, with adjacent deduplication items being two adjacent rows of data in the table.

6. The method according to claim 1, characterized in that, Before determining the statement to be processed, the following is also included: Identify all grouping items in the data source; Determine whether the parameters of the set functions in the statement to be executed have the set flag; If present, then combine the grouping item and the parameters of the set function; Determine whether the combined data is ordered; If ordered, the identifier information of the statement to be executed is set to a set value.

7. A data processing apparatus, characterized in that, include: The first determining module is used to determine the statement to be processed, the statement to be processed including a structured query statement of a set function with a set flag, the set flag including a flag to remove duplicate values ​​in the parameters of the set function; The acquisition module is used to acquire two adjacent grouping items and the deduplication item corresponding to each grouping item in the data source corresponding to the statement to be processed. The data source is ordered and includes multiple grouping items and the deduplication item corresponding to each grouping item. The second determining module is used to determine whether to add the deduplication item corresponding to the earlier group item in the two adjacent group items to the determination of the set function result based on the comparison results of the acquired group items and the acquired deduplication items respectively. If so, the deduplication item corresponding to the earlier group item in the two adjacent group items is added to the determination of the set function result. The return module is used to continue selecting the next group item and the deduplication item corresponding to the group item from the data source, and to take the group item that comes later in the order of the two adjacent group items and the next group item as the new two adjacent group items, and return to continue to determine whether to process through the set function, until the last group item in the data source is obtained; An addition module is used to add the deduplication item corresponding to the last grouping item to the determination of the set function result, thereby obtaining the set function result of the grouping item corresponding to the grouping.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data processing method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the data processing method according to any one of claims 1-6.

Images