Method, device, electronic equipment and storage medium for data processing

Abstract

The application relates to the technical field of data processing, and particularly provides a data processing method and device, electronic equipment and a storage medium. The data processing method comprises the following steps: a storage layer reads target data blocks based on a data aggregation instruction of an execution engine to obtain a category value set; the data to be aggregated is grouped and aggregated according to the category values in the category value set to obtain an aggregation result corresponding to each category value; the category parameters corresponding to each category value are respectively obtained based on a numerical conversion relationship; and the category parameters and the corresponding aggregation results are sent to the execution engine. In this way, the data grouping and aggregation operation are pushed down to the storage layer, the numerical conversion and grouping and aggregation operation are performed based on the numerical conversion relationship corresponding to the local data blocks, the grouping and aggregation result can be obtained and uploaded, the data transmission amount is reduced, and the data processing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of internet technology, people have increasingly higher demands for data processing efficiency. Under current technology, when performing data aggregation, the execution engine typically reads the raw data stored in the storage layer and then groups and aggregates that raw data. The execution engine and the storage layer are different modules of the same device.

[0003] However, using this method requires the storage layer to transfer a large amount of data to the execution engine, resulting in low data processing efficiency. Summary of the Invention

[0004] The purpose of this application is to provide a data processing method, apparatus, electronic device, and storage medium to improve the efficiency of data processing.

[0005] In a first aspect, embodiments of this application provide a data processing method applied to a storage layer in an electronic device, the method comprising:

[0006] Based on the data aggregation instructions of the execution engine, data is read from the target data block to obtain a set of category values; the set of category values ​​is a set of category values ​​obtained by transforming the category parameters based on numerical transformation relationships; the numerical transformation relationships are the correspondence between the category parameters and values ​​set for the target data block; the electronic device also includes an execution engine;

[0007] Based on the category values ​​in the category value set, the data to be aggregated is grouped and aggregated to obtain the aggregation result corresponding to each category value;

[0008] Based on the numerical transformation relationship, the category parameters corresponding to each category value are obtained respectively;

[0009] The category parameters and their corresponding aggregation results are sent to the execution engine.

[0010] In one implementation, before reading data from the target data block based on the data aggregation instructions of the execution engine to obtain the set of category values, the method further includes:

[0011] When a data storage instruction for a target data block is received, the data to be stored in the data storage instruction is retrieved.

[0012] Based on the numerical conversion relationship, the numerical values ​​of the data to be stored are converted to obtain at least one value corresponding to the data to be stored.

[0013] Write the value to be stored into the target data block.

[0014] In one implementation, based on the data aggregation instructions of the execution engine, data is read from the target data block to obtain a set of category values, including:

[0015] Upon receiving a data aggregation instruction from the execution engine, obtain the first region information from the data aggregation instruction;

[0016] Read the data block data corresponding to the first region information from the target data block to obtain the category value set.

[0017] In one implementation, the data to be aggregated is grouped and aggregated according to the category values ​​in the category value set to obtain the aggregation result corresponding to each category value, including:

[0018] Based on the second region information also included in the data aggregation instruction, data is read to obtain the data to be aggregated;

[0019] Based on the category value corresponding to each value to be aggregated in the data to be aggregated, the data to be aggregated is divided to obtain the combination to be aggregated corresponding to each category value;

[0020] Perform aggregation calculations on each combination to be aggregated to obtain the aggregation result corresponding to each category value.

[0021] In one implementation, the first region information is the name of the first column, and the second region information is the name of the second column. The data to be aggregated is divided according to the category value corresponding to each value in the data to be aggregated, obtaining the aggregation combination corresponding to each category value, including:

[0022] Based on the row name of each category value, obtain the set of row names corresponding to each category value;

[0023] Determine the set of row names corresponding to each category value, and the row name of each value to be aggregated;

[0024] The values ​​to be aggregated that belong to the same set of row names are divided into the same aggregation group, and the aggregation group corresponding to each category value is obtained.

[0025] On one hand, embodiments of this application provide a data processing apparatus applied to a storage layer in an electronic device, the apparatus comprising:

[0026] The reading unit is used to read data from the target data block based on the data aggregation instructions of the execution engine to obtain a set of category values. The set of category values ​​is a set of category values ​​obtained by transforming the category parameters based on numerical transformation relationships. The numerical transformation relationships are the correspondence between the category parameters and values ​​set for the target data block. The electronic device also includes an execution engine.

[0027] The aggregation unit is used to group and aggregate the data to be aggregated based on the category values ​​in the category value set, and obtain the aggregation result corresponding to each category value.

[0028] The transformation unit is used to obtain the category parameters corresponding to each category value based on the numerical transformation relationship;

[0029] The sending unit is used to send the category parameters and their corresponding aggregation results to the execution engine.

[0030] In one embodiment, the reading unit is further configured to:

[0031] When a data storage instruction for a target data block is received, the data to be stored in the data storage instruction is retrieved.

[0032] Based on the numerical conversion relationship, the numerical values ​​of the data to be stored are converted to obtain at least one value corresponding to the data to be stored.

[0033] Write the value to be stored into the target data block.

[0034] In one embodiment, the reading unit is used for:

[0035] Upon receiving a data aggregation instruction from the execution engine, obtain the first region information from the data aggregation instruction;

[0036] Read the data block data corresponding to the first region information from the target data block to obtain the category value set.

[0037] In one embodiment, the aggregation unit is used for:

[0038] Based on the second region information also included in the data aggregation instruction, data is read to obtain the data to be aggregated;

[0039] Based on the category value corresponding to each value to be aggregated in the data to be aggregated, the data to be aggregated is divided to obtain the combination to be aggregated corresponding to each category value;

[0040] Perform aggregation calculations on each combination to be aggregated to obtain the aggregation result corresponding to each category value.

[0041] In one implementation, the first region information is the name of a first column, the second region information is the name of a second column, and the aggregation unit is used for:

[0042] Based on the row name of each category value, obtain the set of row names corresponding to each category value;

[0043] Determine the set of row names corresponding to each category value, and the row name of each value to be aggregated;

[0044] The values ​​to be aggregated that belong to the same set of row names are divided into the same aggregation group, and the aggregation group corresponding to each category value is obtained.

[0045] Thirdly, embodiments of this application provide an electronic device, including:

[0046] Processor; and

[0047] The memory stores computer instructions that cause the processor to perform the steps of the methods provided in the various alternative implementations of any of the data processing described above.

[0048] Fourthly, embodiments of this application provide a storage medium storing computer instructions for causing a computer to perform the steps of the methods provided in various alternative implementations of any of the above-described data processing methods.

[0049] In the data processing method, apparatus, electronic device, and storage medium provided in this application embodiment, the storage layer reads data from the target data block based on the data aggregation instructions of the execution engine to obtain a set of category values. The set of category values ​​is a set of category values ​​obtained by transforming category parameters based on numerical transformation relationships. The numerical transformation relationships are the correspondence between category parameters and values ​​set for the target data block. Based on the category values ​​in the set of category values, the data to be aggregated is grouped and aggregated to obtain the aggregation result corresponding to each category value. Based on the numerical transformation relationships, the category parameters corresponding to each category value are obtained respectively. The category parameters and their corresponding aggregation results are sent to the execution engine. In this way, data grouping and aggregation operations are pushed down to the storage layer, and numerical transformation and grouping aggregation operations are performed based on the numerical transformation relationships set locally corresponding to the data blocks. This allows for the acquisition and uploading of grouping and aggregation results, reducing data transmission volume and improving data processing efficiency. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0051] Figure 1 This is a flowchart of a data processing method according to some embodiments of this application.

[0052] Figure 2 This is a flowchart illustrating the implementation of a grouping and aggregation method according to some embodiments of this application.

[0053] Figure 3This is a structural block diagram of a data processing apparatus according to some embodiments of this application.

[0054] Figure 4 This is a schematic diagram of the structure of an electronic device according to some embodiments of this application. Detailed Implementation

[0055] The technical solution of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Furthermore, the technical features involved in the different embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0056] First, some of the terms used in the embodiments of this application will be explained to facilitate understanding by those skilled in the art.

[0057] Terminal devices can be mobile terminals, fixed terminals, or portable terminals, such as mobile phones, sites, units, devices, multimedia computers, multimedia tablets, internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system devices, personal navigation devices, personal digital assistants, audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is also foreseeable that terminal devices can support any type of user-facing interface (e.g., wearable devices).

[0058] Servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0059] In practical applications, data often needs to be grouped and aggregated. With the development of internet technology, the volume of data processing is constantly increasing, and people's demands for data processing efficiency are also constantly rising. Traditionally, data grouping and aggregation are typically performed using the following methods:

[0060] Option 1: Update and maintain the global dictionary encoding in real time when writing data. During grouping and aggregation calculations, group and aggregate data based on the global dictionary encoding. However, this approach consumes significant resources due to the real-time updates and maintenance of the dictionary encoding.

[0061] Method 2: Pre-calculate some specific grouping and aggregation commands, so that the grouping and aggregation results can be directly queried and obtained when the application is in use. However, this method can only be used in scenarios where specific grouping and aggregation commands are received in the application, and its scope of application is relatively small.

[0062] In view of the deficiencies of the above-mentioned related technologies, the embodiments of this application provide a data processing method, apparatus, electronic device and storage medium, which aim to improve data processing performance when performing data processing.

[0063] This application provides a data processing method applicable to the storage layer of an electronic device, which includes a storage layer and an execution engine. This application does not limit the type of electronic device; it can be any suitable type of device, such as a terminal device or a server, which will not be elaborated further.

[0064] See Figure 1 The diagram shown is a flowchart of a data processing method in some embodiments of this application. The following is a description of the method in conjunction with... Figure 1 The method is described below, and the specific implementation process is as follows:

[0065] Step 100: The storage layer reads data from the target data block based on the data aggregation instructions of the execution engine to obtain a set of category values.

[0066] Specifically, the category value set is the set of category values ​​obtained by transforming category parameters based on numerical transformation relationships; the numerical transformation relationship is the correspondence between category parameters and values ​​set for the target data block. The target data block is a microblock. When data is stored on disk in a database, it is generally divided into data blocks for organization, and this data block is called a microblock. It should be noted that this numerical transformation relationship is set locally for a single microblock, not globally; therefore, the resource consumption for maintaining this numerical transformation relationship is relatively small.

[0067] As an example, numerical transformation relationships can take the form of dictionary encoding. Dictionary encoding is a lossless data compression method that maintains a dictionary. During encoding, data is converted into its corresponding index in the dictionary (i.e., category parameters are converted into category values), and during decoding, the original data is restored based on the index (i.e., category values ​​are converted back into category parameters). The characteristics of dictionary encoding can help accelerate computation for certain operators. Dictionary encoding converts data into its corresponding index in the dictionary; the same data will be encoded with the same index, thus naturally grouping the data.

[0068] It should be noted that category parameters are typically text or strings that consume significant storage space. Category values ​​can be numerical values ​​such as natural numbers. Clearly, by converting the category parameters to numerical values, the storage space required and the computational load can be reduced.

[0069] In one implementation, step 100 may include the following steps:

[0070] S100-11: When a data aggregation instruction sent by the execution engine is received, obtain the first region information in the data aggregation instruction.

[0071] The first area information is an indication of the data storage location. Optionally, the first area information can be a row name or a column name (e.g., the column name is a city), or it can be the regional location of the data storage.

[0072] In practical applications, the information in the first area can be set according to the actual application scenario, and there are no restrictions here.

[0073] S100-12: Read the data block data corresponding to the first region information from the target data block to obtain the category value set.

[0074] For example, if the first area information is column name C1, then the values ​​in column name C1 can be read to obtain the set of category values.

[0075] Furthermore, to facilitate subsequent data grouping, identical category values ​​in the category value set are grouped together to obtain at least one category value combination.

[0076] It should be noted that since the category values ​​are all numerical, the amount of data processed by grouping is small, and the processing efficiency is high.

[0077] In this way, the set of category values ​​can be obtained based on the information from the first region.

[0078] Furthermore, when the storage layer receives data to be stored, it can also store the data. In one embodiment, the implementation process of step 100 may further include:

[0079] S100-21: When a data storage instruction for a target data block is received, the data to be stored in the data storage instruction is obtained.

[0080] S100-22: Based on the numerical conversion relationship, convert the numerical values ​​of the data to be stored to obtain at least one numerical value corresponding to the data to be stored.

[0081] S100-23: Write the value to be stored into the target data block.

[0082] In this embodiment of the application, in order to reduce the large amount of storage resources consumed by data storage, the storage layer converts the data to be stored (e.g., text) into numerical values ​​through numerical conversion relationships, obtains the numerical values ​​to be stored, and writes the numerical values ​​to be stored into the target data block, thereby reducing the storage resources consumed and the time spent on subsequent processing.

[0083] Step 101: Based on the category values ​​in the category value set, group and aggregate the data to be aggregated to obtain the aggregation result corresponding to each category value.

[0084] In one implementation, the process of step 101 may include:

[0085] S1011: Based on the second region information also included in the data aggregation instruction, read the data to be aggregated.

[0086] In one implementation, according to the data aggregation instruction, the data block to be aggregated and the second region information are determined, and the data corresponding to the second region information of the data block to be aggregated is read to obtain the data to be aggregated.

[0087] Optionally, the data block to be aggregated can be the target data block or other data blocks.

[0088] Specifically, the second area information is an indication of the data storage location. Optionally, the second area information can be a row name or a column name, or it can be the data storage area location. In actual applications, the second area information can be set according to the actual application scenario, and there are no restrictions here.

[0089] As an example, if the second area information is sequence number C2, then the data with sequence number C2 in the data block to be aggregated can be read.

[0090] In this way, the data to be aggregated can be obtained based on the information from the second region.

[0091] S1012: Based on the category values ​​corresponding to each value to be aggregated in the data to be aggregated, divide the data to be aggregated and obtain the combination to be aggregated corresponding to each category value.

[0092] In one implementation, a set of row names corresponding to each category value is obtained based on the row name of each category value; the values ​​to be aggregated that belong to the same set of row names are divided into the same group to be aggregated, and the group to be aggregated corresponding to each category value is obtained.

[0093] S1013: Perform aggregation calculations on each combination to be aggregated to obtain the aggregation result corresponding to each category value.

[0094] Optionally, aggregation calculations can perform aggregation operations on data in the database, such as counting (COUNT).

[0095] Calculate the minimum value (MIN), calculate the maximum value (MAX), and perform summation (SUM), etc.

[0096] As an example, calculate the sum of the values ​​to be aggregated in a combination to be aggregated, and the summation result is the aggregation result.

[0097] In this way, the data to be aggregated in the second region information can be grouped according to the category values ​​in the first region information to obtain one or more aggregated combinations. Each aggregated combination can then be aggregated separately to obtain the corresponding aggregation results.

[0098] Step 102: Based on the numerical transformation relationship, obtain the category parameters corresponding to each category value.

[0099] This allows the category values ​​to be converted back to the original category parameters after the aggregation calculation.

[0100] Step 103: Send the category parameters and their corresponding aggregation results to the execution engine.

[0101] In this way, the grouping and aggregation calculations of data are moved from the execution engine to the storage layer. After the aggregation calculation, the original category parameters and the aggregation results of each category parameter are sent to the upper-level execution engine, thereby reducing the amount of data transmitted and the amount of data processed by the execution engine, and improving data processing efficiency.

[0102] In one application scenario, when processing data in a database using the Group By operator, the grouping aggregation operation in the Group By operator can be pushed down to the storage layer based on the data processing method provided in the embodiments of this application, and the dictionary encoding of the storage layer can be used to perform grouping aggregation calculations on some data, thereby greatly improving data processing efficiency.

[0103] It's important to note that in databases, operators represent operations on data, such as the array sorting operator (Sort), the join operator (Join), and the Group By operator. The Group By operator in databases is used to group and aggregate datasets based on certain columns.

[0104] In this embodiment of the application, the method of data processing described above is illustrated by taking the grouping and aggregation of data in the database using the Group By operator as an example. (See also...) Figure 2 The diagram shown is a flowchart illustrating the implementation of a grouping and aggregation method applied to electronic devices. The specific implementation process of this method is as follows:

[0105] Step 200: The execution engine sends group aggregation instructions to the storage layer.

[0106] In one implementation, the execution engine responds to a user's grouping and aggregation operation by issuing a grouping and aggregation instruction (i.e., a data aggregation instruction) to the storage layer. The grouping and aggregation instruction includes an aggregation function and column information (i.e., first region information and second region information).

[0107] As an example, a grouping aggregation command could be `SELECT C1, SUM(C2) FROM T Group By C1`. Based on this command, the storage layer can obtain the following information: the Group By column is C1; the data in column C2 (the data to be aggregated) needs to be grouped and summed (i.e., aggregated) according to the data in column C1 (i.e., category values), resulting in the aggregation result `SUM(C2)`. Here, C1 and C2 are different column names.

[0108] Furthermore, the execution engine can first determine whether the target data block uses dictionary encoding for data compression. If so, it sends a grouping and aggregation instruction to the storage layer; otherwise, the execution engine does not send a grouping and aggregation instruction to the storage layer, that is, it does not use the method provided in the embodiments of this application.

[0109] Step 201: The storage layer reads data from the target data block based on the data aggregation instructions of the execution engine to obtain a set of category values.

[0110] Optionally, the target data block is determined according to the grouping and aggregation instruction. There can be one or more, in which case the data in column C1 can be read directly from the target data block to obtain the set of category values.

[0111] The data in column C1 consists of category values ​​obtained by the storage layer through dictionary encoding, which converts the original category parameters into numerical values. The category values ​​in the category value set have been grouped.

[0112] Step 202: The storage layer reads the data to be aggregated based on the data aggregation instructions of the execution engine.

[0113] Specifically, the storage layer reads the data in column C2 of the data block to be aggregated to obtain the data to be aggregated.

[0114] Step 203: The storage layer groups and aggregates the data to be aggregated based on the category values ​​in the category value set, and obtains the aggregation result corresponding to each category value.

[0115] In one implementation, the following steps can be used to group the data:

[0116] Each category value in the category value set can be represented as A[i, C1]. A combination of category values ​​can have at least one category value. Each value to be aggregated in the data to be aggregated can be represented as B[i, C2]. i is a natural number, representing the row name of the data in the data block. If the row name of a value to be aggregated is the same as the row name of a category value, it means that the value to be aggregated corresponds to that category value. The values ​​to be aggregated corresponding to each category value in the same category value combination are grouped into the same group.

[0117] Step 204: The storage layer obtains the category parameters corresponding to each category value based on dictionary encoding.

[0118] Step 205: The storage layer sends the category parameters and their corresponding aggregation results to the Group By operator in the upper layer.

[0119] Specifically, the storage layer sends the category parameters and their corresponding aggregation results to the GroupBy operator in the upper layer of the execution engine.

[0120] Step 206: The upper-level Group By operator performs data processing based on the category parameters and their corresponding aggregation results.

[0121] In this embodiment, the grouping and aggregation operations in the Group By operator are pushed down to the storage layer. This allows for pre-grouping and aggregation based on the numerical transformation relationships maintained by micro-blocks, and the grouping and aggregation results are returned to the upper-layer operator. The upper-layer operator can then perform subsequent calculations based on the received content. In this way, numerical transformation relationships are set and maintained only for micro-blocks (e.g., only the dictionary encoding at the micro-block level in data compression needs to be maintained), without the need to maintain global numerical transformation relationships separately. This reduces the amount of data processing and the equipment resources consumed. Furthermore, the numerical values ​​to be aggregated can be grouped based on the correspondence between category values ​​and row names of the values ​​to be aggregated, simplifying the tedious operation of data grouping. Moreover, based on the numerical transformation relationships of the target data blocks, fast grouping and aggregation can be achieved without maintaining pre-calculation results for specific patterns, reducing the amount of data transmitted and improving the processing efficiency of data grouping and aggregation.

[0122] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0123] Based on the same inventive concept, this application also provides a data processing apparatus. Since the principle of the above-described apparatus and device in solving the problem is similar to that of a data processing method, the implementation of the above-described apparatus can refer to the implementation of the method, and repeated details will not be described again. This apparatus can be applied to electronic devices. This application does not limit the type of electronic device; it can be any suitable type of device, such as smartphones, tablet computers, etc., which will not be described again in this application.

[0124] See Figure 3 The diagram shown is a structural block diagram of a data processing apparatus in some embodiments of this application. In some embodiments, the data processing apparatus exemplified in this application includes:

[0125] The reading unit 301 is used to read data from the target data block based on the data aggregation instructions of the execution engine to obtain a set of category values. The set of category values ​​is a set of category values ​​obtained by transforming the category parameters based on the numerical transformation relationship. The numerical transformation relationship is the correspondence between the category parameters and the values ​​set for the target data block.

[0126] Aggregation unit 302 is used to group and aggregate the data to be aggregated according to the category values ​​in the category value set, and obtain the aggregation result corresponding to each category value;

[0127] The conversion unit 303 is used to obtain the category parameters corresponding to each category value based on the numerical conversion relationship;

[0128] Sending unit 304 is used to send the category parameters and their corresponding aggregation results to the execution engine.

[0129] In one embodiment, the reading unit 301 is further configured to:

[0130] When a data storage instruction for a target data block is received, the data to be stored in the data storage instruction is retrieved.

[0131] Based on the numerical conversion relationship, the numerical values ​​of the data to be stored are converted to obtain at least one value corresponding to the data to be stored.

[0132] Write the value to be stored into the target data block.

[0133] In one embodiment, the reading unit 301 is used for:

[0134] Upon receiving a data aggregation instruction from the execution engine, obtain the first region information from the data aggregation instruction;

[0135] Read the data block data corresponding to the first region information from the target data block to obtain the category value set.

[0136] In one embodiment, the aggregation unit 302 is used for:

[0137] Based on the second region information also included in the data aggregation instruction, data is read to obtain the data to be aggregated;

[0138] Based on the category value corresponding to each value to be aggregated in the data to be aggregated, the data to be aggregated is divided to obtain the combination to be aggregated corresponding to each category value;

[0139] Perform aggregation calculations on each combination to be aggregated to obtain the aggregation result corresponding to each category value.

[0140] In one embodiment, the first region information is the name of the first column, the second region information is the name of the second column, and the aggregation unit 302 is used for:

[0141] Based on the row name of each category value, obtain the set of row names corresponding to each category value;

[0142] Determine the set of row names corresponding to each category value, and the row name of each value to be aggregated;

[0143] The values ​​to be aggregated that belong to the same set of row names are divided into the same aggregation group, and the aggregation group corresponding to each category value is obtained.

[0144] In the data processing method, apparatus, electronic device, and storage medium provided in this application embodiment, the storage layer reads data from the target data block based on the data aggregation instructions of the execution engine to obtain a set of category values. The set of category values ​​is a set of category values ​​obtained by transforming category parameters based on numerical transformation relationships. The numerical transformation relationships are the correspondence between category parameters and values ​​set for the target data block. Based on the category values ​​in the set of category values, the data to be aggregated is grouped and aggregated to obtain the aggregation result corresponding to each category value. Based on the numerical transformation relationships, the category parameters corresponding to each category value are obtained respectively. The category parameters and their corresponding aggregation results are sent to the execution engine. In this way, data grouping and aggregation operations are pushed down to the storage layer, and numerical transformation and grouping aggregation operations are performed based on the numerical transformation relationships set locally corresponding to the data blocks. This allows for the acquisition and uploading of grouping and aggregation results, reducing data transmission volume and improving data processing efficiency.

[0145] This application provides an electronic device, including:

[0146] Processor; and

[0147] The memory stores computer instructions that cause the processor to execute the methods of any of the above-described embodiments.

[0148] This application provides a storage medium storing computer instructions for causing a computer to execute the methods of any of the above embodiments.

[0149] Figure 4 A schematic diagram of the structure of an electronic device 4000 is shown. (See reference...) Figure 4 As shown, the electronic device 4000 includes a processor 4010 and a memory 4020, and optionally may also include a power supply 4030, a display unit 4040, and an input unit 4050.

[0150] The processor 4010 is the control center of the electronic device 4000. It connects various components through various interfaces and lines, and performs various functions of the electronic device 4000 by running or executing software programs and / or data stored in the memory 4020.

[0151] In this embodiment, when the processor 4010 calls the computer program stored in the memory 4020, it executes the steps in the above embodiments.

[0152] Optionally, the processor 4010 may include one or more processing units; preferably, the processor 4010 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 4010. In some embodiments, the processor and memory may be implemented on a single chip; in some embodiments, they may also be implemented separately on independent chips.

[0153] The memory 4020 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, various applications, etc.; the data storage area may store data created based on the use of the electronic device 4000, etc. In addition, the memory 4020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device, etc.

[0154] Electronic device 4000 also includes a power supply 4030 (such as a battery) that supplies power to various components. The power supply can be logically connected to processor 4010 through a power management system, thereby enabling the management of charging, discharging, and power consumption.

[0155] The display unit 4040 can be used to display information input by the user or information provided to the user, as well as various menus of the electronic device 4000. In this embodiment, it is mainly used to display the display interfaces of various applications in the electronic device 4000, as well as text, images, and other objects displayed on the display interfaces. The display unit 4040 may include a display panel 4041. The display panel 4041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0156] The input unit 4050 can be used to receive information such as numbers or category parameters input by the user. The input unit 4050 may include a touch panel 4051 and other input devices 4052. The touch panel 4051, also known as a touch screen, can collect touch operations on or near the user (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 4051).

[0157] Specifically, the touch panel 4051 can detect user touch operations and the signals generated by these operations, convert them into touch point coordinates, send them to the processor 4010, and receive and execute commands from the processor 4010. Furthermore, the touch panel 4051 can be implemented using various types of touch technologies, including resistive, capacitive, infrared, and surface acoustic wave. Other input devices 4052 can include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0158] Of course, the touch panel 4051 can cover the display panel 4041. When the touch panel 4051 detects a touch operation on or near it, it transmits the information to the processor 4010 to determine the type of touch event. Subsequently, the processor 4010 provides corresponding visual output on the display panel 4041 according to the type of touch event. Although in Figure 4 In this embodiment, the touch panel 4051 and the display panel 4041 are two separate components to realize the input and output functions of the electronic device 4000. However, in some embodiments, the touch panel 4051 and the display panel 4041 can be integrated to realize the input and output functions of the electronic device 4000.

[0159] The electronic device 4000 may also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity sensor, etc. Of course, depending on the specific application, the electronic device 4000 may also include other components such as a camera. Since these components are not the focus of this application embodiment, therefore... Figure 4 It is not shown in the text and will not be described in detail here.

[0160] Those skilled in the art will understand that Figure 4 This is merely an example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than shown, or a combination of certain components, or different components.

[0161] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0162] Obviously, the above embodiments are merely examples for clear illustration and are not intended to limit the embodiments. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively list all embodiments here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this application.

Claims

1. A data processing method applied to a storage layer in an electronic device, the method comprising: Based on the data aggregation instructions of the execution engine, data is read from the target data block to obtain a set of category values; The set of category values ​​is a set of category values ​​obtained by transforming category parameters based on numerical transformation relationships; the numerical transformation relationships are the correspondence between category parameters and category values ​​set for the target data block; the electronic device also includes the execution engine; the category parameters are characters to be processed; The category value is obtained by converting the category parameter based on the numerical conversion relationship; Based on the category values ​​in the category value set, the data to be aggregated corresponding to the category value set is divided so that the data to be aggregated corresponding to the same category value is divided into the same aggregation group, so as to obtain the aggregation group corresponding to each category value, and perform aggregation calculation on each aggregation group to obtain the aggregation result corresponding to each category value. Based on the numerical conversion relationship, each category value corresponding to the aggregation result is restored to the category parameter corresponding to the category value; The restored category parameters and their corresponding aggregation results are sent to the execution engine.

2. The method as described in claim 1, before reading data from the target data block based on the data aggregation instructions of the execution engine to obtain the set of category values, the method further includes: When a data storage instruction for the target data block is received, the data to be stored in the data storage instruction is obtained; Based on the numerical conversion relationship, the numerical values ​​of the data to be stored are converted to obtain at least one numerical value corresponding to the data to be stored. The value to be stored is written into the target data block.

3. The method as described in claim 1 or 2, wherein the step of reading data from the target data block based on the data aggregation instructions of the execution engine to obtain a set of category values ​​includes: When it is determined that a data aggregation instruction sent by the execution engine has been received, the first region information in the data aggregation instruction is obtained; The first region information is information indicating the data storage location of the category value set; From the target data block, read the data block data corresponding to the first region information to obtain the category value set.

4. The method of claim 3, further comprising: Based on the second region information also included in the data aggregation instruction, data is read to obtain the data to be aggregated; The second area information is information indicating the data storage location of the data to be aggregated.

5. The method as described in claim 4, wherein the first region information is a first column name, the second region information is a second column name, and the step of dividing the data to be aggregated according to the category value corresponding to each value to be aggregated in the data to be aggregated, to obtain the combination to be aggregated corresponding to each category value, includes: Based on the row name of each category value, obtain the set of row names corresponding to each category value; Determine the set of row names corresponding to each category value, and the row name of each value to be aggregated; The values ​​to be aggregated that belong to the same set of row names are divided into the same aggregation group, and the aggregation group corresponding to each category value is obtained.

6. A data processing apparatus for use in a storage layer of an electronic device, the apparatus comprising: The reading unit is used to read data from the target data block based on the data aggregation instructions of the execution engine to obtain a set of category values; The set of category values ​​is a set of category values ​​obtained by transforming category parameters based on numerical transformation relationships; the numerical transformation relationships are the correspondence between category parameters and category values ​​set for the target data block; the electronic device also includes the execution engine; the category parameters are characters to be processed; The category value is obtained by converting the category parameter based on the numerical conversion relationship; An aggregation unit is used to divide the data to be aggregated corresponding to the category value set according to the category value in the category value set, so as to divide the data to be aggregated corresponding to the same category value into the same aggregation group, obtain the aggregation group corresponding to each category value, and perform aggregation calculation on each aggregation group to obtain the aggregation result corresponding to each category value. The conversion unit is used to restore each category value corresponding to the aggregation result to the category parameter corresponding to the category value based on the numerical conversion relationship. The sending unit is used to send the restored category parameters and their corresponding aggregation results to the execution engine.

7. The apparatus of claim 6, wherein the reading unit is further configured to: When a data storage instruction for the target data block is received, the data to be stored in the data storage instruction is obtained; Based on the numerical conversion relationship, the numerical values ​​of the data to be stored are converted to obtain at least one numerical value corresponding to the data to be stored. The value to be stored is written into the target data block.

8. The apparatus of claim 6 or 7, wherein the reading unit is configured to: When it is determined that a data aggregation instruction sent by the execution engine has been received, the first region information in the data aggregation instruction is obtained; the first region information is information indicating the data storage location of the category value set. From the target data block, read the data block data corresponding to the first region information to obtain the category value set.

9. The apparatus of claim 8, wherein the polymerization unit is further configured to: According to the second region information also included in the data aggregation instruction, data is read to obtain the data to be aggregated; the second region information is information indicating the data storage location of the data to be aggregated.

10. The apparatus of claim 9, wherein the first region information is a first column name, the second region information is a second column name, and the aggregation unit is used for: Based on the row name of each category value, obtain the set of row names corresponding to each category value; Determine the set of row names corresponding to each category value, and the row name of each value to be aggregated; The values ​​to be aggregated that belong to the same set of row names are divided into the same aggregation group, and the aggregation group corresponding to each category value is obtained.

11. An electronic device, comprising: processor; as well as A memory storing computer instructions for causing the processor to perform the method according to any one of claims 1-5.

12. A storage medium storing computer instructions for causing a computer to perform the method according to any one of claims 1-5.

Images