Data processing method and apparatus, computing device, and computer-readable storage medium

Abstract

The embodiment of the present application relates to the technical field of data processing, and discloses a data processing method, which comprises the following steps: receiving an analysis processing request, wherein the analysis processing request comprises a screening condition and analysis requirement information; obtaining target data from a dynamic slope-based storage model according to the screening condition; the dynamic slope-based storage model comprises an upper storage space and a lower storage space which are divided by the dynamic slope; the upper storage space is used for storing high-frequency access data of each data column at a column level, and each row of the lower storage space is used for storing high-frequency access data of a plurality of columns; and performing data processing according to the analysis requirement information and the target data to obtain an analysis processing result. Through the above method, the efficiency of data processing is realized.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data processing, in particular to a data processing method, a data processing device, a computing device and a computer readable storage medium. BACKGROUND

[0002] At present, for massive data to carry out complex multi-dimensional online analysis query processing, the existing technical solution usually adopts columnar storage at the bottom storage layer to replace the row-based data storage of the traditional relational database. However, the present inventors have found in the implementation of the embodiments of the present application that in this scenario, a large amount of rows need to be read, but only a small amount of column data is needed in the multi-data case, and a very high throughput is required for a single query. SUMMARY

[0003] In view of the above problems, the embodiments of the present application provide a data processing method, a data processing device, a computing device and a computer readable storage medium, to solve the technical problem of low data processing efficiency in the prior art.

[0004] According to an aspect of the embodiments of the present application, a data processing method is provided, which comprises:

[0005] receiving an analysis processing request; the analysis processing request comprising a filtering condition and analysis requirement information;

[0006] obtaining target data from a dynamic slope-based storage model according to the filtering condition; the dynamic slope-based storage model comprising an upper storage space and a lower storage space with the dynamic slope as a boundary line; the upper storage space being used to store high-frequency access data of each data column at column level, and each row storage space of the lower storage space being used to store high-frequency access data of a plurality of column combinations;

[0007] performing data processing according to the analysis requirement information and the target data to obtain an analysis processing result.

[0008] In an optional manner, before the receiving an analysis processing request, the method comprises: obtaining original data; constructing a dynamic window space of K times data volume of the original data, K being greater than 1; forming a reference boundary line with the dynamic slope as a reference in the dynamic window space; storing high-frequency access data of each data column at column level in the original data in the upper storage space; storing source data in the original data in the lower storage space below the slope boundary line according to slope gradient to form a source data block; storing high-frequency access data of a plurality of column combinations below the source data block in the lower storage space.

[0009] In an optional mode, after the target data is obtained from the storage model based on the dynamic slope according to the analysis processing request, the method further comprises: updating high-frequency access data in the storage model; and adjusting data positions in the storage model according to the high-frequency access data, so as to adjust the dynamic slope of the storage model.

[0010] In an optional mode, the method further comprises: adjusting data in the upper storage space and the lower storage space according to behavior trend analysis; and adjusting the dynamic slope in real time according to the adjusted data in the upper storage space and the lower storage space.

[0011] In an optional mode, the obtaining of the target data from the storage model based on the dynamic slope according to the screening condition comprises: analyzing the screening condition to obtain data column information, a screening range and a calculation screening condition required by a task calculation; locating to one or more specific rows in the lower storage space in gradient order according to the data column information; sequentially obtaining row data of the corresponding specific rows and comparing the row data with the calculation screening condition; if the row data of the corresponding specific rows meet the calculation screening condition, determining row data in a target row as the target data; and the target behavior meets the row data corresponding to the specific row.

[0012] In an optional mode, after the sequential obtaining of the row data of the corresponding specific rows and the comparison of the row data with the calculation screening condition, the method further comprises: if the calculation screening condition is not met, locating to one or more target column queues in the upper storage space; concurrently obtaining data from the head of the target column queue; comparing the obtained data with the calculation screening condition; if the comparison of the obtained data with the calculation screening condition meets the calculation screening condition, continuing to compare data values in the target column queue until the screening range is met, so as to obtain the target data.

[0013] In an optional mode, after the comparison of the obtained data with the calculation screening condition, the method further comprises: if the comparison of the obtained data with the calculation screening condition does not meet the calculation screening condition, concurrently obtaining an entire column data block from the upper storage space; and scanning the entire column data block to obtain the target data meeting the calculation screening condition in the screening range.

[0014] According to another aspect of an embodiment of the present application, a data processing apparatus is provided, comprising:

[0015] a receiving module configured to receive an analysis processing request, wherein the analysis processing request comprises a screening condition and analysis requirement information;

[0016] The acquisition module is configured to acquire target data from a dynamic slope-based storage model according to the screening condition; the dynamic slope-based storage model comprises an upper storage space and a lower storage space which are divided by the dynamic slope; the upper storage space is configured to store high-frequency access data of each data column at a column level, and each row of the lower storage space is configured to store high-frequency access data of a plurality of column combinations.

[0017] The analysis module is configured to perform data processing according to the analysis requirement information and the target data, and obtain an analysis processing result.

[0018] According to another aspect of the embodiment of the present application, a computing device is provided, comprising a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus;

[0019] The memory is configured to store at least one executable instruction, and the executable instruction causes the processor to perform the operation of the data processing method.

[0020] According to still another aspect of the embodiment of the present application, a computer readable storage medium is provided, the storage medium stores at least one executable instruction, and the executable instruction causes the computing device to perform the operation of the data processing method when the computing device runs.

[0021] The embodiment of the present application receives an analysis processing request, acquires target data based on a dynamic slope-based storage model according to the analysis processing request. The dynamic slope-based storage model comprises an upper storage space and a lower storage space which are divided by the dynamic slope; the upper storage space is configured to store high-frequency access data of each data column at a column level, and each row of the lower storage space is configured to store high-frequency access data of a plurality of column combinations. Then, data processing is performed according to the analysis processing request and the target data, and an analysis processing result is obtained. Through the embodiment of the present application, the dynamic slope can be dynamically adjusted, and the storage space of high-frequency screening or aggregated data can be changed. The high-frequency access data at the column level and the high-frequency access data of a plurality of column combinations at the row level are stored based on the upper and lower boundaries of the slope reference, so as to reduce the scanning operation existing in the original columnar storage data model, thereby further improving the efficiency and the timeliness of interactive response in the complex real-time online analysis query processing scene of massive data, and effectively improving the efficiency of data processing.

[0022] The above description is only a summary of the technical scheme of the embodiment of the present application, in order to more clearly understand the technical means of the embodiment of the present application, the embodiment of the present application can be implemented according to the content of the specification, and in order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific embodiment of the present application is described below. Attached Figure Description

[0023] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0024] Figure 1 A flowchart illustrating the data processing method provided in an embodiment of the present invention is shown;

[0025] Figure 2 A flowchart illustrating existing data storage methods is shown;

[0026] Figure 3 This diagram illustrates the structure of the storage model in the data processing method provided by an embodiment of the present invention.

[0027] Figure 4 This diagram illustrates the dynamic adjustment of the slope of the storage model in the data processing method provided by an embodiment of the present invention.

[0028] Figure 5 A schematic diagram of the storage model update in the data processing method provided in an embodiment of the present invention is shown;

[0029] Figure 6 A flowchart illustrating another embodiment of the data processing method of the present invention is shown;

[0030] Figure 7 A schematic diagram of the structure of the data processing apparatus provided in an embodiment of the present invention is shown;

[0031] Figure 8 A schematic diagram of the structure of a computing device provided in an embodiment of the present invention is shown. Detailed Implementation

[0032] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0033] In the field of big data, complex multidimensional online analysis and query processing of massive amounts of data is extremely important, and the requirements for real-time performance and timely interaction are constantly increasing. Efficient and complex analysis and query processing of massive amounts of multidimensional, high-cardinality, and long-term data still presents technical challenges. Online big data analysis and processing typically involves filtering or aggregating a small portion of data from a large dataset. The amount of data being analyzed and processed is significantly smaller than the source data, and processing a single query usually requires extremely high throughput.

[0034] like Figure 2As shown in 'a', white represents empty data blocks, while light gray and dark gray represent data blocks that actually store data. Dark gray represents the data blocks that actually need to be retrieved. When analyzing, querying, filtering, or aggregating partial rows of data in a few columns, it is necessary to scan the entire row to read all column data and then filter out the data from different rows of the specified column. The actual amount of data retrieved is usually much less than the amount of data retrieved by scanning, resulting in poor performance in large-scale big data scenarios.

[0035] For online analysis and querying scenarios with this scale of data, existing technologies typically replace traditional row-based relational databases with column-based storage at the underlying storage layer. This is because such scenarios require reading a large number of rows, but only a few columns, and individual queries often require extremely high throughput. Therefore, column-based storage can effectively meet these needs. However, looking further ahead, facing massive amounts of data and increasingly complex big data analysis and query processing, as well as the demand for more efficient processing, column-based storage models still involve a large number of unavoidable scan operations in many advanced scenarios. This leads to unnecessary performance overhead, especially in complex data analysis scenarios with huge data scales, multiple dimensions, high cardinality, and long time spans. Column-based storage is beginning to face bottlenecks and struggles to meet the demands of higher real-time data analysis.

[0036] like Figure 2 As shown in b, white represents empty data blocks, while light gray and dark gray represent data blocks containing actual stored data. Dark gray represents the data blocks that actually need to be retrieved. In columnar storage, for analysis queries, filtering, or aggregation of specific columns, the corresponding entire column of data can be directly retrieved. Compared to row-based storage, this avoids scanning unnecessary row-level and column-level data. However, in advanced analysis scenarios, columnar storage still cannot avoid data scanning when analyzing and processing entire columns of data, resulting in performance overhead and bottlenecks in large-scale, complex data analysis scenarios.

[0037] To address the aforementioned issues, this application proposes a novel data storage model based on dynamic slope. Based on this, a data processing method is proposed, and a specific algorithm is combined to dynamically adjust the slope, thereby altering the storage space for frequently filtered or aggregated data. Based on the upper and lower boundaries of the slope benchmark, frequently accessed data at the column level and frequently accessed data combining multiple columns at the row level are stored separately. This reduces the scanning operations present in the original column-based data storage model, thereby further improving efficiency and the timeliness of interactive responses in complex real-time online analysis and query processing scenarios involving massive amounts of data.

[0038] Figure 1 A flowchart of a data processing method provided in an embodiment of the present invention is shown. This method is executed by a computing device. The computing device may be a computer, a terminal, a smart device, etc. Figure 1As shown, the method includes the following steps:

[0039] Step 110: Receive analysis and processing request.

[0040] The analysis and processing request is a request initiated by the user on the client side. This request can be for analyzing and processing certain data, or it can be an SQL statement. For example, the analysis and processing request could be a request to analyze user profiles based on data, and it would include filtering conditions and analysis requirements. Alternatively, it could be an analysis request to update the dynamic slope of a storage model.

[0041] In this embodiment of the invention, after receiving the analysis and processing request, the analysis and processing request is parsed and converted into corresponding execution instructions, such as... Figure 6 As shown, the query analysis parser parses the analysis and processing request into multiple task execution instructions, and sends them to multiple distributed concurrent analysis and processing executors. These multiple analysis and processing executors concurrently process and execute their respective tasks according to their task execution instructions.

[0042] Step 120: Obtain target data based on the storage model of dynamic slope according to the filtering conditions.

[0043] In this embodiment of the invention, according to the analysis and processing request, multiple analysis and processing executors concurrently process and execute corresponding tasks according to their respective task execution instructions. The corresponding tasks are query analysis tasks, and each query analysis task includes corresponding filtering conditions and analysis requirement information. When a query is required, the target data is obtained based on the storage model of dynamic slope.

[0044] Among them, such as Figure 3 As shown, the storage model based on dynamic slope in this embodiment of the invention includes an upper storage space and a lower storage space with the dynamic slope as the dividing line; the upper storage space is used to store high-frequency access data of each data column at the column level, and each row of the lower storage space is used to store high-frequency access data of multiple column combinations.

[0045] In the embodiment of the present application, the construction process of the storage model based on the dynamic slope includes: obtaining original data; constructing a dynamic window space of K times (K>1) storage space of the original data; forming a reference demarcation line in the dynamic window space with the dynamic slope as a reference; storing high-frequency access data of each data column in the original data at the column level in the upper storage space; storing source data in the original data in the lower storage space below the slope demarcation line according to the slope gradient to form a source data block; and storing high-frequency access data of multiple columns below the source data block in the lower storage space, wherein the high-frequency access data is determined according to historical query analysis, and in the embodiment of the present application, the high-frequency access data can be determined according to the LRU (Least Recently Used) algorithm. The dynamic slope is composed of a vector composed of a plurality of two-dimensional coordinate values and pointer index values of a plurality of storage data blocks on the reference demarcation line in the dynamic window space of the storage model. The two-dimensional coordinate values are the position coordinates of the data blocks in the storage model, and the index values are the position coordinates of the previous data block in the dynamic slope. As shown in Figure 4 FIG. 1, wherein Xa1 represents the x-coordinate value of the a1 data block, Ya1 represents the y-coordinate value of the a1 data block, Ia1 represents the index value of the next data block pointed to by the a1 data block, and so on, thereby forming a vector composed of a plurality of two-dimensional coordinate values and pointer index values of a plurality of storage data blocks on the reference demarcation line in the dynamic window space of the storage model. It can be understood that the dynamic slope is only a figurative name, and it does not mean that the dynamic slope must be a straight line. It can also be a polyline or a multi-polyline.

[0046] Specifically, according to the amount of original data, K times (K>=2) storage space of the existing data amount is constructed as a dynamic window space. The storage space is dynamically adjusted and changed, and is not one-time fixed full storage. Therefore, it is elastically expandable, but the overall space amount will not exceed K times of the original data amount, and the storage space of the entire dynamic window will be continuously corrected and incrementally expanded with the dynamic increase of the original data amount. As shown in Figure 3As shown, a reference dividing line is formed in the dynamic window space with the dynamic slope as the reference, and the blank area above the dividing line is used as the storage space for high-frequency access data at the column level. Below the dividing line, there are specific source data according to the slope gradient, and below the source data block, there is a blank area inside the dynamic window, which is used as the storage space for high-frequency access data at the row level. These special storage spaces are collectively referred to as upper / lower dividing line storage spaces, which are divided into upper storage space and lower storage space. The upper storage space forms a multi-column vertical queue at the column level, which determines the hot data of high-frequency access in each column queue through data analysis and query of past periods (especially filtering or aggregation, deduplication, and other types of analysis and calculation for specific columns), so that the access frequency is stored in order. And according to the LRU (Least Recently Used) algorithm, it is replaced or the queue is expanded (involving slope change). The lower storage space forms a multi-row gradient combination at the row level, which is used for data analysis and query of past periods (especially filtering or aggregation, deduplication, and other types of analysis and calculation for multi-dimensional or multiple column level combined data), and each row storage space stores high-frequency access data of multiple column combinations, and dynamically replaces hot row level data according to gradient combination, while dynamically expanding storage space (involving slope change). That is to say, in the upper storage space, the data stored is the data stored by column for each dimension; in the lower storage space, after filtering, aggregation, and other processing of the original data, the data stored is the data stored by row for multiple dimensions or multiple column level combined data. In most analysis scenarios, the amount of data actually processed and obtained through filtering, aggregation, and other analysis and calculation methods will be much smaller than the actual overall data amount of the source data, and usually the analysis has convergence or focus. The upper and lower dividing line storage spaces usually do not exceed the original data amount as a whole, and are conducive to efficient direct data acquisition for analysis query, which can avoid unnecessary performance loss caused by scanning and acquiring data at the column level or row level storage space level.

[0047] In the dynamic slope-based storage model query, in order to obtain the target data, since the analysis processing request generally involves multiple dimensions and multiple column-level combined data, the target data is generally obtained by querying the lower storage space first; and the data not obtained in the lower storage space is obtained from the upper storage space. Specifically, first, the filtering condition is analyzed to obtain the data column information, filtering range and calculation filtering condition required for task calculation; one or more specific rows are located in the lower storage space in gradient order according to the data column information, wherein before locating one or more specific rows in the lower storage space in gradient order according to the data column information, the dynamic slope of the storage model needs to be analyzed first to obtain the data block on the dynamic slope (reference boundary line), so as to locate the row and column positions taking these data blocks as the base points, and one or more specific rows are located in the lower storage space in gradient order according to the current task instruction to be executed by the analysis processing executor. Then the row data of the corresponding specific row is obtained in sequence and compared with the calculation filtering condition; if the row data of the corresponding specific row meets the calculation filtering condition, the row data in the target row is determined as the target data, and the target behavior is the specific row corresponding to the row data meeting the calculation filtering condition. If the calculation filtering condition is not met, that is, the target data is not obtained or all target data is not completely obtained in the lower storage space, one or more target column queues corresponding to the upper storage space are located, and data is obtained from the head of the target column queue in sequence; the obtained data is compared with the calculation filtering condition; if the obtained data meets the calculation filtering condition, the data values in the target column queue are compared until the filtering range is met, and the target data is obtained. If the obtained data does not meet the calculation filtering condition, the whole column data block is obtained from the upper storage space in parallel; the target data meeting the calculation filtering condition in the filtering range is obtained by scanning the whole column data block.

[0048] In the embodiment of the application, the dynamic slope of the storage model is dynamically adjusted, specifically, the data of the upper storage space and the lower storage space is adjusted according to the analysis processing behavior trend; and the dynamic slope is adjusted in real time according to the data of the adjusted upper storage space and lower storage space. Figure 5 and Figure 6As shown, in this embodiment of the invention, according to a preset period, frequently accessed hot data in the upper and lower storage spaces are updated using the LRU algorithm, thereby replacing the data order or expanding the data queue in the upper and / or lower storage spaces. For example, after querying the vertical column queue in the upper storage space, the corresponding data in the corresponding column vertical queue can be replaced according to the LRU (Least Recently Used) algorithm principle and sorted by the most recent access time as the weight, and placed into data blocks from the front to the back of the queue in order of access time. Simultaneously, if there are multiple column combinations, the corresponding row in the lower storage space will be located sequentially according to the column position and gradient order, and all columns in that row except for one or more columns containing the source data will be replaced, thereby updating the data in the lower storage space. Through this periodic update, the dynamic slope of the storage model changes, such as... Figure 5 As shown.

[0049] like Figure 6 As shown, the storage model strategy adjustment device can establish communication with one or more analysis processing executors, receive analysis processing behavior signals, and thus construct the behavior trend of analysis queries. The storage model strategy adjustment device asynchronously monitors the storage and replacement status of the upper and lower storage spaces within the dynamic window space and adjusts the dynamic slope in real time. Specifically, it can be divided into the following cases:

[0050] 1) If the upper or lower storage space is full, but the dynamic window space is still less than K times (K>=2) the storage space of the existing data, and the replacement frequency continues to increase, then expand the corresponding upper or lower storage space, thereby indirectly changing the relative position of the baseline boundary line (dynamic slope).

[0051] 2) Under the conditions in 1), when the current dynamic window space is close to or exceeds K times (K>=2) the storage space of the existing data volume, based on the received analysis and processing behavior signaling and the replacement rate of the corresponding row or column in the upper or lower storage space, the data block of the baseline boundary (dynamic slope) and the data block below the corresponding column of that data block are moved. For example, if the replacement rate of the upper storage space continues to increase or one or more columns have recently had continuous analysis and processing query behavior, then the data blocks of these columns are moved downwards, so that the vertical queue space of the corresponding column in the upper boundary storage space is expanded. Similarly, if the replacement rate of the lower storage space continues to increase or one or more row-level analysis and processing query behavior formed by the corresponding multi-column combination has recently had continuous occurrence, then the corresponding multi-column data blocks are moved upwards, so that the gradient space of the multi-column combination of the corresponding column in the lower storage space is expanded. Figure 5As shown, while the data blocks are moved, the corresponding index pointers are changed, thereby jointly changing the overall reference boundary line (dynamic slope), and the latest dynamic slope is calculated according to the two-dimensional position of the data blocks on the reference boundary line in the entire dynamic window space (embodied as a vector composed of a two-dimensional coordinate value and an index value). The storage model strategy adjustment device returns the dynamically updated dynamic slope to the analysis processing executor for processing.

[0052] Step 130: performing data processing according to the analysis requirement information and the target data to obtain an analysis processing result.

[0053] Wherein, after obtaining the target data, each analysis processing executor performs calculation according to the obtained target data and corresponding task execution instruction, and sends the calculation result to the analysis processing aggregator. After gathering all data, the analysis processing aggregator performs iterative aggregation, and continuously communicates with the multiple analysis processing executors to perform multiple iterative aggregation calculations, thereby obtaining the analysis processing result. Each task execution instruction is an instruction allocated to each analysis processing executor according to the analysis requirement information.

[0054] The embodiment of the present application receives an analysis processing request, and obtains target data based on a storage model of a dynamic slope according to the analysis processing request. Wherein, the storage model of the dynamic slope includes an upper storage space and a lower storage space with the dynamic slope as a boundary line; the upper storage space is used to store high-frequency access data of each data column at a column level, and each row storage space of the lower storage space is used to store high-frequency access data of multiple column combinations. Then, data processing is performed according to the analysis processing request and the target data to obtain an analysis processing result. Through the embodiment of the present application, the dynamic slope can be dynamically adjusted, and the storage space of high-frequency screening or aggregation data can be changed. The high-frequency access data at the column level and the high-frequency access data of multiple column combinations at the row level are stored based on the upper and lower boundaries of the slope reference, so as to reduce the scanning operation existing in the original columnar storage data model, thereby further improving the efficiency and timeliness of interactive response in the complex real-time online analysis query processing scene of massive data, and effectively improving the efficiency of data processing.

[0055] Figure 7 The structure schematic diagram of the data processing device provided by the embodiment of the present application is shown. As shown in the figure, Figure 7 The device 200 includes:

[0056] The receiving module 210 is configured to receive an analysis processing request; the analysis processing request includes a screening condition and analysis requirement information;

[0057] The acquisition module 220 is configured to acquire target data from the dynamic slope-based storage model according to the screening condition; the dynamic slope-based storage model comprises an upper storage space and a lower storage space which are divided by the dynamic slope; the upper storage space is used to store high-frequency access data of each data column at a column level, and each row storage space of the lower storage space is used to store high-frequency access data of a plurality of column combinations.

[0058] The analysis module 230 is configured to perform data processing according to analysis requirement information and the target data, and obtain an analysis processing result.

[0059] In an optional manner, before the analysis processing request is received, the method comprises: acquiring original data; constructing a dynamic window space of K times storage space of the data volume of the original data, K>1; forming a reference boundary line with the dynamic slope as a reference in the dynamic window space; storing high-frequency access data of each data column at a column level in the original data in the upper storage space; storing source data in the original data below the slope boundary line in the slope gradient in the lower storage space to form a source data block; and storing high-frequency access data of a plurality of column combinations below the source data block in the lower storage space.

[0060] In an optional manner, after the target data is acquired from the dynamic slope-based storage model according to the analysis processing request, the method further comprises: updating high-frequency access data in the storage model; and adjusting a data position in the storage model according to the high-frequency access data to adjust the dynamic slope of the storage model.

[0061] In an optional manner, the method further comprises: adjusting data of the upper storage space and the lower storage space according to an analysis processing behavior trend; and adjusting the dynamic slope in real time according to the adjusted data of the upper storage space and the lower storage space.

[0062] In an optional manner, the target data is acquired from the dynamic slope-based storage model according to the screening condition, which comprises: analyzing the screening condition to acquire data column information, a screening range and a calculation screening condition required for task calculation; locating to one or more specific rows in the gradient order from the lower storage space according to the data column information; sequentially acquiring row data of the corresponding specific rows and comparing with the calculation screening condition; if the row data of the corresponding specific rows meet the calculation screening condition, determining the row data in the target row as the target data; and the specific row corresponding to the row data of the target behavior meeting the calculation screening condition.

[0063] In an alternative manner, after the corresponding row data of a specific row is sequentially obtained and compared with the calculation filtering condition, if the calculation filtering condition is not met, the corresponding one or more target column queues are located from the upper storage space; data is sequentially obtained from the head of the target column queue; the obtained data is compared with the calculation filtering condition; if the obtained data compared with the calculation filtering condition meets the calculation filtering condition, the data values in the target column queue are continuously compared until the filtering range is met, and the target data is obtained.

[0064] In an alternative manner, after the obtained data is compared with the calculation filtering condition, if the obtained data compared with the calculation filtering condition does not meet the calculation filtering condition, the whole column data block is concurrently obtained from the upper storage space; the target data meeting the calculation filtering condition in the filtering range is obtained by scanning the whole column data block.

[0065] The specific structure of the data processing device of the embodiment of the present application is generally consistent with the specific execution steps of the data processing method of the above-mentioned embodiment, and will not be described here.

[0066] The embodiment of the present application receives an analysis processing request, and obtains target data based on a dynamic slope-based storage model according to the analysis processing request. The dynamic slope-based storage model includes an upper storage space and a lower storage space with the dynamic slope as a boundary line. The upper storage space is used to store high-frequency access data of each data column at a column level, and each row storage space of the lower storage space is used to store high-frequency access data of a plurality of column combinations. Then, data processing is performed according to the analysis processing request and the target data to obtain an analysis processing result. Through the embodiment of the present application, the dynamic slope can be dynamically adjusted, and the storage space of high-frequency filtering or aggregated data can be changed. The high-frequency access data at the column level and the high-frequency access data of the plurality of column combinations at the row level are respectively stored based on the upper and lower boundaries of the slope reference, so as to reduce the scanning operation existing in the original columnar storage data model, thereby further improving the efficiency and timeliness of interactive response in the complex real-time online analysis query processing scene of massive data, and effectively improving the efficiency of data processing.

[0067] Figure 8 The structure schematic diagram of the computing device provided by the embodiment of the present application is shown, and the specific implementation of the computing device is not limited by the embodiment of the present application.

[0068] As Figure 8As shown, the computing device can include a processor 302, a communications interface 304, a memory 306, and a communications bus 308.

[0069] The processor 302, the communications interface 304, and the memory 306 communicate with each other through the communications bus 308. The communications interface 304 is configured to communicate with network elements such as clients or other servers. The processor 302 is configured to execute the program 310, and in particular, can execute the steps in the above-described data processing method embodiments.

[0070] In particular, the program 310 can include program code including computer-executable instructions.

[0071] The processor 302 can be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present application. The one or more processors included in the computing device can be the same type of processor, such as one or more CPUs, or different types of processors, such as one or more CPUs and one or more ASICs.

[0072] The memory 306 is configured to store the program 310. The memory 306 can include a high-speed RAM memory, and can also include a non-volatile memory, such as at least one disk memory.

[0073] The program 310 can be specifically invoked by the processor 302 to cause the computing device to perform the following operations:

[0074] Receive an analysis processing request; the analysis processing request includes a filtering condition and analysis requirement information;

[0075] According to the filtering condition, obtain target data from a dynamic slope-based storage model; the dynamic slope-based storage model includes an upper storage space and a lower storage space divided by the dynamic slope; the upper storage space is configured to store high-frequency access data of each data column at a column level, and each row of the lower storage space is configured to store high-frequency access data of a plurality of column combinations;

[0076] According to the analysis requirement information and the target data, perform data processing to obtain an analysis processing result.

[0077] In an alternative way, before the receiving the analysis processing request, the method comprises: acquiring original data; constructing a dynamic window space of K times storage space of the original data, K>1; forming a reference demarcation line in the dynamic window space with the dynamic slope as a reference; storing high-frequency access data of each data column in the original data at a column level in the upper storage space; storing source data in the original data in the lower storage space below the slope demarcation line according to the slope gradient to form a source data block; and storing high-frequency access data of multiple column combinations below the source data block in the lower storage space.

[0078] In an alternative way, after the acquiring target data from the storage model based on the dynamic slope according to the analysis processing request, the method further comprises: updating high-frequency access data in the storage model; and adjusting data positions in the storage model according to the high-frequency access data to adjust the dynamic slope of the storage model.

[0079] In an alternative way, the method further comprises: adjusting data in the upper storage space and the lower storage space according to an analysis processing behavior trend; and adjusting the dynamic slope in real time according to the adjusted data in the upper storage space and the lower storage space.

[0080] In an alternative way, the acquiring target data from the storage model based on the dynamic slope according to the filtering condition comprises: analyzing the filtering condition to acquire data column information, a filtering range, and a calculation filtering condition required for task calculation; locating to one or more specific rows in the lower storage space according to the data column information in gradient order; sequentially acquiring row data of the corresponding specific rows and comparing the row data with the calculation filtering condition; if the row data of the corresponding specific rows meet the calculation filtering condition, determining row data in a target row as the target data; and the target behavior meets the calculation filtering condition.

[0081] In an alternative way, after the sequentially acquiring row data of the corresponding specific rows and comparing the row data with the calculation filtering condition, the method further comprises: if the calculation filtering condition is not met, locating to one or more target column queues in the upper storage space; concurrently acquiring data from the head of the target column queue; comparing the acquired data with the calculation filtering condition; if the comparison of the acquired data with the calculation filtering condition meets the calculation filtering condition, continuing to compare data values in the target column queue until the filtering range is met to obtain the target data.

[0082] In an alternative manner, after the acquired data is compared with the calculated filtering condition, if the acquired data does not meet the calculated filtering condition, the entire column data block is concurrently acquired from the upper storage space; and the target data meeting the calculated filtering condition in the filtering range is acquired by scanning the entire column data block.

[0083] The embodiment of the present application receives an analysis processing request, acquires target data based on a dynamic slope-based storage model according to the analysis processing request. The dynamic slope-based storage model includes an upper storage space and a lower storage space with the dynamic slope as a demarcation line. The upper storage space is used to store high-frequency access data of each data column at a column level, and each row of the lower storage space is used to store high-frequency access data of a plurality of column combinations. Then, data processing is performed according to the analysis processing request and the target data to obtain an analysis processing result. Through the embodiment of the present application, the dynamic slope can be dynamically adjusted, and the storage space of high-frequency filtering or aggregated data can be changed. The high-frequency access data at the column level and the high-frequency access data of the plurality of column combinations at the row level are stored based on the upper and lower demarcation lines of the slope reference, so as to reduce the scanning operation existing in the original columnar storage data model, thereby further improving the efficiency and the timeliness of interactive response in the complex real-time online analysis query processing scenario of massive data, and effectively improving the data processing efficiency.

[0084] The embodiment of the present application provides a computer readable storage medium, and the storage medium stores at least one executable instruction. When the executable instruction runs on a computing device, the computing device executes the data processing method in any method embodiment.

[0085] The executable instruction can be specifically used to make the computing device execute the following operations:

[0086] An analysis processing request is received. The analysis processing request includes filtering conditions and analysis requirement information.

[0087] Target data is acquired from a dynamic slope-based storage model according to the filtering conditions. The dynamic slope-based storage model includes an upper storage space and a lower storage space with the dynamic slope as a demarcation line. The upper storage space is used to store high-frequency access data of each data column at a column level, and each row of the lower storage space is used to store high-frequency access data of a plurality of column combinations.

[0088] Data processing is performed according to the analysis requirement information and the target data to obtain an analysis processing result.

[0089] In an alternative, before the receiving the analysis processing request, the method comprises: obtaining original data; constructing a dynamic window space of K times storage space of the original data, K>1; forming a reference demarcation line in the dynamic window space with the dynamic slope as a reference; storing high-frequency access data of each data column in the original data at a column level in the upper storage space; storing source data in the original data in the lower storage space below the slope demarcation line according to slope gradient to form a source data block; and storing high-frequency access data of multiple column combinations below the source data block in the lower storage space.

[0090] In an alternative, after the obtaining target data from the storage model based on dynamic slope according to the analysis processing request, the method further comprises: updating high-frequency access data in the storage model; and adjusting data positions in the storage model according to the high-frequency access data to adjust the dynamic slope of the storage model.

[0091] In an alternative, the method further comprises: adjusting data in the upper storage space and the lower storage space according to analysis processing behavior trends; and adjusting the dynamic slope in real time according to the adjusted data in the upper storage space and the lower storage space.

[0092] In an alternative, the obtaining target data from the storage model based on dynamic slope according to the filtering condition comprises: analyzing the filtering condition to obtain data column information, filtering range and calculation filtering condition required for task calculation; locating to one or more specific rows in the lower storage space according to gradient order according to the data column information; sequentially obtaining row data of the corresponding specific rows and comparing with the calculation filtering condition; if the row data of the corresponding specific rows meet the calculation filtering condition, determining row data in a target row as the target data; and the target behavior meets the row data corresponding to the specific row.

[0093] In an alternative, after the sequentially obtaining row data of the corresponding specific rows and comparing with the calculation filtering condition, the method further comprises: if the calculation filtering condition is not met, locating to one or more target column queues in the upper storage space; concurrently obtaining data from the head of the target column queue; comparing the obtained data with the calculation filtering condition; if the comparison of the obtained data with the calculation filtering condition meets the calculation filtering condition, continuing to compare data values in the target column queue until the filtering range is met to obtain the target data.

[0094] In an alternative manner, after the acquired data is compared with the calculated screening condition, if the acquired data does not meet the calculated screening condition, the entire column data block is concurrently acquired from the upper storage space; and the target data meeting the calculated screening condition in the screening range is acquired by scanning the entire column data block.

[0095] The embodiment of the present application receives an analysis processing request, and acquires target data based on a dynamic slope-based storage model according to the analysis processing request. The dynamic slope-based storage model includes an upper storage space and a lower storage space, which are divided by the dynamic slope. The upper storage space is used to store high-frequency access data of each data column at the column level, and each row of the lower storage space is used to store high-frequency access data of a plurality of columns. Then, data processing is performed according to the analysis processing request and the target data, and an analysis processing result is obtained. Through the embodiment of the present application, the dynamic slope can be dynamically adjusted, and the storage space of high-frequency screening or aggregated data can be changed. The high-frequency access data at the column level and the high-frequency access data of a plurality of columns at the row level are stored based on the upper and lower boundaries of the slope, so as to reduce the scanning operation in the original columnar storage data model, thereby further improving the efficiency and the timeliness of interactive response in the complex real-time online analysis query processing scenario of massive data, and effectively improving the data processing efficiency.

[0096] The embodiment of the present application provides a data processing device for executing the above-mentioned data processing method.

[0097] The embodiment of the present application provides a computer program, which can be called by a processor to make a computing device execute the data processing method in any method embodiment.

[0098] The embodiment of the present application provides a computer program product, which includes a computer program stored on a computer readable storage medium. The computer program includes program instructions, which, when executed on a computer, cause the computer to execute the data processing method in any method embodiment.

[0099] The algorithms and displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems can be used with these teachings, or with modifications that are within the scope of the present application. In accordance with the foregoing description, various aspects of the present application are constructed in a manner that is most effective for achieving the intended result. In addition, the present application is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the present application as described herein, and any references below to specific languages are provided for disclosure of enablement of the best mode of the present application.

[0100] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order not to obscure the understanding of this description.

[0101] Similarly, it is to be understood that the embodiments of the application can be adapted to other applications and that their features can be interchanged in any way. In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order not to obscure the understanding of this description.

[0102] Those skilled in the art will appreciate that modules in the apparatuses in the embodiments can be adapted and arranged in one or more apparatuses other than the embodiments. Modules or units or components in the embodiments can be combined into one module or unit or component, and can be divided into a plurality of sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, all combinations of all features disclosed in this specification (including the accompanying claims, abstract and drawings), and all processes or units of any methods or apparatuses disclosed thus can be adopted. Unless explicitly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) can be replaced by an alternative feature providing the same, equivalent or similar function.

[0103] It is to be noted that the above-mentioned embodiments illustrate rather than limit the application, and that one skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps other than those listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim, several devices can be listed, even though they are not, in fact, incoporated into a single physical device. The use of the word 'at least' followed by a list of one or more items does not exclude additional such items. The use of the terms 'first','second' and 'third', etc. does not limit the scope of the application, but merely identifies a name of an element. Steps in the methods provided herein do not have to be performed in the exact order disclosed, unless explicitly stated.

Claims

1. A data processing method, characterized by, The method comprises: receiving an analysis processing request; the analysis processing request comprises a screening condition and analysis requirement information; acquiring target data from a dynamic slope-based storage model according to the screening condition; the dynamic slope-based storage model comprises an upper storage space and a lower storage space divided by the dynamic slope; the upper storage space is used to store high-frequency access data of each data column at a column level, and each row of the lower storage space is used to store high-frequency access data of a plurality of columns; the dynamic slope is adjusted in real time according to data of the upper storage space and the lower storage space adjusted according to an analysis processing behavior trend; performing data processing according to the analysis requirement information and the target data to obtain an analysis processing result.

2. The method of claim 1, wherein, Before the receiving of the analysis processing request, the method comprises: acquiring original data; constructing a dynamic window space with a data volume K times of the original data, K being greater than 1; forming a reference boundary line with the dynamic slope as a reference in the dynamic window space; storing high-frequency access data of each data column at a column level in the original data in the upper storage space; storing source data in the original data below the slope boundary line in the lower storage space to form a source data block according to a slope gradient; storing high-frequency access data of a plurality of columns below the source data block in the lower storage space.

3. The method of claim 2, wherein, After the acquiring of the target data from the dynamic slope-based storage model according to the analysis processing request, the method further comprises: updating high-frequency access data in the storage model; adjusting data positions in the storage model according to the high-frequency access data to adjust the dynamic slope of the storage model.

4. The method according to any one of claims 1 to 3, characterized in that, The dynamic slope is formed by a vector composed of a plurality of two-dimensional coordinate values and pointer index values of a plurality of storage data blocks on the reference boundary line in the dynamic window space of the storage model; the two-dimensional coordinate values are position coordinates of the storage data blocks in the storage model, and the index values are position coordinates of a previous storage data block in the dynamic slope.

5. The method of claim 4, wherein, The acquiring of the target data from the dynamic slope-based storage model according to the screening condition comprises: analyzing the screening condition to acquire data column information, a screening range and a calculation screening condition required for task calculation; positioning to one or more specific rows in a gradient order from the lower storage space according to the data column information; sequentially acquiring row data of the specific rows and comparing the row data with the calculation screening condition; if the row data of the specific rows meet the calculation screening condition, determining row data in a target row as the target data; the target behavior meets the row data corresponding to the specific row of the calculation screening condition.

6. The method of claim 5, wherein, After the sequentially acquiring of the row data of the specific rows and the comparing of the row data with the calculation screening condition, the method further comprises: if the calculation screening condition is not met, positioning to one or more target column queues from the upper storage space; sequentially acquiring data from the head of the target column queue in parallel; comparing the acquired data with the calculation screening condition; If the acquired data meets the calculated screening condition in comparison, the data values in the target column queue are continuously compared until the screening range is met, and the target data is obtained.

7. The method of claim 6, wherein, After the acquired data is compared with the calculated screening condition, the following is included: If the acquired data does not meet the calculated screening condition in comparison, the entire column data block is concurrently acquired from the upper storage space; The target data meeting the calculated screening condition in the screening range is scanned and acquired from the entire column data block.

8. A data processing apparatus, characterized by, The device includes: A receiving module for receiving an analysis processing request; the analysis processing request includes a screening condition and analysis requirement information; An acquiring module for acquiring target data from a dynamic slope-based storage model according to the screening condition; the dynamic slope-based storage model includes an upper storage space and a lower storage space with the dynamic slope as a boundary line; the upper storage space is used to store high-frequency access data of each data column at the column level, and each row of the lower storage space is used to store high-frequency access data of multiple column combinations; the dynamic slope is adjusted in real time according to the data of the upper storage space and the lower storage space adjusted according to the analysis processing behavior trend; An analysis module for performing data processing according to the analysis requirement information and the target data to obtain an analysis processing result.

9. A computing device, comprising: It includes: A processor, a memory, a communication interface, and a communication bus, which complete mutual communication through the communication bus; The memory is used to store at least one executable instruction, which makes the processor execute the operations of the data processing method in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when running on a computing device, makes the computing device execute the operations of the data processing method in any one of claims 1-7.

Images