Method, device, electronic equipment, storage medium and product for data processing

Abstract

Embodiments of the present application provide a data processing method and device, electronic equipment, storage medium and product. The method comprises: obtaining pre-stored to-be-processed data, and performing dimension division processing on the to-be-processed data according to a preset dimension to obtain data in at least two dimensions; performing data volume processing on the data in each dimension according to the data volume of the data in each dimension in the data in the at least two dimensions to obtain processed data in each dimension; and sending the processed data in each dimension to multiple different nodes for data processing. The above technical means can improve the efficiency of ClickHouse in processing data in the case of large data volume.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, storage medium and product. Background Technology

[0002] ClickHouse, a high-performance distributed database designed for real-time big data analysis, is widely favored for its outstanding performance.

[0003] In the process of realizing this invention, the inventors discovered that the prior art has at least the following problems: Since the high performance of ClickHouse is guaranteed by squeezing the server performance, with the server remaining unchanged, as the amount of data for aggregate computing and the number of concurrent cluster computing increases, ClickHouse can easily reach its performance bottleneck.

[0004] Therefore, there is an urgent need for a data processing method that can improve ClickHouse's efficiency in processing large amounts of data. Summary of the Invention

[0005] This invention provides a data processing method that can improve the efficiency of ClickHouse in processing large amounts of data.

[0006] On one hand, embodiments of the present invention provide a data processing method, including:

[0007] Obtain pre-stored data to be processed, and perform dimensional segmentation processing on the data to be processed according to preset dimensions to obtain data under at least two dimensions;

[0008] Based on the data volume of each dimension in the data of the at least two dimensions, data volume processing is performed on the data of each dimension to obtain the processed data of each dimension;

[0009] The processed data from each dimension is sent to multiple different nodes for further processing.

[0010] Optionally, based on the data volume of each dimension in the at least two dimensions, data volume processing is performed on the data in each dimension to obtain processed data for each dimension, including:

[0011] For each dimension of the data in the at least two dimensions, if it is determined that the amount of data in that dimension is greater than a preset data storage threshold, then data reduction processing is performed based on the data in that dimension and the data in any other dimension among the at least two dimensions to obtain the processed data in that dimension.

[0012] If the amount of data in a given dimension is less than or equal to a preset data storage threshold, then the data in that dimension is considered the processed data for that dimension.

[0013] Optionally, if it is determined that the amount of data in this dimension is greater than a preset data storage threshold, then data reduction processing is performed based on the data in this dimension and the data in any other dimension among the data in the at least two dimensions to obtain the processed data in this dimension, including:

[0014] If it is determined that the amount of data under this dimension is greater than the preset data storage threshold, then the intersection between the data under this dimension and the data under any other dimension among the data under the at least two dimensions is extracted to obtain the processed data under this dimension. The processed data under this dimension is the data under the intersection of the data under this dimension and the data under any other dimension among the data under the at least two dimensions.

[0015] Optionally, the processed data for each dimension corresponds to at least one node; the processed data for each dimension is sent to multiple different nodes for data processing, including:

[0016] If it is determined that the current data processing volume of each node is greater than the preset data processing threshold, then a portion of the processed data corresponding to that node will be sent to that node for processing. The portion of data refers to the data in the processed data corresponding to that node that is within the range of the preset data processing threshold. The remaining data in the processed data corresponding to that node will be sent to other nodes for processing. The remaining data refers to the data in the processed data corresponding to that node that exceeds the preset data processing threshold.

[0017] Optionally, the preset dimension includes multiple dimension values; the data to be processed is divided according to the preset dimension to obtain data in at least two dimensions, including:

[0018] Based on each dimension value, determine the data under each dimension value in the data to be processed, so as to obtain the data under the at least two dimensions.

[0019] Optionally, before performing data volume processing on the data in each dimension based on the data volume of the data in each of the at least two dimensions to obtain the processed data in each dimension, the method further includes:

[0020] Identify and remove invalid data under invalid dimensions.

[0021] Optionally, after sending the processed data for each dimension to multiple different nodes for further data processing, the process may also include:

[0022] Obtain the data processing results for each node;

[0023] The data processing results corresponding to each dimension are merged to obtain the merged result;

[0024] The merged result and the invalid data are merged to obtain the data to be queried.

[0025] One embodiment of the above invention has the following advantages or beneficial effects:

[0026] In this embodiment, pre-stored data to be processed is acquired and divided into dimensions according to preset dimensions to obtain data in at least two dimensions. Based on the data volume of each dimension, data volume processing is performed on the data in each dimension to obtain processed data for each dimension. The processed data in each dimension is then sent to multiple different nodes for further processing. Using this technique, the efficiency of ClickHouse's data processing can be improved when dealing with large amounts of data.

[0027] On the other hand, embodiments of the present invention provide a data processing apparatus, comprising:

[0028] The partitioning processing module is used to acquire pre-stored data to be processed and perform dimensional partitioning processing on the data to be processed according to preset dimensions to obtain data in at least two dimensions.

[0029] The data volume processing module is used to perform data volume processing on the data in each dimension based on the data volume of the data in each of the at least two dimensions, so as to obtain the processed data in each dimension.

[0030] The sending module is used to send the processed data from each dimension to multiple different nodes for data processing.

[0031] Optionally, a data volume processing module is used to perform data reduction processing on the data in each of the at least two dimensions if it is determined that the data volume in that dimension is greater than a preset data storage threshold, and to obtain the processed data in that dimension based on the data in that dimension and the data in any other dimension among the at least two dimensions.

[0032] If the amount of data in a given dimension is less than or equal to a preset data storage threshold, then the data in that dimension is considered the processed data for that dimension.

[0033] Optionally, a data volume processing module is used to extract the intersection between the data in this dimension and the data in any other dimension among the data in the at least two dimensions if it is determined that the data volume in this dimension is greater than a preset data storage threshold, thereby obtaining the processed data in this dimension. The processed data in this dimension is the data at the intersection of the data in this dimension and the data in any other dimension among the data in the at least two dimensions.

[0034] Optionally, the sending module is configured to, if it is determined that the current data processing volume of each node is greater than a preset data processing threshold, send a portion of the processed data corresponding to that node to that node for processing, wherein the portion of data is data within the range of the preset data processing threshold in the processed data corresponding to that node, and send the remaining data in the processed data corresponding to that node to other nodes for processing, wherein the remaining data is data exceeding the preset data processing threshold in the processed data corresponding to that node.

[0035] Optionally, a partitioning processing module is used to determine the data under each dimension value in the data to be processed based on each dimension value, so as to obtain the data under the at least two dimensions.

[0036] Optionally, the device further includes:

[0037] The elimination module is used to identify and eliminate invalid data under invalid dimensions.

[0038] Optionally, the device further includes:

[0039] The data to be queried module is used to obtain the data processing results of each node;

[0040] The data processing results corresponding to each dimension are merged to obtain the merged result;

[0041] The merged result and the invalid data are merged to obtain the data to be queried.

[0042] In this embodiment, pre-stored data to be processed is acquired and divided into dimensions according to preset dimensions to obtain data in at least two dimensions. Based on the data volume of each dimension, data volume processing is performed on the data in each dimension to obtain processed data for each dimension. The processed data in each dimension is then sent to multiple different nodes for further processing. Using this technique, the efficiency of ClickHouse's data processing can be improved when dealing with large amounts of data. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0044] Figure 1 This is a flowchart illustrating a data processing method according to Embodiment 1 of the present invention;

[0045] Figure 2 This is a flowchart illustrating a data processing method according to Embodiment 2 of the present invention;

[0046] Figure 3 This is a flowchart illustrating a data processing method according to Embodiment 3 of the present invention;

[0047] Figure 4 This is a schematic diagram of the structure of a data processing apparatus according to Embodiment 4 of the present invention;

[0048] Figure 5 This is a block diagram of a terminal device provided according to Embodiment 5 of the present invention.

[0049] The accompanying drawings have illustrated specific embodiments of this disclosure, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this disclosure to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0050] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0051] First, the terms used in the embodiments of this invention will be explained:

[0052] The pre-stored data to be processed refers to the data table stored in the ClickHouse library, which stores a large amount of data to be processed in a row and column format.

[0053] Preset dimensions refer to dimensions based on the type of data to be processed. Specifically, preset dimensions can be mandatory dimensions, cascading dimensions, and invalid dimensions. Mandatory dimensions are those that the data processed by the node must possess. Cascading dimensions are dimensions that the data processed by the node must have; specifically, cascading dimensions refer to dimensions with hierarchical relationships, such as first-level departments, second-level departments, and third-level departments. Invalid dimensions are dimensions that are not needed in the data processed by the node. Furthermore, the definition of invalid dimensions differs for different types of data to be processed.

[0054] The data storage threshold refers to the cluster's memory threshold, which is the upper limit of the cluster's memory. Within this upper limit, the cluster can work normally; if the upper limit is exceeded, the cluster cannot work normally.

[0055] The preset data processing threshold refers to the upper limit of the data that a node can process. Within this upper limit, the node can work normally. If the upper limit is exceeded, the node cannot work normally.

[0056] The specific application scenario of this invention is to calculate a large amount of data in the ClickHouse database.

[0057] The data processing method provided in this invention aims to solve the above-mentioned technical problems of the prior art.

[0058] The technical solutions of the present invention and how they solve the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0059] Figure 1 This is a flowchart illustrating a data processing method according to Embodiment 1 of the present invention.

[0060] Specifically, Example 1 includes the following steps:

[0061] S110. Obtain pre-stored data to be processed, and perform dimensional division processing on the data to be processed according to preset dimensions to obtain data in at least two dimensions.

[0062] In this embodiment, the data in at least two dimensions is non-aggregated data, and the sum of the data in at least two dimensions is the pre-stored data to be processed. In this embodiment, after obtaining the pre-stored data to be processed, the data to be processed is divided and processed according to preset dimensions, which is equivalent to classifying the data to be processed according to the data type, so as to obtain data in at least two dimensions.

[0063] S120. Based on the data volume of each dimension in the data of the at least two dimensions, perform data volume processing on the data of each dimension to obtain the processed data of each dimension.

[0064] In this embodiment, after obtaining data for each of the at least two dimensions, the data volume for each dimension is calculated, and data volume processing is performed on the data volume for each dimension. For example, if the data volume for each of the at least two dimensions is 1 billion, 500 million, and 800 million respectively, then data volume processing is performed on the 1 billion data in the first dimension, the 500 million data in the second dimension, and the 800 million data in the third dimension to obtain the processed data for each dimension. Specifically, it could be 500 million data in the first dimension, or 500 million data in both the second and third dimensions.

[0065] S130. Send the processed data under each dimension to multiple different nodes for data processing.

[0066] In this embodiment, the processed data for each dimension is sent to at least one corresponding node for data processing. For example, if each dimension includes a first dimension, a second dimension, and a third dimension, the processed data for the first dimension is sent to at least one node for data processing, the processed data for the second dimension is sent to at least one node for data processing, and the processed data for the third dimension is sent to at least one node for data processing. The at least one node sent for the first dimension, the at least one node sent for the second dimension, and the at least one node sent for the third dimension are all different nodes. Furthermore, if there is more than one node sent for the first dimension, these nodes are also different nodes. The cases for the second and third dimensions are similar and will not be described further here.

[0067] In this embodiment, pre-stored data to be processed is acquired and divided into dimensions according to preset dimensions to obtain data in at least two dimensions. Based on the data volume of each dimension, data volume processing is performed on the data in each dimension to obtain processed data for each dimension. The processed data in each dimension is then sent to multiple different nodes for further processing. Using this technique, the efficiency of ClickHouse's data processing can be improved when dealing with large amounts of data.

[0068] Figure 2 This is a flowchart illustrating a data processing method according to Embodiment 2 of the present invention.

[0069] Specifically, Example 2 includes the following steps:

[0070] S210. Obtain pre-stored data to be processed, and perform dimensional division processing on the data to be processed according to preset dimensions to obtain data in at least two dimensions.

[0071] In this embodiment, optionally, the preset dimension includes multiple dimension values; the data to be processed is divided into dimensions according to the preset dimension to obtain data under at least two dimensions, including:

[0072] Based on each dimension value, determine the data under each dimension value in the data to be processed, so as to obtain the data under the at least two dimensions.

[0073] In this embodiment, for example, multiple dimension values ​​can be platform, first-level department, second-level department, and third-level department. By determining the dimension value of "platform," data is determined in the data to be processed based on this dimension value.

[0074] S220. For the data in each of the at least two dimensions, determine whether the amount of data in that dimension is greater than a preset data storage threshold; if yes, then execute S230; if no, then execute S240.

[0075] S230. If it is determined that the amount of data in this dimension is greater than the preset data storage threshold, then data reduction processing is performed based on the data in this dimension and the data in any other dimension among the data in the at least two dimensions to obtain the processed data in this dimension.

[0076] In this embodiment, data reduction processing refers to reducing the amount of data in a dimension by increasing the dimension, so that the amount of data in that dimension is within a preset storage threshold range. Specifically, if the number of data in a dimension is 1 billion and the preset data storage threshold is 800 million, then after performing data reduction processing on this dimension and the data in any other dimension among at least two other dimensions, the processed data amount in that dimension can be 800 million. This 800 million data points constitute the processed data for that dimension.

[0077] In this embodiment, optionally, if it is determined that the amount of data in this dimension is greater than a preset data storage threshold, then data reduction processing is performed based on the data in this dimension and the data in any other dimension among the data in the at least two dimensions to obtain the processed data in this dimension, including:

[0078] If it is determined that the amount of data under this dimension is greater than the preset data storage threshold, then the intersection between the data under this dimension and the data under any other dimension among the data under the at least two dimensions is extracted to obtain the processed data under this dimension. The processed data under this dimension is the data under the intersection of the data under this dimension and the data under any other dimension among the data under the at least two dimensions.

[0079] In this embodiment, when the amount of data in this dimension is greater than the preset data storage threshold, error data information can be returned to facilitate the extraction of the intersection between the data in this dimension and the data in any other dimension among the data in the at least two dimensions.

[0080] For example, if the data volume in this dimension is 1 billion, the preset data storage threshold is 800 million, and the data volume in any other dimension among the data in at least two dimensions is 500 million or 600 million. If the data volume in the selected other dimension is 500 million, then the intersection of the data in this dimension with the data in other dimensions is extracted to obtain the processed data in this dimension, which can be 400 million data points.

[0081] S240. If it is determined that the amount of data in this dimension is less than or equal to the preset data storage threshold, then the data in this dimension is determined to be the processed data in this dimension.

[0082] In this embodiment, if the amount of data in this dimension is less than or equal to the preset data storage threshold, it means that the data in this dimension is within the range that the cluster can process. Therefore, the data in this dimension is the processed data.

[0083] S250: Send the processed data under each dimension to multiple different nodes for data processing.

[0084] In this embodiment, by reducing the data volume in this dimension, the processed data in this dimension can be obtained. By splitting the data and performing a series of optimizations, the amount of computational data can be indirectly reduced, thus avoiding the limitation on the amount of computational data caused by the cluster performance bottleneck.

[0085] Figure 3 This is a flowchart illustrating a data processing method according to Embodiment 3 of the present invention.

[0086] Specifically, Example 3 includes the following steps:

[0087] S310. Obtain pre-stored data to be processed, and perform dimensional division processing on the data to be processed according to preset dimensions to obtain data in at least two dimensions.

[0088] S320. Identify and remove invalid data under invalid dimensions.

[0089] In this embodiment, an invalid dimension is one of at least two dimensions, and invalid data under an invalid dimension is data that does not need to be processed.

[0090] S330. Based on the data volume of each dimension in the data of the at least two dimensions, perform data volume processing on the data of each dimension to obtain the processed data of each dimension.

[0091] S340. The processed data in each dimension corresponds to at least one node; if it is determined that the current data processing volume of each node is greater than the preset data processing threshold, then a portion of the processed data corresponding to that node is sent to that node for processing.

[0092] In this embodiment, the partial data refers to the data within the preset data processing threshold range of the processed data corresponding to the node, and the remaining data in the processed data corresponding to the node is sent to other nodes for processing. The remaining data refers to the data in the processed data corresponding to the node that exceeds the preset data processing threshold.

[0093] Specifically, in this embodiment, for each dimension, the processed data corresponds to at least one node, and the preset data processing threshold for each node is different. The preset data processing threshold for each node is determined by the node's CPU, memory, disk usage, and network connection status.

[0094] For example, if the processed data in the first dimension corresponds to 3 nodes, and the processed data volume in the first dimension is 400 million, then when sending data to the 3 nodes, the preset data processing threshold for the first node is 100 million, the preset data processing threshold for the second node is 200 million, and the preset data processing threshold for the third node is 200 million. First, 100 million data points are sent to the first node, then 200 million data points are sent to the second node, and finally, 100 million data points are sent to the third node. In this case, the 100 million data points in the first node represent a portion of the data, and the 200 million data points in the second node represent the remaining data.

[0095] S350: Obtain the data processing results for each node.

[0096] S360. Merge the data processing results corresponding to each dimension to obtain the merged result.

[0097] S370. The merged result and the invalid data are merged to obtain the data to be queried.

[0098] In this embodiment, the merging result is the sum of the data processing results under each dimension, and this sum is merged with invalid data to obtain the data to be queried, thus ensuring the integrity of the data to be queried.

[0099] Furthermore, during the processing of data by each node, if the data to be processed is data that has been processed previously, it can be pre-stored in a preset storage area. When the data needs to be retrieved, it can be retrieved from the preset storage area. The advantage of this setup is that it can improve the efficiency of node data processing.

[0100] Furthermore, the accuracy of the query result can be ensured by computing the pre-stored data to be processed in another database and comparing the computing results with the query result. For example, the other database could be a Hive database, where Spark SQL is used for computation.

[0101] In this embodiment, by comparing the current data processing volume of each node with the preset data processing threshold, the amount of data processed by each node is limited to within the preset data processing threshold. In this way, the amount of data calculated by each node each time will be very small, which can improve the high concurrency capability of the cluster while ensuring the stability of the cluster, and at the same time increase the amount of data for the cluster to perform aggregation calculations.

[0102] Figure 4 This is a schematic diagram of a data processing apparatus provided according to Embodiment 4 of the present invention.

[0103] Specifically, Example 4 includes:

[0104] The partitioning processing module 410 is used to acquire pre-stored data to be processed and perform dimensional partitioning processing on the data to be processed according to preset dimensions to obtain data in at least two dimensions.

[0105] The data volume processing module 420 is used to perform data volume processing on the data in each dimension based on the data volume of the data in each of the at least two dimensions, so as to obtain the processed data in each dimension.

[0106] The sending module 430 is used to send the processed data under each dimension to multiple different nodes for data processing.

[0107] Optionally, the data volume processing module 420 is used to perform data reduction processing on the data in each of the at least two dimensions if it is determined that the data volume in that dimension is greater than a preset data storage threshold, and to obtain the processed data in that dimension based on the data in that dimension and the data in any other dimension among the at least two dimensions.

[0108] If the amount of data in a given dimension is less than or equal to a preset data storage threshold, then the data in that dimension is considered the processed data for that dimension.

[0109] Optionally, the data volume processing module 420 is used to extract the intersection between the data in this dimension and the data in any other dimension among the data in the at least two dimensions if it is determined that the data volume in this dimension is greater than a preset data storage threshold, so as to obtain the processed data in this dimension. The processed data in this dimension is the data in the intersection of the data in this dimension and the data in any other dimension among the data in the at least two dimensions.

[0110] Optionally, the sending module 430 is configured to, if it is determined that the current data processing volume of each node is greater than a preset data processing threshold, send a portion of the processed data corresponding to that node to that node for processing, wherein the portion of data is data within the range of the preset data processing threshold in the processed data corresponding to that node, and send the remaining data in the processed data corresponding to that node to other nodes for processing, wherein the remaining data is data exceeding the preset data processing threshold in the processed data corresponding to that node.

[0111] Optionally, the partitioning processing module 410 is used to determine the data under each dimension value in the data to be processed based on each dimension value, so as to obtain the data under the at least two dimensions.

[0112] Optionally, the device further includes:

[0113] The elimination module 440 is used to identify and eliminate invalid data under invalid dimensions.

[0114] Optionally, the device further includes:

[0115] The data to be queried module 450 is used to obtain the data processing results of each node;

[0116] The data processing results corresponding to each dimension are merged to obtain the merged result;

[0117] The merged result and the invalid data are merged to obtain the data to be queried.

[0118] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0119] Figure 5 This is a block diagram of a terminal device according to Embodiment 5 of the present invention. The device may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0120] The device 500 may include one or more of the following components: a processing component 502, a memory 504, a power supply component 506, a multimedia component 508, an audio component 510, an input / output (I / O) interface 512, a sensor component 514, and a communication component 516.

[0121] Processing component 502 typically controls the overall operation of device 500, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 502 may include one or more modules to facilitate interaction between processing component 502 and other components. For example, processing component 502 may include a multimedia module to facilitate interaction between multimedia component 508 and processing component 502.

[0122] Memory 504 is configured to store various types of data to support the operation of device 500. Examples of such data include instructions for any application or method operating on device 500, contact data, phonebook data, messages, pictures, videos, etc. Memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0123] Power supply component 506 provides power to various components of device 500. Power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 500.

[0124] Multimedia component 508 includes a screen that provides an output interface between the device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 508 includes a front-facing camera and / or a rear-facing camera. When the device 500 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0125] Audio component 510 is configured to output and / or input audio signals. For example, audio component 510 includes a microphone (MIC) configured to receive external audio signals when device 500 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 504 or transmitted via communication component 516. In some embodiments, audio component 510 also includes a speaker for outputting audio signals.

[0126] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0127] Sensor assembly 514 includes one or more sensors for providing status assessments of various aspects of device 500. For example, sensor assembly 514 may detect the on / off state of device 500, the relative positioning of components such as the display and keypad of device 500, changes in the position of device 500 or a component of device 500, the presence or absence of user contact with device 500, the orientation or acceleration / deceleration of device 500, and temperature changes of device 500. Sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 514 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0128] Communication component 516 is configured to facilitate wired or wireless communication between device 500 and other devices. Device 500 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 516 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0129] In an exemplary embodiment, the apparatus 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0130] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, which can be executed by a processor 520 of the device 500 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0131] A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of a terminal device, the terminal device is enabled to perform the aforementioned data processing method of the terminal device.

[0132] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0133] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data processing method applied to a ClickHouse database, characterized in that, The method includes: The system acquires pre-stored data to be processed and performs dimensional segmentation on the data according to preset dimensions to obtain data in at least two dimensions; the pre-stored data to be processed refers to the data table stored in the ClickHouse database. For the data volume of each dimension in the data of the at least two dimensions, if it is determined that the data volume of the data in that dimension is greater than a preset data storage threshold, then the intersection between the data in that dimension and the data in any other dimension in the data of the at least two dimensions is extracted to reduce the data volume and obtain the processed data in that dimension; If the amount of data in a given dimension is less than or equal to a preset data storage threshold, then the data in that dimension will be directly used as the processed data for that dimension. The processed data from each dimension is sent to multiple different nodes for further processing. These nodes are device nodes in a distributed cluster that have CPU, memory, disk, and network connectivity.

2. The method according to claim 1, characterized in that, The processed data for each dimension corresponds to at least one node; the processed data for each dimension is sent to multiple different nodes for further data processing, including: If it is determined that the current data processing volume of each node is greater than the preset data processing threshold, then a portion of the processed data corresponding to that node will be sent to that node for processing. The portion of data refers to the data in the processed data corresponding to that node that is within the range of the preset data processing threshold. The remaining data in the processed data corresponding to that node will be sent to other nodes for processing. The remaining data refers to the data in the processed data corresponding to that node that exceeds the preset data processing threshold.

3. The method according to claim 1, characterized in that, The preset dimensions include multiple dimension values; the data to be processed is divided according to the preset dimensions to obtain data under at least two dimensions, including: Based on each dimension value, determine the data under each dimension value in the data to be processed, so as to obtain the data under the at least two dimensions.

4. The method according to claim 1, characterized in that, Before performing data volume processing on the data in each dimension based on the data volume of the data in each of the at least two dimensions to obtain the processed data in each dimension, the process further includes: Identify and remove invalid data under invalid dimensions.

5. The method according to claim 4, characterized in that, After sending the processed data from each dimension to multiple different nodes for further processing, the process also includes: Obtain the data processing results for each node; The data processing results corresponding to each dimension are merged to obtain the merged result; The merged result and the invalid data are merged to obtain the data to be queried.

6. A data processing apparatus, applied to a ClickHouse database, characterized in that, The device includes: The partitioning module is used to acquire pre-stored data to be processed and perform dimensional partitioning on the data to be processed according to preset dimensions to obtain data in at least two dimensions; the pre-stored data to be processed refers to the data table stored in the ClickHouse database; The data volume processing module is used to, for each dimension of the data in the at least two dimensions, if it is determined that the data volume of the data in that dimension is greater than a preset data storage threshold, then extract the intersection between the data in that dimension and the data in any other dimension of the at least two dimensions to reduce the data volume and obtain the processed data in that dimension; if it is determined that the data volume of the data in that dimension is less than or equal to the preset data storage threshold, then directly use the data in that dimension as the processed data in that dimension. The sending module is used to send the processed data under each dimension to multiple different nodes for data processing. The nodes are device nodes in a distributed cluster that have CPU, memory, disk and network connection functions.

7. An electronic device, characterized in that, include: Memory, processor; Memory; Memory used to store the processor's executable instructions; The processor is configured to perform the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-5.

Images