A data processing method, device and medium based on Clickhouse

Abstract

The present application provides a data processing method, device and medium based on Clickhouse, by determining the label of business data, wherein the business data is data collected by several terminal devices, and classifying each business data according to the label of the business data. The classified business data is converted into the corresponding data format to obtain the data to be stored. According to the label of the data to be stored, the data to be stored is stored in the Clickhouse data warehouse to obtain the warehouse data. The label of the data to be stored is obtained according to the label of the business data. Based on the user's operation, the user's demand data in the Clickhouse data warehouse is determined. The demand data is generated according to the warehouse data. Through the above method, the user's data needs are met in a timely manner, and a good service experience is provided to the user.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data processing method, device and medium based on Clickhouse. Background Art

[0002] With the development of data centers, the data in projects of various industries is increasing, and the corresponding data required by users and the data used by developers are also increasing. The increasing amount of data undoubtedly increases the workload of developers during development. And under the accumulation of data, the development speed of developers is far slower than the speed of changes in user needs.

[0003] If developers cannot meet user needs in the first place, it will result in a poor user experience, and developers will not be able to recognize the services provided by developers. At the same time, developers' work enthusiasm may also decline. Therefore, there is an urgent need for a technical solution that can provide services to users in a timely manner and meet their data needs, so as to improve the experience of users with data needs for data services. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, device and medium based on Clickhouse, which are used to provide user data services in a timely manner to meet the user's data needs.

[0005] On the one hand, the present application provides a data processing method based on Clickhouse, the method comprising:

[0006] Determine the label of the business data. Business data is data collected by several terminal devices. Classify each business data according to the label of the business data. Convert the classified business data into the corresponding data format to obtain the data to be stored. According to the label of the data to be stored, store the data to be stored in the Clickhouse data warehouse to obtain the warehouse data. The label of the data to be stored is obtained according to the label of the business data. Based on the user's operation, determine the user's demand data in the Clickhouse data warehouse. The demand data is generated according to the warehouse data.

[0007] In one implementation of the present application, the data format of the business data is determined. According to the data format of the business data, a preset data conversion standard is matched. According to the matched data conversion standard, the business data is converted into a data format to obtain data to be stored. The data to be stored is in a lightweight data exchange JSON format.

[0008] In one implementation of the present application, data to be stored is compressed according to the label of the data to be stored. And the compressed data to be stored is encrypted according to the RSA encryption algorithm. Before the data to be stored is written into the Clickhouse data warehouse, the encrypted data to be stored is decrypted.

[0009] In one implementation of the present application, a query interface is displayed to a user terminal. The query interface includes several application programming interfaces (APIs) for querying storage data in a Clickhouse data warehouse. A query API is generated based on user operations on the query interface. The query API includes at least one API. According to the query API, storage data in the Clickhouse data warehouse corresponding to the information in the query API is determined as user demand data.

[0010] In one implementation of the present application, a query interface is displayed to a user terminal. The query interface is used to input a structured query language SQL. When the user terminal sends input completion information, a query API is generated according to the SQL input by the user terminal. According to the query API, storage data corresponding to the information in the query API in the Clickhouse data warehouse is determined as the user's demand data.

[0011] In one implementation of the present application, after the query API is generated, parameter information is sent to the user terminal to prompt the user to enter the query parameters of the query API. According to the parameters entered by the user terminal, the storage data corresponding to the information in the query API in the Clickhouse data warehouse is determined through the query API as the user's demand data.

[0012] In one implementation of the present application, the label of the business data includes at least one of the following: syslog, database log, server log, and user portrait.

[0013] In one implementation of the present application, in the case where there is no user demand data in the Clickhouse data warehouse, demand information is generated and sent to a maintenance terminal, feedback information of the maintenance terminal is determined, and the feedback information is forwarded to a user terminal corresponding to the user.

[0014] On the other hand, the present application also provides a data processing device based on Clickhouse, the device comprising:

[0015] At least one processor; and a memory in communication with the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0016] Determine the label of the business data. Business data is data collected by several terminal devices. Classify each business data according to the label of the business data. Convert the classified business data into the corresponding data format to obtain the data to be stored. According to the label of the data to be stored, store the data to be stored in the Clickhouse data warehouse to obtain the warehouse data. The label of the data to be stored is obtained according to the label of the business data. Based on the user's operation, determine the user's demand data in the Clickhouse data warehouse. The demand data is generated according to the warehouse data.

[0017] On the other hand, the present application also provides a non-volatile computer storage medium based on Clickhouse data processing, storing computer executable instructions, wherein the computer executable instructions are configured to:

[0018] Determine the label of the business data. Business data is data collected by several terminal devices. Classify each business data according to the label of the business data. Convert the classified business data into the corresponding data format to obtain the data to be stored. According to the label of the data to be stored, store the data to be stored in the Clickhouse data warehouse to obtain the warehouse data. The label of the data to be stored is obtained according to the label of the business data. Based on the user's operation, determine the user's demand data in the Clickhouse data warehouse. The demand data is generated according to the warehouse data.

[0019] This application obtains a number of business data, determines the label of each business data, and classifies each business data according to the label of each business data. The classified business data is formatted and written into the Clickhouse data warehouse. According to the user's operation, the required data is provided to the user from the Clickhouse data warehouse. By utilizing the characteristics of the Clickhouse data warehouse, the reading speed of the stored data is improved, allowing users to obtain the required data in a timely manner. By writing the business data classification into the Clickhouse data warehouse, a large amount of business data can be quickly and conveniently written into the Clickhouse data warehouse, saving the time of business data storage. Through the above scheme, this application can provide users with required data in a timely manner and provide users with good data demand services. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1A flow chart of a data processing method based on Clickhouse in an embodiment of the present application;

[0022] Figure 2 This is a structural schematic diagram of a data processing device based on Clickhouse in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0024] The development of the Internet has enabled all industries to conduct projects and provide services to users through the Internet. For network services, the indispensable personnel are the developers in the background. When users experience network services through the Internet, if they face new service requirements, developers need to provide timely feedback to give users a good service experience. However, due to the increasing amount of data and the large number of people using network services, the needs of users may not be fed back by developers in a timely manner, which reduces the network service experience of users whose needs cannot be met.

[0025] If we can provide a technical solution that can promptly meet users' data needs in the current data service scenario, we can improve users' experience of data services.

[0026] Based on this, the embodiments of the present application provide a data processing method, device and medium based on Clickhouse, which are used to provide timely data services to users and improve the user experience of data services. The present application is a data processing method based on Clickhouse. Clickhouse is a column storage database. Since it stores data in a column storage manner, it reads data quickly and occupies less storage space than other database storage methods. Using Clickhouse to store project data of users and developers can ensure that data can be read in a timely manner, laying the foundation for providing users with timely data services.

[0027] The following describes in detail various embodiments of the present application in conjunction with the accompanying drawings.

[0028] The present application embodiment provides a data processing method based on Clickhouse, such as Figure 1 As shown, the method may include steps 101 to 105:

[0029] Step 101: The server determines a label of business data.

[0030] Among them, business data refers to data collected by several terminal devices.

[0031] In the embodiment of the present application, the business data can be sent to the server by multiple collection terminals, or can be data collected by a server cluster composed of multiple servers. The collection terminal can be a terminal device such as a laptop computer, a mobile phone, or other devices, and the present application does not specifically limit this.

[0032] The executor of the present application may be a server, or a server cluster consisting of several servers. In the server cluster, there are servers responsible for different functions, such as a data acquisition server, a data processing server, etc. The servers in the server cluster may communicate with each other to implement the Clickhouse-based data processing method of the present application.

[0033] After the business data is collected, the server can determine the label of the business data. In the embodiment of the present application, the label of the business data may include one of the following: syslog, database log, server log, user portrait. In actual use, due to the growth of business data, the type of label may include more, and the present application does not specifically limit the number and type of labels for business data.

[0034] For example, if a developer writes data that has not been used by a user into a server log during the development process, the server may determine that the tag of the unused data is a server log.

[0035] It should be noted that the server, as the executor of the data processing method based on Clickhouse, is only an example. The executor is not limited to the server, and this application does not make any specific limitations on this.

[0036] Step 102: The server classifies each business data according to the tags of the business data.

[0037] After the server determines the tags of the business data, the server may classify the business data according to the tags of the business data to process the classified business data.

[0038] By classifying business data, there will be no interfering data when the business data is processed, ensuring that the data processing is convenient and error-prone.

[0039] Step 103: the server converts the classified business data into corresponding data formats to obtain data to be stored.

[0040] In an embodiment of the present application, the server may perform data format conversion on the business data to facilitate transmission and storage of the business data. The server may perform format conversion on the business data by executing the following method:

[0041] First, the server determines the data format of the business data.

[0042] Business data may be in various formats, such as log files.

[0043] Then, the server matches the preset data conversion standard according to the data format of the business data.

[0044] The server may pre-store several data conversion standards, which can process business data in one data format into data in another data format, but the content recorded in the data remains unchanged. For example, the data conversion standard converts log files into a lightweight data exchange (JavaScript Object Notation, JS object notation) JSON format.

[0045] Finally, the server converts the business data into a data format according to the matching data conversion standard to obtain the data to be stored.

[0046] The data to be stored is in a lightweight data exchange JSON format.

[0047] The server converts the business data in various data formats into data to be stored in JSON format, so that the format of the data to be stored is unified when it is processed, which makes it easier for the server to process it and saves time for data processing.

[0048] After the server obtains the data to be stored, it can store the data to be stored in the Clickhouse data warehouse, so that it can be provided to users in a timely manner when users have needs for the stored data. Since the server needs to collect data and write the data to be stored into the Clickhouse data warehouse, in order to avoid data loss and ensure data security. Therefore, before writing the data to be stored into the Clickhouse data warehouse, this application can also execute the following method:

[0049] First, the server compresses the data to be stored according to the tags of the data to be stored.

[0050] The server can compress the data to be stored with labels A and B respectively according to the labels of the data to be stored. The server data compression can compress the data to be stored into a compressed file, such as a GNUzip file, by using compression software.

[0051] Secondly, the server encrypts the compressed data to be stored according to the RSA encryption algorithm.

[0052] While or after encrypting the data to be stored, the server may also perform an encryption operation on the compressed data to be stored. During encryption, the server may use an RSA encryption algorithm to encrypt the data to be stored.

[0053] Due to the high security of the RSA encryption algorithm, it is not easy to be cracked by the algorithm. Therefore, when the project data volume is large and the security requirements for various types of data to be stored, such as user data and developer data, are high, the server encrypts the data to be stored through the RSA encryption algorithm, which can better ensure the security of the data and ensure that the data can be accurately provided to users.

[0054] Finally, the server decrypts the encrypted data to be stored before writing it into the ClickHouse data warehouse.

[0055] Before the server stores the data to be stored in the Clickhouse data warehouse, it is necessary to decrypt the compressed data to be stored using a decryption key stored on the server itself and an encryption key in the server, thereby ensuring the accuracy of the data to be stored in the server and the credibility of the data in the Clickhouse data warehouse.

[0056] Step 104, the server stores the data to be stored in the Clickhouse data warehouse according to the label of the data to be stored to obtain the warehouse data.

[0057] The label of the data to be stored is obtained according to the label of the business data.

[0058] The server stores the data to be stored one by one in the Clickhouse data warehouse in a columnar manner according to the tags of the data to be stored. The server batch processes the data to be stored under the C tag and writes it to Clickhouse through a pre-stored batch script.

[0059] The label of the data to be stored can be obtained according to the label of the business data. For example, the label of the business data can be one of syslog, database log, server log, and user portrait. The label of the data to be stored generated by the server according to the business data can correspond to the label of the business data. For example, if the label of the business data is syslog, then the label of the data to be stored is also syslog.

[0060] The server can provide ClickHouse's concurrent writing capabilities through batch processing of data, thereby increasing the speed of data processing.

[0061] Step 105: Based on the user's operation, the server determines the user's required data in the Clickhouse data warehouse.

[0062] Among them, demand data is generated based on warehouse data.

[0063] In the embodiment of the present application, the server can interact with the user terminal through the user terminal interaction interface corresponding to the user. The server interacts with the user terminal to obtain the user's demand data, which can be achieved by executing the following method:

[0064] First, the server displays the query interface to the user terminal.

[0065] The query interface includes several application programming interfaces (APIs) for querying the storage data in the Clickhouse data warehouse.

[0066] The server displays to the user terminal a query interface having a plurality of application programming interfaces (APIs) for querying the storage data in the Clickhouse data warehouse.

[0067] Then, the server generates a query API based on the user's operation on the query interface.

[0068] The query API includes at least one API.

[0069] The user can perform corresponding operations on several application programming interfaces (APIs) in the query interface, such as sliding APIs, clicking APIs, dragging APIs, etc. The user can select a location in the query interface to place the selected API or determine the selected API through preset operations in the query interface. The server generates a query API based on the user's selected API, and the query API includes at least one API.

[0070] Finally, the server determines the storage data in the Clickhouse data warehouse corresponding to the information in the query API based on the query API as the user's demand data.

[0071] The server can obtain the user's required data in the Clickhouse data warehouse according to the query API, and the data corresponds to the information in the query API. For example, if the query API is data 1, data 2, and data 3, the server can quickly determine data 1, data 2, and data 3 from the Clickhouse data warehouse according to the labels corresponding to data 1, data 2, and data 3, and provide them to the user to meet the user's needs.

[0072] Taking software as an example, in the software, the user demand data is a page turning function of a software interface. According to the user's first selected API in the query interface as "page turning", and the second selected API as "such and such software interface", the server generates a query API with the first selected API and the second selected API. According to the query API, the server obtains the corresponding storage data including "page turning" and "such and such software interface" from the Clickhouse data warehouse to provide it to the user.

[0073] In another embodiment of the present application, the server may also display an interface of an API written by the user to the user, and provide the user with required data through the API defined by the user, specifically:

[0074] First, the server displays a query interface to the user terminal, wherein the query interface is used to input structured query language SQL.

[0075] Then, when the user terminal sends input completion information, the server generates a query API according to the SQL input by the user terminal.

[0076] In the query interface, a preset area for input completion can be set. When the user operates the preset area, the server can determine that the user terminal sends input completion information to generate a query API according to the SQL input by the user terminal.

[0077] Finally, the server determines the storage data in the Clickhouse data warehouse corresponding to the information in the query API based on the query API as the user's demand data.

[0078] The steps for obtaining the user's required data according to the query API have been discussed in the above steps and will not be repeated here.

[0079] After the server determines the query API, it also needs to determine the parameters of the API to accurately determine the user's required data in the Clickhouse data warehouse. For example, if there is a sub-parameter in the data 1 of the query API, the user needs to define the sub-parameter. The details are as follows:

[0080] After the query API is generated, the server sends parameter information to the user terminal to prompt the user to input query parameters of the query API.

[0081] For example, if the user's required data is a map data, there are several map data in the Clickhouse data warehouse, and the names are all composed of multiple letters and multiple numbers. If the user does not define the name parameter of the map data, the server cannot determine which map data the user's required data is.

[0082] Based on the parameters input by the user terminal, the server determines the storage data in the Clickhouse data warehouse corresponding to the information in the query API through the query API as the user's demand data.

[0083] In the embodiment of the present application, there may be storage data in the Clickhouse data warehouse that does not meet the user's needs. The present application can provide the user with the required data by executing the following method:

[0084] First, the server generates demand information in the Clickhouse data warehouse when there is no user demand data, and sends the demand information to the maintenance terminal.

[0085] Then, the server determines the feedback information of the maintenance terminal.

[0086] The feedback information may be the required data provided to the user, or other information used to meet the required data of the user, for example: "At such and such time, we will provide you with the required data."

[0087] Finally, the server forwards the feedback information to the user terminal corresponding to the user.

[0088] The server sends the demand information to the maintenance terminal, so that the maintenance terminal can provide data services to the user terminal in a timely manner. Before the method provided by the present application is used in the prior art, the user can only send the demand data information to the developer first, and the developer performs development operations on the user demand data, which may delay the user's use time. Due to the increase in project data, the developer may not be able to obtain the demand data information of some users in a timely manner when the workload is huge, which further increases the waiting time of the user.

[0089] Through the technical solution provided by this application, various log data, user portraits, etc. can be stored in the Clickhouse data warehouse before the user proposes the required data. When the user has data needs, he can directly obtain the required data by operating in the query interface provided by the server, without the need for developers to develop. The technical solution provided by this application can meet the data needs of users more timely and provide users with better data services.

[0090] Figure 2 A schematic diagram of a data processing device based on Clickhouse provided in an embodiment of the present application, the device comprising:

[0091] At least one processor; and a memory in communication with the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0092] Determine the label of the business data. Business data is data collected by several terminal devices. Classify each business data according to the label of the business data. Convert the classified business data into the corresponding data format to obtain the data to be stored. According to the label of the data to be stored, store the data to be stored in the Clickhouse data warehouse to obtain the warehouse data. The label of the data to be stored is obtained according to the label of the business data. Based on the user's operation, determine the user's demand data in the Clickhouse data warehouse. The demand data is generated according to the warehouse data.

[0093] The embodiment of the present application also provides a non-volatile computer storage medium for data processing based on Clickhouse, storing computer executable instructions, wherein the computer executable instructions are configured as follows:

[0094] Determine the label of the business data. Business data is data collected by several terminal devices. Classify each business data according to the label of the business data. Convert the classified business data into the corresponding data format to obtain the data to be stored. According to the label of the data to be stored, store the data to be stored in the Clickhouse data warehouse to obtain the warehouse data. The label of the data to be stored is obtained according to the label of the business data. Based on the user's operation, determine the user's demand data in the Clickhouse data warehouse. The demand data is generated according to the warehouse data.

[0095] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0096] The devices, media and methods provided in the embodiments of the present application correspond one to one, and therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0097] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0098] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0101] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0102] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0103] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0104] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0105] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A data processing method based on Clickhouse, characterized in that: The method comprises: Determine a label for business data; wherein the business data is data collected by a number of terminal devices; Classifying each of the business data according to the label of the business data; Convert the classified business data into corresponding data formats to obtain data to be stored; According to the label of the data to be stored, the data to be stored is stored in the Clickhouse data warehouse to obtain warehouse data; wherein the label of the data to be stored is obtained according to the label of the business data; Based on the user's operation, determine the user's demand data in the Clickhouse data warehouse; wherein the demand data is generated according to the warehouse data; The classified business data is converted into a data format to obtain data to be stored, which specifically includes: Determining a data format of the business data; According to the data format of the business data, matching the preset data conversion standard; According to the matched data conversion standard, the business data is converted into a data format to obtain data to be stored; wherein the data to be stored is in a lightweight data exchange JSON format; Wherein, based on the user's operation, the user's demand data in the Clickhouse data warehouse is determined, specifically including: Displaying a query interface to a user terminal; wherein the query interface includes a plurality of application programming interfaces (APIs) for querying the storage data in the Clickhouse data warehouse; Based on the user's operation on the query interface, a query API is generated; wherein the query API includes at least one of the APIs; According to the query API, determining the storage data in the Clickhouse data warehouse corresponding to the information in the query API as the demand data of the user; Wherein, the method further comprises: After the query API is generated, parameter information is sent to the user terminal to prompt the user to input query parameters of the query API; According to the parameters input by the user terminal, the storage data corresponding to the information in the query API in the Clickhouse data warehouse is determined through the query API as the demand data of the user.

2. The method according to claim 1, characterized in that: According to the label of the data to be stored, before writing the data to be stored into the Clickhouse data warehouse, the method further includes: Compressing the data to be stored according to the label of the data to be stored; and Encrypting the compressed data to be stored according to the RSA encryption algorithm; Before writing the data to be stored into the Clickhouse data warehouse, the encrypted data to be stored is decrypted.

3. The method according to claim 1, characterized in that: Based on the user's operation, the user's demand data in the Clickhouse data warehouse is determined, specifically including: Displaying a query interface to a user terminal; wherein the query interface is used to input a structured query language SQL; When the user terminal sends input completion information, generating a query API according to the SQL input by the user terminal; According to the query API, the storage data in the Clickhouse data warehouse corresponding to the information in the query API is determined as the demand data of the user.

4. The method according to claim 1, characterized in that: The labels of the business data include at least one of the following: syslog, database log, server log, and user portrait.

5. The method according to claim 1, characterized in that: The method further comprises: In the Clickhouse data warehouse, if the demand data of the user does not exist, generating demand information, and sending the demand information to the maintenance terminal; Determining feedback information of the maintenance terminal; and The feedback information is forwarded to a user terminal corresponding to the user.

6. A data processing device based on Clickhouse, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a data processing method based on Clickhouse as described in any one of claims 1 to 5 above.

7. A non-volatile computer storage medium for data processing based on Clickhouse, storing computer executable instructions, characterized in that: The computer executable instructions can execute a data processing method based on Clickhouse as described in any one of claims 1 to 5 above.

Images