Data processing method and system, electronic equipment and storage medium

Abstract

The invention provides a data processing method and system, electronic equipment and a storage medium, and the method comprises the steps: determining whether a user identifier in a data processing request has a high-quality permission, namely, a permission of processing data through a preset high-quality cluster; the preset high-optimal cluster is a presto cluster; if yes, determining whether the residual resource quantity of the high-optimization cluster meets a preset processing condition of the data processing request or not; and if yes, routing the data processing request to the high-optimization cluster for processing. According to the method, presto is used as a processing cluster, different processing clusters are configured according to priorities, and under the condition that a user has the authority of a high-quality cluster and residual resources of the high-quality cluster can meet processing of a data request needing to be processed by the user, the high-quality cluster is called to process the data processing request of the user. And the data processing request of the user can be timely processed through a dual mechanism of the cluster permission and the cluster residual resources, so that the data processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to a data processing method, system, electronic device and storage medium. Background Art

[0002] As the volume and variety of data continue to grow, Internet applications are increasingly demanding real-time data analysis, such as requiring data processing systems to be able to make accurate business decisions in real time based on ever-changing data. This means that valuable information must be mined from massive amounts of data and various indicators closely related to the business must be defined in advance. Such requirements require not only that the underlying architecture have powerful data storage and analytical computing capabilities, but also that these capabilities be able to meet real-time changing data analysis needs in order to achieve the goals of the enterprise's business. Summary of the invention

[0003] In view of this, embodiments of the present invention provide a data processing method, system, electronic device and storage medium to improve data processing efficiency.

[0004] According to one aspect of the present invention, there is provided a data processing method, the method comprising:

[0005] receiving a data processing request, wherein the data processing request includes a user identifier that sends the data processing request;

[0006] Determine whether the user identifier has high-priority permissions, where the high-priority permissions are permissions to process data through a preset high-priority cluster; the preset high-priority cluster is a presto cluster;

[0007] In the case where the user identifier has high priority authority, determining whether the remaining resources of the high priority cluster meet the preset processing conditions of the data processing request;

[0008] When the remaining resources of the high-priority cluster meet the preset processing conditions, the data processing request is routed to the high-priority cluster for processing.

[0009] In a possible embodiment, the method further includes:

[0010] In the case where the user identifier does not have the high priority authority or the remaining resources of the high priority cluster do not meet the preset processing conditions, determining whether the user identifier has medium priority authority, wherein the medium priority authority is the authority to process data through a preset medium priority cluster, and the machine performance included in the medium priority cluster is less than that of the high priority cluster;

[0011] In the case where the user identifier has medium-priority authority, determining whether the remaining resources of the medium-priority cluster meet the preset processing conditions of the data processing request;

[0012] When the remaining resources of the medium-optimal cluster meet the preset processing conditions, routing the data processing request to the medium-optimal cluster for processing;

[0013] When the remaining resources of the medium-priority cluster do not meet the preset processing conditions, the data processing request is routed to a low-priority cluster for processing, and the performance of machines included in the low-priority cluster is less than that of the high-priority cluster.

[0014] In a possible embodiment, the data processing request further includes user group information, and a user group includes multiple users; the method further includes:

[0015] Determine whether the user group identifier has high priority authority, and if the user identifier has high priority authority, determine whether the remaining resources of the high priority cluster meet the preset processing conditions of the data processing request.

[0016] In a possible embodiment, determining whether the remaining resources of the high-priority cluster meet the preset processing conditions of the data processing request includes:

[0017] Determine a historical request similar to the data processing request in a stored historical request library, wherein the historical request similar to the data processing request has the same processing type and search data amount as the data processing request;

[0018] In a case where the remaining resources of the high-priority cluster are greater than the resources consumed by the historical request, it is determined that the remaining resources of the high-priority cluster meets the preset processing condition of the data processing request.

[0019] In a possible embodiment, the data processing request is issued by a user based on a front-end page, and the method further includes:

[0020] Obtaining a data visualization request, wherein the data visualization request includes a display type, wherein the display type includes a bar chart, a line chart, and a heat map; and displaying a processing result of the data processing request on the front-end page according to the data visualization request;

[0021] Obtain a data export request, wherein the data export request includes a data export format; fill the processing result of the data processing request into a preset template corresponding to the data export format to obtain an export file; and display the export file on the front-end page.

[0022] According to another aspect of the present invention, there is provided a data processing system, the system comprising:

[0023] A receiving module, configured to receive a data processing request, wherein the data processing request includes a user identifier that sends the data processing request;

[0024] A first determination module is used to determine whether the user identifier has a high priority authority, where the high priority authority is the authority to process data through a preset high priority cluster; the preset high priority cluster is a presto cluster;

[0025] A second determination module is used to determine whether the remaining resources of the high-priority cluster meet the preset processing conditions of the data processing request when the user identifier has high-priority authority;

[0026] The routing module is used to route the data processing request to the high priority cluster for processing when the remaining resources of the high priority cluster meet the preset processing conditions.

[0027] In a possible embodiment, the system further includes:

[0028] A third determination module is used to determine whether the user identifier has medium-priority authority if the user identifier does not have the high-priority authority or the remaining resources of the high-priority cluster do not meet the preset processing conditions, wherein the medium-priority authority is the authority to process data through a preset medium-priority cluster, and the machine performance included in the medium-priority cluster is less than that of the high-priority cluster; if the user identifier has medium-priority authority, determine whether the remaining resources of the medium-priority cluster meet the preset processing conditions of the data processing request;

[0029] The routing module is used to route the data processing request to the medium-priority cluster for processing when the remaining resources of the medium-priority cluster meet the preset processing conditions; and to route the data processing request to the low-priority cluster for processing when the remaining resources of the medium-priority cluster do not meet the preset processing conditions, and the low-priority cluster contains machines with less performance than the high-priority cluster.

[0030] In a possible embodiment, determining whether the remaining resources of the high-priority cluster meet the preset processing conditions of the data processing request includes:

[0031] Determine a historical request similar to the data processing request in a stored historical request library, wherein the historical request similar to the data processing request has the same processing type and search data amount as the data processing request;

[0032] In a case where the remaining resources of the high-priority cluster are greater than the resources consumed by the historical request, it is determined that the remaining resources of the high-priority cluster meets the preset processing condition of the data processing request.

[0033] According to another aspect of the present invention, there is provided an electronic device, comprising:

[0034] Processor; and

[0035] Memory for storing programs,

[0036] The program includes instructions, which, when executed by the processor, enable the processor to perform any of the above-mentioned data processing methods.

[0037] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute any of the above-mentioned data processing methods.

[0038] One or more technical solutions provided in the embodiments of the present invention utilize presto as a processing cluster and configure different processing clusters according to priority. When a user has the authority of a high-priority cluster and the remaining resources of the high-priority cluster can satisfy the processing of data requests that the user needs to process, the high-priority cluster is called to process the user's data processing request. The dual mechanism of cluster authority and cluster remaining resources is used to ensure that the user's data processing request can be processed in a timely manner, thereby improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0040] Figure 1 A schematic diagram of a flow chart of a data processing method provided by an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of a flow chart for determining cluster processing authority in a data processing method provided in an embodiment of the present invention;

[0042] Figure 3 An architecture diagram of a data processing platform for implementing the data processing method provided in an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of the structure of a data processing system provided by an embodiment of the present invention;

[0044] Figure 5 A block diagram of an exemplary electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0045] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0046] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0047] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0048] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0049] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0050] In order to improve data processing efficiency, the embodiments of the present invention provide a data processing method, system, electronic device and storage medium. The data processing method provided by the embodiments of the present invention can be applied to any electronic device with data processing function, and the electronic device can be a server, a computer or a mobile terminal, etc. The scheme of the present invention is described below with reference to the accompanying drawings:

[0051] Figure 1 A flow chart of a data processing method provided by an embodiment of the present invention may include the following steps:

[0052] S101, receiving a data processing request, wherein the data processing request includes a user identifier that sends the data processing request;

[0053] S102, determining whether the user identifier has a high priority authority, where the high priority authority is an authority to process data through a preset high priority cluster; the preset high priority cluster is a presto cluster;

[0054] S103: If the user identifier has high priority authority, determine whether the remaining resources of the high priority cluster meet the preset processing conditions of the data processing request;

[0055] S104: When the remaining resources of the high-priority cluster meet the preset processing conditions, route the data processing request to the high-priority cluster for processing.

[0056] By applying the embodiment of the present invention, presto is used as a processing cluster, and different processing clusters are configured according to priorities. When a user has the authority of a high-priority cluster and the remaining resources of the high-priority cluster can satisfy the processing of data requests that the user needs to process, the high-priority cluster is called to process the user's data processing request. The dual mechanism of cluster authority and cluster remaining resources is used to ensure that the user's data processing request can be processed in a timely manner, thereby improving data processing efficiency.

[0057] The above S101-S104 are exemplarily described below:

[0058] The data processing request is issued by a user, and may include the user identifier of the user who sends the request, the type of processing request, and the identifier of the data to be processed, etc. The user identifier may be a user number, a user name, etc. The data processing request may be of various types, such as a data change request, a data query request, etc. The data identifier may be composed of the business to which the data belongs, the data name, the data code, or the time when the data is generated, etc.

[0059] In a possible embodiment, the data processing method provided by the embodiment of the present invention can be implemented by a data processing platform, and the data processing platform can include a front-end page and a back-end processing device, wherein the user can implement the exchange with the back-end processing device based on the front-end page, and the back-end processing device is usually a server. Accordingly, in a possible embodiment, the user can issue the above-mentioned data processing request based on the front-end page.

[0060] The above-mentioned data processing request can be a pre-registered request or a temporary data processing task. The above-mentioned pre-registered request refers to the pre-registration of the type of data processing request, business identification, related database and other information in the data processing platform. The above-mentioned temporary data processing task is a data processing request issued by the user to flexibly respond to sudden data needs.

[0061] In a possible embodiment, the data processing platform can pre-authorize users to ensure that only authorized users can access the platform. Different permissions can also be set for different users, such as query permissions, change permissions, etc. The above permissions can be set for databases, data tables or individual data, so as to limit users' access rights to specific data. At the same time, the data processing platform can record user operation logs to track and audit user behavior.

[0062] As a possible implementation, the user may input data processing instructions in the front-end page, such as query SQL, which may include the database where the data the user needs to query is located, the data identifier to be queried, etc.

[0063] In a possible embodiment, since different users may have different database modification or query permissions, the front-end page may provide database table information for which the user has permissions, so that the user can quickly filter out the table structure information available to him. As a possible implementation method, the data processing platform may pre-store the correspondence between users and databases, and may provide a query page. When the user enters the query page, the back-end processing device may determine the databases for which the user has modification or query permissions based on the user identifier, and display these databases in the query page in the form of a table. Specifically, the database name, database code, and other information may be displayed.

[0064] In a possible embodiment, the above-mentioned data processing platform can also provide a one-click quick generation of simple query SQL function, so that users can quickly retrieve the required data. As a possible implementation, a query SQL template can be pre-configured, and the variables in the template, such as database identifiers and data identifiers, can be replaced by preset placeholders. When the user issues a one-click query request based on the database displayed on the query page, the database identifier displayed on the query page can be used to replace the corresponding position placeholder in the query SQL template, thereby generating query SQL.

[0065] In a possible embodiment, the correspondence between cluster processing permissions and user identifiers can be pre-stored in the back-end processing device, and the correspondence can be flexibly configured according to the actual business scenario. In the embodiment of the present invention, clusters of different priorities can be configured according to actual needs. As a possible implementation method, high-priority clusters, medium-priority clusters and low-priority clusters can be configured. These three processing clusters are all presto clusters. Presto is an independent query engine with great flexibility. It can be integrated with a variety of data sources and is very simple and convenient to deploy. The above-mentioned high-priority cluster can be composed of high-performance servers, which refers to servers with higher CPU and memory performance. The medium-priority cluster is composed of medium-performance servers, which refers to servers with lower CPU and memory performance than high-performance servers. The low-priority cluster is composed of low-performance servers, and the CPU and memory performance of the low-performance server are lower than those of the medium-performance server. After determining to configure different processing clusters, permissions can be configured for different clusters. The permissions may include update or query permissions. Specifically, permission types and user identifiers can be stored for different clusters.

[0066] After receiving the data processing request, the back-end processing device can determine whether the user included in the data processing request has the corresponding cluster authority based on the authority types and user identifiers of the above-mentioned different clusters. As a possible implementation method, it can be first determined whether there is a corresponding relationship between the target user identifier and the target processing type in the corresponding relationship between the authority type and the user identifier in the high-priority cluster. If so, it means that the user has high-priority authority, and then it can be determined whether the remaining resources of the high-priority cluster meet the processing conditions of the data processing request. If not, it means that the user does not have high-priority authority, and it can be determined whether the user has medium-priority authority. If the user does not have medium-priority authority, the data processing request can be sent to the low-priority cluster for processing. In the embodiment of the present invention, the process of determining whether the user has high-priority authority and medium-priority authority is the same, and the process of determining whether the remaining resources in high-priority authority, medium-priority authority or low-priority authority meet the processing conditions of the data processing request is also the same. Here, the high-priority authority is used as an example to illustrate the above two processes. The two processing processes of medium-priority authority and low-priority authority can refer to the description of high-priority authority, which will not be repeated in this article.

[0067] After determining that the user has high priority permissions, it can be determined whether the remaining resources in the high priority cluster meet the processing conditions of the data processing request, which is the amount of resources required to process the data processing request. As a possible implementation, the amount of data included in the data processing request or the amount of data required to be returned for the data processing request can be used as the processing condition of the data processing request.

[0068] In a possible embodiment, the resource usage in the high-priority cluster can be monitored in real time, including CPU, memory, storage, network bandwidth, etc. If the target user has high-priority permissions, it can be determined based on the monitoring data whether the remaining amount of resources in the high-priority cluster meets the processing conditions of the data processing request. As a possible implementation method, a conversion relationship between indicators such as CPU resources and memory and the amount of data can be established in advance, and based on the amount of data corresponding to the remaining resources in the high-priority cluster and the amount of data included in the data processing request, it can be determined whether the high-priority cluster can process the data processing request.

[0069] In a possible embodiment, queues can be established for tasks of different clusters, and the queue length can be set based on the resources of the cluster. When the number of tasks to be processed in the queue corresponding to the high-priority cluster exceeds a preset threshold, it can also be determined that the remaining resources of the high-priority cluster are unable to process the data processing request.

[0070] In a possible embodiment, the resource demand when executing tasks similar to the current data processing request can be predicted based on historical task execution data and trend analysis to determine whether high-priority cluster resources are sufficient for fast and efficient execution of the task.

[0071] As a possible implementation method, historical requests similar to the data processing request can be determined in the stored historical request library, and the historical requests similar to the data processing request have the same processing type and the same amount of search data as the data processing request; when the remaining resources of the high-priority cluster are greater than the resources consumed by the historical requests, it is determined that the remaining resources of the high-priority cluster meet the preset processing conditions of the data processing request.

[0072] The above-mentioned historical request library stores each received historical request. Specifically, the business to which the request belongs, the request type, the amount of query data, and the receiving time can be stored accordingly. When determining similar requests to the data processing request, the historical requests of the same business as the data processing request can be determined based on the business identifier, and then the historical requests with the same request type as the data processing request can be determined from these historical requests, and then the requests with the same amount of data can be determined from these requests of the same type. The same here can mean that the difference in data volume is less than a preset quantity threshold.

[0073] In a possible embodiment, when a low-priority cluster is also unable to process a data processing request, insufficient resources information may be displayed on the front-end page to remind the user to submit the task later.

[0074] In a possible embodiment, the above data processing request may also include user group information to which the user belongs. The user group may include multiple users. The user group may be set according to actual business needs, such as users belonging to the same business may be set as the same user group. Correspondingly, the permissions preset for each processing cluster may also include user group information. After receiving the data processing request, it may be determined whether the corresponding relationship between the user identifier or user group identifier and the processing type exists in the permissions of the processing cluster. The determination process is similar to the above, and only a brief description is given here, and no further elaboration is given.

[0075] like Figure 2 As shown, Figure 2 A schematic diagram of a process for determining a processing cluster for processing a data processing request in an embodiment of the present invention may include the following process:

[0076] S201. Obtain user information, which includes user name and user group information.

[0077] S202: Determine whether the user has high priority permissions. If yes, execute S206; if not, execute S203.

[0078] S203: Determine whether the user group has high priority permissions. If yes, execute S206; if no, execute S204.

[0079] S204: Determine whether the user has ordinary authority. If so, execute S207; otherwise, execute S205.

[0080] The ordinary permissions here are the medium-priority permissions mentioned above.

[0081] S205: Determine whether the user group has common permissions. If yes, execute S207; otherwise, execute S206.

[0082] S206: Determine whether the high-priority cluster is idle. If it is idle, select the high-priority cluster. If it is not idle, execute S207.

[0083] S207: Determine whether the common cluster is idle. If it is idle, select the common cluster. If it is not idle, execute S208.

[0084] S208: Determine whether the low-priority cluster is idle. If so, select the low-priority cluster. If not, return insufficient resource information to remind the user to submit the task later.

[0085] In the case where the high-priority cluster cannot process a data processing request, in addition to determining whether the medium-priority cluster can process the data processing request, the cluster's resource configuration can be adjusted in a timely manner to meet future business needs and avoid insufficient or wasted resources. Specifically, a resource utilization threshold can be set for each device in the high-priority cluster. When it is detected that the resource utilization of a device is higher than the above threshold, the task allocation to the device can be stopped, and the data processing tasks that have been allocated to the device can be migrated, such as to other devices in the same cluster or to other clusters.

[0086] In a possible embodiment, users can upload local or external data, files, etc. to the data processing platform, and the data processing platform can automatically create corresponding data tables or data sets for subsequent data processing and analysis. As a possible implementation method, data structure and field information can be identified, and the data type corresponding to the field can be used as a key to establish a KV data table or data set, so that the data processing platform can handle richer data requests.

[0087] In a possible embodiment, users can authorize the publication of data processing tasks, that is, publish data tasks to the platform for other users to query and solve. This can help team members share knowledge and experience, jointly solve complex data analysis problems, and accelerate the iteration and optimization of data analysis processes.

[0088] In a possible embodiment, faults and abnormal conditions in the cluster can also be detected regularly, and early warning notifications can be issued in a timely manner, including node downtime, memory or storage resource exhaustion, network unavailability, etc., so that the administrator can take timely measures to solve the problem and ensure the stable operation of the system.

[0089] In a possible embodiment, the data processing platform can also manage the data processing request. In addition to executing the operations of creating, modifying, and publishing requests by users, it also supports operations such as data rollback. Specifically, different requests corresponding to the same task identifier can be saved. The task identifier can be a business identifier. When receiving the comparison or rollback request output by the user, the data returned for each request can be compared or rolled back, which helps to track the change history of the task and ensure the consistency and traceability of data processing. For the above temporary tasks, there is no need to save the request version.

[0090] After processing the above data processing request, the data processing platform can display the query results to the user in an intuitive form by opening it in online Excel. In a possible embodiment, the user can send a visualization request based on the front-end page, and the visualization request can include a display type, which can include a bar chart, a line chart, a pie chart, a heat map, etc. The user can choose a suitable visualization method according to his needs to intuitively understand and analyze the data.

[0091] In a possible embodiment, the user can also export the data processing results. Specifically, export templates corresponding to files in different formats can be pre-set. The user can send a data export request based on the front-end page. The data export request can include the export format, such as TXT, csv, Excel, etc. The data processing platform can fill in the query results into the corresponding template based on the data export request and return it to the user. The user can export the data to the local for further processing or share it with others.

[0092] In a possible embodiment, the front-end page also supports a page customization function, and users can customize and personalize the interface according to their needs, such as adjusting the interface layout, selecting favorite themes and color schemes, and setting shortcuts to commonly used functions to improve user experience and work efficiency.

[0093] like Figure 3 As shown, Figure 3 An architecture diagram of a data processing platform for implementing the data processing method provided in an embodiment of the present invention, the data processing platform may include a page module, a function module, an execution module, a detection module and a presto cluster, wherein the page module provides an intuitive and friendly user interface for users, enabling users to easily interact with the platform and perform data query, visualization and analysis. The page module provides the database table information of the user's permissions, and the user can quickly filter out the table structure information available to him, and provides a one-key quick generation of a simple query SQL function, which can quickly retrieve the required data. The query results can be displayed to the user in an intuitive form by opening it in online Excel. Users can choose a suitable visualization method according to their needs, intuitively understand and analyze the data, and the supported data visualization types include bar charts, line charts, pie charts, heat maps, etc. It also supports users to customize and personalize the interface according to their needs. Users can adjust the interface layout, select favorite themes and color schemes, and set shortcuts for commonly used functions to improve user experience and work efficiency. The page module also supports users to generate reports or export data based on query results.

[0094] The functional module provides temporary task functions, task version management, data upload generation table, permission management and collaborative work functions. Among them, data upload generation table means allowing users to upload local or external data to the big data platform and automatically create corresponding data tables or data sets for subsequent data processing and analysis. The above data can be data or files. The permission management function provides user identity authentication and authorization functions to ensure that only authorized users can access the platform; supports role-based permission management to define the permissions and access scope of different roles; provides data-level permission control to limit user access to specific data; records user operation logs to track and audit user behavior. The collaborative work function means that users can publish and authorize their own tasks, allowing users to help solve problems.

[0095] The detection module is used to implement functions such as resource monitoring, resource prediction, fault detection and early warning, resource scheduling and management, etc. The implementation methods of the above functions have been exemplarily described above and will not be repeated here.

[0096] By applying the embodiment of the present invention, an efficient resource allocation and intelligent queue control system is realized for the innovative technology of cluster resource utilization and task scheduling. By comprehensively considering the priority of the task, resource requirements and the actual load of the cluster, an advanced dynamic scheduling algorithm and intelligent queue management strategy are adopted to ensure that high-priority tasks can quickly obtain the required resources and enter the execution stage in the most optimized way. While realizing the rapid generation of key data, this technology can also maximize the performance and resource utilization efficiency of the overall system, providing reliable technical support for the stable operation and efficient management of the big data processing platform. In addition, based on advanced user interaction design and data visualization technology, the embodiment of the present invention provides an intuitive, friendly and powerful data analysis platform. Users can interact efficiently with the big data processing system through the platform, and use the rich analysis tools built in it to deeply mine and analyze the data. In addition to basic data query and visualization functions, the module also provides a variety of advanced analysis tools, such as pivot tables, time series analysis, etc., to meet the diverse needs of users for data analysis. In addition, functions such as interactive visualization and custom Excel generation also provide users with a more flexible and customized data analysis experience, helping users to quickly discover the value and insights in the data and achieve the goal of data-driven decision-making.

[0097] Based on the same inventive concept, the embodiment of the present invention also provides a data processing system, such as Figure 4 As shown, the system 400 includes:

[0098] A receiving module 401 is used to receive a data processing request, wherein the data processing request includes a user identifier that sends the data processing request;

[0099] The first determination module 402 is used to determine whether the user identifier has a high priority permission, where the high priority permission is the permission to process data through a preset high priority cluster; the preset high priority cluster is a presto cluster;

[0100] The second determination module 403 is used to determine whether the remaining resources of the high-priority cluster meet the preset processing conditions of the data processing request when the user identifier has high-priority authority;

[0101] The routing module 404 is used to route the data processing request to the high priority cluster for processing when the remaining resources of the high priority cluster meet the preset processing conditions.

[0102] In a possible embodiment, the system further includes:

[0103] A third determination module is used to determine whether the user identifier has medium-priority authority if the user identifier does not have the high-priority authority or the remaining resources of the high-priority cluster do not meet the preset processing conditions, wherein the medium-priority authority is the authority to process data through a preset medium-priority cluster, and the machine performance included in the medium-priority cluster is less than that of the high-priority cluster; if the user identifier has medium-priority authority, determine whether the remaining resources of the medium-priority cluster meet the preset processing conditions of the data processing request;

[0104] The routing module is used to route the data processing request to the medium-priority cluster for processing when the remaining resources of the medium-priority cluster meet the preset processing conditions; and to route the data processing request to the low-priority cluster for processing when the remaining resources of the medium-priority cluster do not meet the preset processing conditions, and the low-priority cluster contains machines with less performance than the high-priority cluster.

[0105] In a possible embodiment, determining whether the remaining resources of the high-priority cluster meet the preset processing conditions of the data processing request includes:

[0106] Determine a historical request similar to the data processing request in a stored historical request library, wherein the historical request similar to the data processing request has the same processing type and search data amount as the data processing request;

[0107] In a case where the remaining resources of the high-priority cluster are greater than the resources consumed by the historical request, it is determined that the remaining resources of the high-priority cluster meets the preset processing condition of the data processing request.

[0108] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0109] The exemplary embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present invention when executed by the at least one processor.

[0110] Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0111] An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform a method according to an embodiment of the present invention.

[0112] refer to Figure 5 , a block diagram of an electronic device 500 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0113] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0114] A plurality of components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 may be any type of device capable of inputting information to the electronic device 500, and the input unit 506 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 507 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 508 may include, but is not limited to, a disk, an optical disk. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0115] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the above-mentioned data processing method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. In some embodiments, the computing unit 501 may be configured to perform the above-mentioned data processing method by any other appropriate means (e.g., by means of firmware).

[0116] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0117] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0118] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0119] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0120] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0121] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A data processing method, characterized in that: The method comprises: receiving a data processing request, wherein the data processing request includes a user identifier that sends the data processing request; Determine whether the user identifier has high-priority permissions, where the high-priority permissions are permissions to process data through a preset high-priority cluster; the preset high-priority cluster is a presto cluster; In the case where the user identifier has high priority authority, determining whether the remaining resources of the high priority cluster meet the preset processing conditions of the data processing request; When the remaining resources of the high-priority cluster meet the preset processing conditions, the data processing request is routed to the high-priority cluster for processing.

2. The method according to claim 1, characterized in that The method further comprises: In the case where the user identifier does not have the high priority authority or the remaining resources of the high priority cluster do not meet the preset processing conditions, determining whether the user identifier has medium priority authority, wherein the medium priority authority is the authority to process data through a preset medium priority cluster, and the machine performance included in the medium priority cluster is less than that of the high priority cluster; In the case where the user identifier has medium-priority authority, determining whether the remaining resources of the medium-priority cluster meet the preset processing conditions of the data processing request; When the remaining resources of the medium-optimal cluster meet the preset processing conditions, routing the data processing request to the medium-optimal cluster for processing; When the remaining resources of the medium-priority cluster do not meet the preset processing conditions, the data processing request is routed to a low-priority cluster for processing, and the performance of machines included in the low-priority cluster is less than that of the high-priority cluster.

3. The method according to claim 1, characterized in that The data processing request also includes user group information, and a user group includes multiple users; the method also includes: Determine whether the user group identifier has high priority authority, and if the user identifier has high priority authority, determine whether the remaining resources of the high priority cluster meet the preset processing conditions of the data processing request.

4. The method according to claim 1, characterized in that: The determining whether the remaining resources of the high priority cluster meet the preset processing conditions of the data processing request includes: Determine a historical request similar to the data processing request in a stored historical request library, wherein the historical request similar to the data processing request has the same processing type and search data amount as the data processing request; In a case where the remaining resources of the high-priority cluster are greater than the resources consumed by the historical request, it is determined that the remaining resources of the high-priority cluster meets the preset processing condition of the data processing request.

5. The method according to claim 1, characterized in that The data processing request is issued by the user based on the front-end page, and the method further includes: Obtaining a data visualization request, wherein the data visualization request includes a display type, wherein the display type includes a bar chart, a line chart, and a heat map; and displaying a processing result of the data processing request on the front-end page according to the data visualization request; Obtain a data export request, wherein the data export request includes a data export format; fill the processing result of the data processing request into a preset template corresponding to the data export format to obtain an export file; and display the export file on the front-end page.

6. A data processing system, characterized in that: The system comprises: A receiving module, configured to receive a data processing request, wherein the data processing request includes a user identifier that sends the data processing request; A first determination module is used to determine whether the user identifier has a high priority authority, where the high priority authority is the authority to process data through a preset high priority cluster; the preset high priority cluster is a presto cluster; A second determination module is used to determine whether the remaining resources of the high-priority cluster meet the preset processing conditions of the data processing request when the user identifier has high-priority authority; The routing module is used to route the data processing request to the high priority cluster for processing when the remaining resources of the high priority cluster meet the preset processing conditions.

7. The system according to claim 6, characterized in that The system further comprises: A third determination module is used to determine whether the user identifier has medium-priority authority if the user identifier does not have the high-priority authority or the remaining resources of the high-priority cluster do not meet the preset processing conditions, wherein the medium-priority authority is the authority to process data through a preset medium-priority cluster, and the machine performance included in the medium-priority cluster is less than that of the high-priority cluster; if the user identifier has medium-priority authority, determine whether the remaining resources of the medium-priority cluster meet the preset processing conditions of the data processing request; The routing module is used to route the data processing request to the medium-priority cluster for processing when the remaining resources of the medium-priority cluster meet the preset processing conditions; and to route the data processing request to the low-priority cluster for processing when the remaining resources of the medium-priority cluster do not meet the preset processing conditions, and the low-priority cluster contains machines with less performance than the high-priority cluster.

8. The system according to claim 6, characterized in that The determining whether the remaining resources of the high priority cluster meet the preset processing conditions of the data processing request includes: Determine a historical request similar to the data processing request in a stored historical request library, wherein the historical request similar to the data processing request has the same processing type and search data amount as the data processing request; In a case where the remaining resources of the high-priority cluster are greater than the resources consumed by the historical request, it is determined that the remaining resources of the high-priority cluster meets the preset processing condition of the data processing request.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 5.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-5.

Images