Data processing method, device, equipment, storage medium and product

Abstract

The present application discloses a data processing method, apparatus, device, storage medium and product, which belongs to the field of data processing technology. The method includes: displaying a storage data management page; determining the storage management configuration information corresponding to the target data cluster in response to the information configuration operation received on the storage data management page; sending the storage management configuration information so that the first device performs the data processing task for the target data cluster when the cluster operation information corresponding to the target data cluster meets the storage data management conditions; receiving the task execution data corresponding to the data processing task; based on the task execution data, displaying the task execution information corresponding to the target data cluster in a preset page. In the technical solution provided by the embodiment of the present application, automated data operation and maintenance can be achieved by configuring the storage management configuration information on the storage data management page, and the data operation and maintenance information can also be displayed on the page, which improves the visibility of the data operation and maintenance information.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, storage medium and product. Background Art

[0002] With the research and advancement of computer technology and Internet technology, more and more data are generated in all walks of life, and how to manage massive amounts of stored data has become a problem of concern to people.

[0003] In related technologies, users can use the current big data processing framework to write their own program code to submit multiple tasks, and then control the equipment to perform data operations and processing of corresponding tasks on the data stored in the database to realize the operation and maintenance of the stored data.

[0004] In related technologies, users need to frequently write and modify program codes, the labor cost of data operation and maintenance is high, the degree of automation is low, and the information displayed by data operation and maintenance results is limited. Summary of the Invention

[0005] The embodiments of the present application provide a data processing method, apparatus, equipment, storage medium, and product that can automate data operation and maintenance, reduce data operation and maintenance workload, improve data operation and maintenance efficiency, and data storage efficiency within a data cluster, and can increase the amount of information displayed in the data operation and maintenance results, thereby improving the user experience.

[0006] According to one aspect of an embodiment of the present application, a data processing method is provided, the method comprising:

[0007] Display the storage data management page;

[0008] In response to an information configuration operation received on the storage data management page, determining storage management configuration information corresponding to a target data cluster, wherein the storage management configuration information is used to determine a storage data management condition corresponding to the target data cluster;

[0009] Sending the storage management configuration information so that the first device executes a data processing task for the target data cluster if the cluster operation information corresponding to the target data cluster satisfies the storage data management condition, where the storage data management condition refers to a triggering condition for the data processing task;

[0010] receiving task execution data corresponding to the data processing task;

[0011] Based on the task execution data, task execution information corresponding to the target data cluster is displayed on a preset page.

[0012] According to one aspect of an embodiment of the present application, a data processing method is provided, the method comprising:

[0013] Get the cluster operation information corresponding to the target data cluster;

[0014] Receive storage management configuration information corresponding to the target data cluster, where the storage management configuration information is configuration information determined by the second device according to an information configuration operation received on a storage data management page;

[0015] Determining storage data management conditions corresponding to the target data cluster based on the storage management configuration information;

[0016] When the cluster operation information satisfies the storage data management condition, executing a data processing task for the target data cluster and obtaining task execution data corresponding to the data processing task;

[0017] The task execution data is sent so that the second device displays task execution information corresponding to the target data cluster in a preset page based on the task execution data.

[0018] According to one aspect of an embodiment of the present application, a data processing device is provided, the device comprising:

[0019] Management page display module, used to display the storage data management page;

[0020] a configuration information determining module, configured to determine storage management configuration information corresponding to a target data cluster in response to an information configuration operation received on the storage data management page, wherein the storage management configuration information is used to determine storage data management conditions corresponding to the target data cluster;

[0021] a configuration information sending module, configured to send the storage management configuration information so that the first device executes the data processing task for the target data cluster if the cluster operation information corresponding to the target data cluster satisfies the storage data management condition, wherein the storage data management condition refers to a triggering condition for the data processing task;

[0022] A task data receiving module, configured to receive task execution data corresponding to the data processing task;

[0023] The task information display module is used to display the task execution information corresponding to the target data cluster in a preset page based on the task execution data.

[0024] According to one aspect of an embodiment of the present application, a data processing device is provided, the device comprising:

[0025] The operation information acquisition module is used to obtain the cluster operation information corresponding to the target data cluster;

[0026] a configuration information receiving module, configured to receive storage management configuration information corresponding to a target data cluster, wherein the storage management configuration information is configuration information determined by the second device according to an information configuration operation received on a storage data management page;

[0027] A management condition determination module, configured to determine a storage data management condition corresponding to the target data cluster based on the storage management configuration information;

[0028] A data task execution module, configured to execute a data processing task for the target data cluster and obtain task execution data corresponding to the data processing task if the cluster operation information satisfies the storage data management condition;

[0029] The task data sending module is used to send the task execution data so that the second device displays the task execution information corresponding to the target data cluster in a preset page based on the task execution data.

[0030] According to one aspect of an embodiment of the present application, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the above-mentioned data processing method.

[0031] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the above-mentioned data processing method.

[0032] According to one aspect of an embodiment of the present application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to execute and implement the above-described data processing method.

[0033] The technical solutions provided in the embodiments of the present application can bring the following beneficial effects:

[0034] By receiving the information configuration operation on the storage data management page, the storage management configuration information corresponding to the target data cluster is determined and the storage management configuration information is sent, so that the first device can determine the storage data management conditions based on the received storage management configuration information and judge the cluster operation information based on the conditions, thereby automatically executing the corresponding data processing tasks and sending the task execution data to the local device. The local device can display the task execution information within the data cluster on a preset page based on the received task execution data, thereby realizing data operation and maintenance automation, reducing data operation and maintenance workload, improving data operation and maintenance efficiency and data storage efficiency within the data cluster, and also increasing the amount of information displayed in the data operation and maintenance results, thereby improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0036] Figure 1 is a schematic diagram of an application program operating environment provided by an embodiment of the present application;

[0037] Figure 2 This is the process of the data processing method provided by an embodiment of the present application Figure 1 ;

[0038] Figure 3 This is the process of the data processing method provided by an embodiment of the present application Figure 2 ;

[0039] Figure 4 A schematic diagram of a storage data management page is exemplarily shown;

[0040] Figure 5 This is the process of the data processing method provided by an embodiment of the present application Figure 3 ;

[0041] Figure 6 This is the process of the data processing method provided by an embodiment of the present application Figure 4 ;

[0042] Figure 7 This is the process of the data processing method provided by an embodiment of the present application Figure 5 ;

[0043] Figure 8 The following is a schematic diagram showing the technical architecture of an automated data operation and maintenance kernel;

[0044] Figure 9 The following is a diagram showing an example of a technical architecture of an automated data operation and maintenance service;

[0045] Figure 10 This is the interactive process of the data processing method provided by an embodiment of the present application Figure 1 ;

[0046] Figure 11 This is the interactive process of the data processing method provided by an embodiment of the present application Figure 2 ;

[0047] Figure 12 An overall flow chart of an automated data operation and maintenance service is shown as an example;

[0048] Figure 13 It is a frame of a data processing device provided by an embodiment of the present application. Figure 1 ;

[0049] Figure 14 It is a frame of a data processing device provided by an embodiment of the present application. Figure 2 ;

[0050] Figure 15 This is a structural frame of a computer device provided by an embodiment of the present application. Figure 1 ;

[0051] Figure 16 This is a structural frame of a computer device provided by an embodiment of the present application. Figure 2 . DETAILED DESCRIPTION

[0052] The data processing method provided in the embodiments of the present application involves cloud technology, which is briefly described below to facilitate understanding by those skilled in the art.

[0053] Cloud computing is a computing model that distributes computing tasks across a resource pool consisting of a large number of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network that provides these resources is called the "cloud." To users, these resources appear infinitely scalable and can be accessed at any time, used on demand, expanded at any time, and paid for on a per-use basis.

[0054] As a provider of cloud computing infrastructure, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) is established. Various types of virtual resources are deployed in the resource pool for external customers to choose and use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0055] Based on logical functional divisions, the PaaS (Platform as a Service) layer can be deployed on top of the IaaS (Infrastructure as a Service) layer, and the SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. SaaS can also be deployed directly on top of IaaS. PaaS is a platform for software execution, such as databases and web containers. SaaS is a variety of business software, such as web portals and text messaging tools. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0056] Cloud storage is a new concept that has been extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0057] Currently, storage systems utilize a method for creating logical volumes. When creating a logical volume, physical storage space is allocated for each logical volume. This physical storage space may consist of disks on a specific storage device or several storage devices. When a client stores data on a logical volume, it stores the data on a file system. The file system divides the data into multiple parts, each of which is an object. An object contains not only the data but also additional information such as the data identifier (ID) of the data entity. The file system writes each object to the physical storage space of the logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can provide access to the data based on the storage location information of each object.

[0058] The storage system allocates physical storage space to logical volumes by pre-dividing the physical storage space into stripes based on the estimated capacity of the objects to be stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the Redundant Array of Independent Disks (RAID) groupings. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0059] A database, in short, can be thought of as a digital filing cabinet—a place where electronic files are stored, where users can add, query, update, and delete data. A database is a collection of data stored in a specific way, shared by multiple users, with minimal redundancy, and independent of applications.

[0060] A database management system (DBMS) is a computer software system designed for managing databases, typically providing basic functions such as storage, retrieval, security, and backup. DBMSs can be categorized by the database model they support, such as relational or XML (Extensible Markup Language); by the type of computer they support, such as server clusters or mobile phones; by the query language used, such as SQL (Structured Query Language) or XQuery; by performance priorities, such as maximum scale or maximum speed; or by other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, supporting multiple query languages simultaneously.

[0061] Big data refers to collections of data that cannot be captured, managed, and processed within a specific timeframe using conventional software tools. These massive, rapidly growing, and diverse information assets require new processing models to enhance decision-making, insight discovery, and process optimization. With the advent of the cloud era, big data has attracted increasing attention. Big data requires specialized technologies to efficiently process large amounts of time-sensitive data. Technologies suitable for big data include massively parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the internet, and scalable storage systems.

[0062] In order to store big data in a more reasonable organizational format and reduce the cost of user learning and operation and maintenance, the embodiment of the present application provides a data processing method to realize automated data operation and maintenance of data in the data lake.

[0063] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0064] Please refer to Figure 1 , which shows a schematic diagram of an application program running environment provided by an embodiment of the present application. The application program running environment may include: a terminal 10 and a server 20.

[0065] The terminal 10 includes, but is not limited to, electronic devices such as mobile phones, computers, intelligent voice interaction devices, smart home appliances, car terminals, game consoles, e-book readers, multimedia playback devices, wearable devices, etc. The terminal 10 may be installed with a client of an application.

[0066] In the embodiments of the present application, the above-mentioned application can be any application that can provide data operation and maintenance services. Typically, the application is a browser application. Of course, in addition to browser applications, other types of applications can also provide data operation and maintenance services, and the embodiments of the present application do not limit this. In addition, for different applications, the data operation and maintenance services they provide will also be different, and the types and attributes of the data involved will also be different. This can be pre-configured according to actual needs, and the embodiments of the present application do not limit this.

[0067] The server 20 is used to provide background services for the client of the application in the terminal 10. For example, the server 20 can be the background server of the above-mentioned application. The server 20 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the server 20 provides background services for applications in multiple terminals 10 at the same time.

[0068] Optionally, the terminal 10 and the server 20 may communicate with each other via a network 30. The terminal 10 and the server 20 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0069] In an exemplary embodiment, a highly available data operation and maintenance service operating environment configuration scheme is provided. The above-mentioned server 20 includes a proxy server 21 and a back-end server 22. The back-end server can manage the main database 23, and the backup database 24 is used to back up the main database 23. Optionally, the data operation and maintenance service adopts spring microservices, which has the advantages of statelessness, multi-active, unified domain name, floating IP access, etc. The above-mentioned proxy server 21 can be an nginx server, which is used to achieve load balancing, can reasonably schedule the requests sent by the terminal 10, and support dual-instance operation of the front end. Specifically, the terminal 10 sends a Hypertext Transfer Protocol (HTTP) request to the nginx server, the nginx server receives the request and distributes it to the appropriate back-end server 22 according to the load balancing algorithm, and the back-end server 22 sends the processing information to the main database 23. In the above-mentioned high-availability implementation mechanism, it has the following characteristics:

[0070] 1. User requests are sent based on the domain name.

[0071] 2. Evenly distribute the nginx load balancer to several backend servers 23 in the background.

[0072] 3. Several back-end servers 23 are deployed at the back end to perform data processing tasks.

[0073] 4. All servers are connected to a master database, and the database implements master-slave and disaster recovery.

[0074] Optionally, the above data transmission method is not limited to the HTTP request method, and data transmission can also be performed through methods such as RPC (Remote Procedure Call Protocol).

[0075] Please refer to Figure 2 , which shows the process of the data processing method provided by an embodiment of the present application Figure 1 This method can be applied to computer equipment, which refers to electronic equipment with data calculation and processing capabilities. For example, the execution subject of each step can be Figure 1 The terminal 10 in the application running environment is shown. The method may include the following steps (210-250).

[0076] Step 210: Display the storage data management page.

[0077] The storage data management page is used to configure data operations and maintenance services for the target data cluster. It is used to receive configuration operations for the configuration attribute fields of the target data cluster. The storage data management page includes configuration components corresponding to multiple configuration attribute fields of the target data cluster. Optionally, these configuration attribute fields have corresponding default values. The storage data management page also allows users to set specific data corresponding to the configuration attribute fields according to their needs, thereby generating the storage management configuration information (table config) described below.

[0078] In a possible implementation, the above-mentioned storage data management page includes but is not limited to a browser page, a client page, and a program development editor page.

[0079] In specific data operation and maintenance service scenarios, users can set data operation and maintenance parameters on the front-end interface, namely the storage data management page, and store them in the database without having to read the HDFS (Hadoop Distributed File System), achieving lightweight read configuration. Several optimization properties for the table can be added on the storage data management page, such as the interval event time. When enabling configuration, the user must be authenticated to determine whether the user has permission to the table. The storage data management page can display status such as the total number of file rows in the table to guide users in setting parameters. The data operation and maintenance service corresponding to the storage data management page supports a variety of services, including file merging services, data snapshot cleanup services, expired file cleanup services, orphan file cleanup services, and data layout optimization services. The storage data management page supports configuration operations such as SQL (Structured Query Language), OpenAPI (Open Application Programming Interface), multi-language SDKs (Software Development Kits), and Service HA (Service High Available).

[0080] In an exemplary embodiment, the storage data management page may display management configuration prompt information, wherein the management configuration prompt information is management configuration information adaptively determined based on cluster operation information and is used to provide configuration reference information of the management configuration information to the target object.

[0081] Correspondingly, such as Figure 3 As shown, the above method further includes the following steps (260-270): Figure 3 The flow chart of the data processing method provided by one embodiment of the present application is shown as follows Figure 2 .

[0082] Step 260: Receive management configuration prompt information corresponding to the target data cluster.

[0083] During the process of receiving the target object's information configuration operation on the storage data management page, the management configuration prompt information sent by the first device is received to prompt the target object to perform the information configuration operation that matches the target data cluster operation status. The management configuration prompt information is reference configuration information corresponding to the preset configuration attribute field displayed to the target object.

[0084] The management configuration prompt information includes a recommended value corresponding to at least one configuration attribute field. The recommended value can be used as reference information for performing information configuration operations on the target object.

[0085] The management configuration prompt information includes at least one suggested option corresponding to a configuration attribute field. The suggested option can be used as reference information for performing information configuration operations on the target object.

[0086] Step 270: Display management configuration prompt information on the storage data management page.

[0087] Correspondingly, after receiving the management configuration prompt information sent by the first device, the management configuration prompt information may be displayed on the storage data management page, such as displaying a recommended value or recommended option corresponding to at least one configuration attribute field.

[0088] In this embodiment, the system can make personalized recommendation settings for data operation and maintenance based on the data operation and maintenance information, generate the above-mentioned management configuration prompt information, and meet different operation and maintenance needs.

[0089] Step 220 : In response to the information configuration operation received on the storage data management page, determine the storage management configuration information corresponding to the target data cluster.

[0090] The storage management configuration information is used to determine the storage data management conditions corresponding to the target data cluster. The storage data management conditions are used to trigger the execution of data processing tasks provided by the data operation and maintenance service.

[0091] The user can perform configuration operations for data operation and maintenance of the target data cluster on the storage data management page to generate the above storage management configuration information to determine storage data management conditions for processing storage data in the target data cluster.

[0092] The configuration data corresponding to the preset configuration attribute field in the above storage management configuration information can be used as the threshold corresponding to the storage data management condition.

[0093] In one possible implementation, the stored data management page includes configuration components corresponding to multiple configuration attribute fields. The information configuration operations include, but are not limited to, value entry and option selection for the configuration components. Users can perform corresponding operations on the configuration components, such as value entry and option selection, to set configuration data corresponding to the configuration attribute fields.

[0094] In one example, if Figure 4 , which illustrates an exemplary schematic diagram of a storage data management page. The storage data management page 40 includes configuration components 41 corresponding to multiple configuration attribute fields. Users can perform corresponding operations on the configuration components 41, such as entering values or selecting options, to configure storage management configuration information.

[0095] Optionally, the above-mentioned information configuration operation includes but is not limited to a threshold setting operation, an API (Application Programming Interface) interface setting operation, a data table type setting operation, and a key-value data pair setting operation.

[0096] In an exemplary embodiment, the above storage management configuration information can also be determined based on the management configuration prompt information displayed on the storage data management page. Figure 3 As shown, the above method further includes the following step 280.

[0097] Step 280: In response to a confirmation operation on the management configuration prompt information, determine to store the management configuration information.

[0098] In some application scenarios, the user can quickly set the storage management configuration information based on the reference data corresponding to each configuration attribute field in the management configuration prompt information, and directly determine the above management configuration prompt information as the storage management configuration information.

[0099] The storage management configuration information is described below through a specific example. See Table 1 below. Table 1 exemplarily shows a storage management configuration information table.

[0100] Table 1

[0101]

[0102]

[0103]

[0104] The data processing services corresponding to the above storage management configuration information table are as follows:

[0105] 1. rewriteDataFiles: This file merging service is used to merge small files in a data cluster whose file size is smaller than a preset file threshold. Because each write to the underlying storage system for big data, such as HDFS, is a single file and appends are not allowed, a large number of small files are generated when inserting multiple small batches or when there are many partitions. Therefore, this file merging service can be used to efficiently manage small files in the data lake, reducing the amount of small file data and improving data query time.

[0106] 2. expireSnapshots: This service is used to clean up data snapshots and metadata. These data snapshots are a kernel transaction mechanism that contains all file information generated during a specific period. This service supports time travel, allowing you to specify a snapshot to read historical data. When data is no longer needed, the service cleans up the snapshots and metadata.

[0107] 3. delete: Expired file cleanup service, supports table-level, column-level, and partition-level lifecycle management, deletes expired data, and supports recovery within a fixed period.

[0108] 4. removeOrphanFiles: This orphan file cleanup service is used to remove orphan files from the data cluster. Commit (information submission) is a kernel transaction mechanism that generates a snapshot during a commit. A successful commit indicates a successful operation, while a failed commit generates orphan files that occupy storage space. Therefore, this orphan file cleanup service effectively manages orphan files in the data lake, removes useless orphan files, and improves data storage efficiency.

[0109] 5. Clustering / Order By: This service optimizes data layout based on the cluster key or order by key. For example, it can rearrange the table based on the values of the cluster key or sort key. Another example is sorting the table based on the values of the sort key.

[0110] Step 230 : Send storage management configuration information so that the first device executes the data processing task for the target data cluster when the cluster operation information corresponding to the target data cluster meets the storage data management condition.

[0111] The above target data cluster is a data cluster based on the data lake, which is a big data storage layer table format that stores structured and unstructured data.

[0112] Storage data management conditions refer to the triggering conditions of data processing tasks.

[0113] In an exemplary embodiment, the storage management configuration information includes file merge configuration information, and the data processing task includes a file merge task. The file merge configuration information is used to determine file merge conditions, which are used to trigger a file merge task for a target data cluster. The storage data management conditions include the file merge conditions. The file merge task refers to a task that merges data files in the target data cluster whose file sizes are smaller than a preset threshold. The file merge configuration information includes configuration data corresponding to the configuration attribute fields corresponding to the file merge service in Table 1.

[0114] In an exemplary embodiment, the storage management configuration information includes file cleanup configuration information, and the data processing tasks include file cleanup tasks. The file cleanup configuration information is used to determine file cleanup conditions, which are used to trigger file cleanup tasks for a target data cluster. The storage data management conditions include file cleanup conditions. The file cleanup task refers to a task that cleans target data files in the target data cluster. The file cleanup configuration information includes configuration data corresponding to the configuration attribute fields corresponding to the data snapshot cleanup service, expired file cleanup service, and orphan file cleanup service in Table 1.

[0115] In an exemplary embodiment, the storage management configuration information includes data table configuration information, and the data processing task includes a data table adjustment task. The data table configuration information is used to determine data table adjustment conditions, which are used to trigger a data table adjustment task for a target data cluster. The storage data management conditions include the data table adjustment conditions. The data table adjustment task refers to a task that adjusts and optimizes the stored data in a data table within the target data cluster. The data table configuration information includes configuration data corresponding to the configuration attribute fields corresponding to the data layout optimization service in Table 1.

[0116] Step 240: Receive task execution data corresponding to the data processing task.

[0117] When the cluster operation information satisfies the file merging condition corresponding to the file merging configuration information, the first device executes the file merging task, obtains the file merging data, and sends the file merging data to the local device. Accordingly, the local device receives the file merging data corresponding to the file merging task.

[0118] When the cluster operation information satisfies the file cleaning condition corresponding to the file cleaning configuration information, the first device executes the file cleaning task, obtains the file cleaning data, and sends the file cleaning data to the local device. Accordingly, the local device receives the file cleaning data corresponding to the file cleaning task.

[0119] When the cluster operation information satisfies the data table adjustment condition corresponding to the data table configuration information, the first device executes the data table adjustment task, obtains data table adjustment data, and sends the data table adjustment data to the local device. Accordingly, the local device receives the data table adjustment data corresponding to the data table adjustment task.

[0120] Step 250 : Based on the task execution data, display the task execution information corresponding to the target data cluster in a preset page.

[0121] In an exemplary embodiment, as Figure 3 As shown, the implementation process of the above step 250 includes the following sub-steps (251-253).

[0122] Step 251 : Based on the file merging data corresponding to the file merging task, display the file merging information in the target data cluster on a preset page.

[0123] The file merging data is task execution data obtained by executing the file merging task when the cluster operation information satisfies the file merging conditions corresponding to the file merging configuration information. The file merging information is used to represent the file merging status in the target data cluster.

[0124] Optionally, the above-mentioned file merging information includes but is not limited to data information corresponding to a merged file quantity index and a cluster file reduction quantity index.

[0125] Optionally, the file merging information in the target data cluster is displayed in a graphical form on a preset page.

[0126] Step 252 : Based on the file cleaning data corresponding to the file cleaning task, display the file cleaning information in the target data cluster on a preset page.

[0127] The file cleaning data is the task execution data obtained by executing the file cleaning task when the cluster operation information meets the file cleaning conditions corresponding to the file cleaning configuration information. The file cleaning information is used to characterize the file cleaning status in the target data cluster.

[0128] Optionally, the above-mentioned file cleaning information includes but is not limited to data information corresponding to an expired file cleaning quantity indicator, a data snapshot file cleaning quantity indicator, and an orphan file cleaning quantity indicator.

[0129] Optionally, the file cleanup information in the target data cluster is displayed in a graphical form on a preset page.

[0130] Step 253 : Based on the data table adjustment data corresponding to the data table adjustment task, display the data table adjustment information in the target data cluster on a preset page.

[0131] The data table adjustment data is the task execution data obtained by executing the data table adjustment task when the cluster operation information meets the data table adjustment conditions corresponding to the data table configuration information. The data table adjustment information is used to represent the data table adjustment status in the target data cluster.

[0132] Optionally, the above-mentioned data table adjustment information includes but is not limited to data information corresponding to the data volume reduction index of the data table in the target data cluster and layout information of the data table after adjustment.

[0133] Optionally, the data table adjustment information in the target data cluster is displayed in a graphical form on a preset page.

[0134] To sum up, the technical solution provided by the embodiment of the present application determines the storage management configuration information corresponding to the target data cluster and sends the storage management configuration information by receiving the information configuration operation on the storage data management page, so that the first device can determine the storage data management conditions based on the received storage management configuration information and judge the cluster operation information based on the conditions, thereby automatically executing the corresponding data processing tasks and sending the task execution data to the local device. The local device can display the task execution information within the data cluster on a preset page based on the received task execution data, thereby realizing data operation and maintenance automation, reducing data operation and maintenance workload, improving data operation and maintenance efficiency and data storage efficiency within the data cluster, and also increasing the amount of information displayed in the data operation and maintenance results, thereby improving user experience.

[0135] Compared with the data operation and maintenance solutions in the prior art that do not have front-end interface settings and require self-written Java code settings, the technical solution provided by the embodiment of the present application only needs to configure the data operation and maintenance information on the page to realize automated data operation and maintenance, thereby improving data operation and maintenance efficiency; the prior art cannot display the status of events and other events in the table in the data cluster, and cannot guide users to set appropriate parameters; the technical solution provided by the embodiment of the present application can display task execution information on the preset interface, and can display management configuration prompt information on the page based on the cluster operation information, which greatly improves the visibility of data operation and maintenance information, improves user experience, and is conducive to timely discovery of alarm information. In addition, the technical solution provided by the embodiment of the present application supports a variety of automated data operation and maintenance services, greatly reducing the workload of data operation and maintenance, and a variety of application programming extension interfaces can be configured on the page, which improves the compatibility of data operation and maintenance services.

[0136] In addition, in specific application scenarios, several data operation and maintenance tasks can be performed at appropriate times based on the snapshot information of the data table, such as merging small files and creating indexes. When the number of data files exceeds the quantity threshold, the processing speed of the target data processing task is adjusted to the first speed; when the number of data files does not reach the quantity threshold, the processing speed of the target data processing task is adjusted to the second speed, wherein the first speed is greater than the second speed. In addition to the speed of the data processing task being adjustable, the task execution frequency and the allocated computing resources corresponding to the data processing task can be adjusted according to the situation of the system reading and writing data. For example, when there are many data files, the speed of executing small files is faster, and when there are fewer data files, the speed of merging small files is slower. The speed of merging also determines the speed of user queries, which can ensure the stability of user queries. It is also possible to determine the expiration time of the metafiles that need to be cleaned up based on the user's query pattern, and delete useless indexes in time, thereby reducing the user's storage cost. The embodiment of the present application can provide operation and maintenance services in a timely and effective manner based on the user's usage pattern.

[0137] Please refer to Figure 5 , which shows the process of the data processing method provided by an embodiment of the present application Figure 3 This method can be applied to computer equipment, which refers to electronic equipment with data calculation and processing capabilities. For example, the execution subject of each step can be Figure 1 The server 20 in the application running environment is shown. The method may include the following steps (510-550).

[0138] Step 510: Obtain cluster operation information corresponding to the target data cluster.

[0139] The cluster operation information includes data operation event information, which refers to information corresponding to data table read and write operation events in the target data cluster. The data table operation event information includes data table operation records of at least one data table.

[0140] In an exemplary embodiment, as Figure 6 As shown, after the above step 510, the following steps (560-570) are also included: Figure 6 The flow chart of the data processing method provided by one embodiment of the present application is shown as follows Figure 4 .

[0141] Step 560: Adaptively determine management configuration prompt information for the target data cluster based on the cluster operation information.

[0142] Cluster operation information includes data operation events (Events). Based on these Events, the device determines reference configuration data corresponding to the preset configuration attribute fields in the storage management configuration information (Table config) and generates management configuration prompts based on the reference configuration data. Users can configure the storage management configuration information for the data cluster based on the reference configuration data determined by the system.

[0143] In a possible implementation, the generation method of the management configuration prompt information includes but is not limited to the following four methods.

[0144] (1) Determine the reference value corresponding to the file periodic cleanup interval indicator based on the historical query operation time consumption. Provide a recommended reference configuration value for the file periodic cleanup interval indicator (expireSeconds) based on the longest time spent on the query operation (query) executed by the user.

[0145] (2) Generate index deletion reference information based on the index unused time. For example, display the index unused time on a preset page to provide users with reference information for deleting the index.

[0146] (3) Determine the file merging reference configuration information based on the average size of the query data files. If there are many small files in the target data cluster, it is necessary to configure the various configuration attribute fields in the file merging configuration information to speed up the file merging process.

[0147] (4) Generate a delete event reference number threshold (delete event number), and execute the orphan file cleaning task according to the delete event reference number threshold.

[0148] In actual application scenarios, the automated data operation and maintenance system can deploy a separate reference configuration information generation component (advisor).

[0149] Step 570: Send management configuration prompt information so that the storage data management page displays the management configuration prompt information.

[0150] The management configuration prompt information is sent to the second device, so that the storage data management page of the second device displays the management configuration prompt information.

[0151] Step 520: Receive storage management configuration information corresponding to the target data cluster.

[0152] The storage management configuration information is configuration information determined by the second device according to an information configuration operation received on the storage data management page.

[0153] Step 530: Determine the storage data management conditions corresponding to the target data cluster based on the storage management configuration information.

[0154] In an exemplary embodiment, the storage management configuration information includes file merging configuration information, file cleaning configuration information, and data table configuration information.

[0155] Accordingly, based on the file merging configuration information, the file merging conditions corresponding to the target data cluster are determined; based on the file cleaning configuration information, the file cleaning conditions corresponding to the target data cluster are determined; based on the data table configuration information, the data table adjustment conditions corresponding to the target data cluster are determined.

[0156] In a possible implementation, a file size threshold in the file merging configuration information, a preset file reception quantity threshold corresponding to the file merging service, and a quantity threshold of received event information corresponding to the file merging service are obtained.

[0157] Based on the file size threshold and the preset file receiving quantity threshold, a first file merging condition is determined. The first file merging condition includes that the number of received preset files (files with a size smaller than the file size threshold are preset files) in the cluster operation information is greater than or equal to the preset file receiving quantity threshold.

[0158] Based on the number threshold of received event information corresponding to the file merging service, a second file merging condition is determined, wherein the second file merging condition includes that the number of received event information in the cluster operation information reaches the number threshold of received event information corresponding to the file merging service.

[0159] In one possible implementation, the file periodic cleaning time threshold, the number threshold of received event information corresponding to the data snapshot cleaning service, the data retention period threshold, the number threshold of received event information corresponding to the orphan file cleaning service, and the number threshold of deletion events corresponding to the orphan file cleaning service are obtained from the file cleaning configuration information.

[0160] Based on the above-mentioned file regular cleanup time threshold, a first data snapshot cleanup condition is determined. The first data snapshot cleanup condition includes that the regular cleanup time interval in the cluster operation information reaches the regular cleanup time threshold.

[0161] A second data snapshot cleaning condition is determined based on a threshold value of received event information corresponding to the data snapshot cleaning service. The second data snapshot cleaning condition includes the number of received event information in the cluster operation information reaching the threshold value of received event information corresponding to the data snapshot cleaning service.

[0162] Based on the data retention period threshold, the expired file cleanup condition is determined. The expired file cleanup condition includes the creation time of each data file in the cluster operation information reaching the data retention period threshold.

[0163] Based on the threshold of the number of received event information corresponding to the orphan file cleaning service, a first orphan file cleaning condition is determined. The first orphan file cleaning condition includes that the number of received event information in the cluster operation information reaches the threshold of the number of received event information corresponding to the orphan file cleaning service.

[0164] Based on the deletion event number threshold corresponding to the orphan file cleaning service, a second orphan file cleaning condition is determined. The second orphan file cleaning condition includes the number of received deletion event information in the cluster operation information reaching the deletion event number threshold corresponding to the orphan file cleaning service.

[0165] In a possible implementation, key field information and index adjustment interval in data table configuration information are obtained.

[0166] Based on the key field information, a first data table adjustment condition is determined. The first data table adjustment condition includes determining whether there is a condition column corresponding to the key field information in the data table.

[0167] Based on the index adjustment interval, a second data table adjustment condition is determined, wherein the second data table adjustment condition includes that the index adjustment interval duration in the cluster operation information reaches an index adjustment interval threshold.

[0168] Step 540 : When the cluster operation information satisfies the storage data management condition, the data processing task for the target data cluster is executed to obtain task execution data corresponding to the data processing task.

[0169] The data lake provides the corresponding API basic capabilities. This service determines whether to execute the API based on user configuration and certain rules. The above data processing tasks can be implemented by calling the API.

[0170] For the above services, it also supports calling single tasks using SQL or OpenAPI, and supports calling multi-language SDKs such as Python and Go.

[0171] The above-mentioned storage data management condition includes any one of the conditions determined in the above-mentioned step 530 .

[0172] In an exemplary embodiment, as Figure 6 As shown, the implementation process of the above step 540 includes the following steps (541-544).

[0173] Step 541 : When the data operation event information satisfies the target storage data management condition, determine at least one data table corresponding to the target data processing task.

[0174] The target storage data management condition is any storage data management condition, and the target data processing task is a data processing task corresponding to the target storage data management condition.

[0175] In an exemplary embodiment, the stored data management conditions include at least one of a file merging condition, a file cleanup condition, and a data table adjustment condition. The file merging conditions include, but are not limited to, the first file merging condition and the second file merging condition. The file cleanup conditions include, but are not limited to, the first data snapshot cleanup condition, the second data snapshot cleanup condition, the expired file cleanup condition, the first orphan file cleanup condition, and the second orphan file cleanup condition. The data table adjustment conditions include, but are not limited to, the first data table adjustment condition and the second data table adjustment condition.

[0176] Correspondingly, the data processing task includes at least one of a file merging task corresponding to the file merging condition, a file cleaning task corresponding to the file cleaning condition, and a data table adjustment task corresponding to the data table adjustment condition.

[0177] The first file merge condition and the second file merge condition are trigger conditions for executing the file merge task. The first data snapshot cleanup condition and the second data snapshot cleanup condition are trigger conditions for executing the data snapshot cleanup task. The expired file cleanup condition is trigger conditions for executing the expired file cleanup task. The first orphan file cleanup condition and the second orphan file cleanup condition are trigger conditions for executing the orphan file cleanup task. The first data table adjustment condition and the second data table adjustment condition are trigger conditions for executing the data table adjustment task. The target storage data management condition is any of the above conditions.

[0178] Step 542: Create at least one data table operation task corresponding to the data table.

[0179] The target data processing task includes at least one data table operation task corresponding to a data table. The above data table operation task refers to a task for performing a preset operation on data, and the task content of the above data table operation task is associated with the task content of the target data processing task.

[0180] Step 543: Allocate the data table operation task corresponding to the at least one data table to at least one task queue to generate a data table operation task package.

[0181] In an exemplary embodiment, at least one data table operation task in a task queue that meets the task assembly criteria is combined to generate a data table operation task package. This data table operation task package can be executed as a data processing engine task. Because the data processing engine takes a long time to start up, submitting a data table operation task package allows the execution of multiple data table operation tasks corresponding to the same table to be started at once, thereby improving data processing efficiency.

[0182] Step 544: execute the data table operation task in the data table operation task package to obtain task execution data.

[0183] In an exemplary embodiment, as Figure 7 As shown, before the above step 544, the following steps (545-546) are also included: Figure 7 The flow chart of the data processing method provided by one embodiment of the present application is shown as follows Figure 5 .

[0184] Step 545: Obtain resource load information corresponding to the target data cluster and data table attribute information corresponding to at least one data table.

[0185] Resource load information includes the total resources and remaining available resources corresponding to the target data cluster. Optionally, resource load information also includes resource allocation information based on historical execution of the data table. The system can allocate computing resources to the current data table based on the resource allocation information based on historical execution of the data table.

[0186] Step 546: Determine the computing resources corresponding to the data table operation task based on the resource load information and the data table attribute information.

[0187] According to the above total resources, remaining available resources and data table attribute information, appropriate computing resources, including memory and processing core units, can be allocated to the data table operation task.

[0188] Accordingly, the above step 544 can be replaced by the following step 544a.

[0189] Step 544a: Based on the computing resources corresponding to the data table operation tasks in the data table operation task package, execute the data table operation tasks in the data table operation task package to obtain task execution data.

[0190] Based on the memory and the processing core unit corresponding to the data table operation task in the data table operation task package, the data table operation task in the data table operation task package is executed to obtain task execution data.

[0191] The task execution data is task execution data corresponding to the target data processing task.

[0192] Step 550 : Send the task execution data so that the second device displays the task execution information corresponding to the target data cluster on a preset page based on the task execution data.

[0193] For the introduction of task execution data, please refer to the corresponding introduction in the previous embodiment, which will not be repeated here.

[0194] The following describes the above data processing methods in conjunction with specific data operation and maintenance scenarios. Figure 8 , which exemplarily shows a technical architecture diagram of an automated data operation and maintenance kernel. The above automated data operation and maintenance kernel can be deployed in a computer device. Figure 8 The arrows in the middle illustrate the components and workflows in the automated data operations core.

[0195] 1. Data generated by users using the target service can be transmitted to the automated data operation and maintenance kernel through its interfaces. These interfaces include, but are not limited to, database kernel write interfaces, data compaction interfaces, and read interfaces.

[0196] 2. The data received by the automated data operation and maintenance kernel can be transmitted to the task indicator reporter (MetricsReporter). The above received data can be transmitted to the task indicator reporter after the creation operation, deletion operation, snapshot operation, and replacement operation.

[0197] 3. The task indicator reporter (Metrics Reporter) reports various indicator data of the data table (such as the number of files generated by this submission event, file size, total number of files in the table, total file size, and other indicator data) based on the commit information on the automated data operation and maintenance kernel side to the task indicator event manager (Metrics Event Handler) for corresponding processing, such as insert, delete, update, data table indicator processing, etc., and inputs it to the data table event manager (TableEvent Handler) to obtain data operation event (Event) information. The data table event manager can send event information to the message queue (Message Queue, MQ). For example, message queues such as Kafka. Kafka is a high-throughput distributed publish-subscribe messaging system that can process all action stream data generated by consumers on the website.

[0198] Two types of Event information are exemplarily shown below: data snapshot creation event (CreateSnapshotEvent) information and scan event (ScanEvent) information.

[0199] CreateSnapshotEvent includes: tableName: "table"; addedFiles: 400; deletedFiles: 500; operation: "APPEND"; data partition analysis information (partitionAnaplysis): {partitionId: "a=1 / b=2"; fileSize: 1000; fileNums: 100; mse: 690000}

[0200] ScanEvent: scanTime; expression: a = 1 and b = 2; average-file-size: 60.

[0201] If the operation attribute field (operation) is DELETE, it means deleting data; if it is APPEND, it means inserting data; if it is REPLACE, it means merging data files.

[0202] Next, please refer to Figure 9 , which exemplarily shows a technical architecture diagram of an automated data operation and maintenance service.

[0203] 4. The message queue sends the received event information to the external information receiving component in the automated data operation and maintenance service system. In addition, the external information receiving component can also receive the user's configuration information in the front-end page (i.e., the storage management configuration information mentioned above) and the historical execution log data of the task (Job) in the target data cluster.

[0204] 5. The external information receiving component submits the received message to the job event manager (Job EventHandler). The job event manager is composed of various components (not shown in the figure), including the job configuration information acquisition component (Job config), the task execution condition judgment component (Job rule), the task resource scheduler (Job resource), the task scheduler (Job scheduler), the task execution component (Job executor), the task submission component (Job submit), and the task reporting component (Job reporter). The functions of each component are as follows:

[0205] Job config: Receives user configuration information for the front-end page, stores it in a database table, and configures thresholds and parameters for executing APIs, such as expireSeconds (cleanup cycle) and how long ago snapshots (data snapshots) should be deleted.

[0206] Job rule: Based on the information obtained from the job config, it determines whether the cluster's running information meets any of the above task execution conditions. If so, it sends the table operation task for each data table corresponding to the task to Jobscheduler. If not, it records the cumulative running information after the condition is executed and stores it in the database. The next time it receives event information, it continues the cumulative calculation. If it meets the conditions, it clears the information; if not, it continues the accumulation.

[0207] Job resource: The job scheduler calls the task resource scheduler. The job resource allocates appropriate computing resources to table operation tasks based on total resources, currently available resources, table attributes, historical table resource consumption, and task priority. Enabling dynamic resource allocation for Spark tasks requires setting information such as the initial executor, executor cores, driver cores, and driver memory.

[0208] Job scheduler: Receives jobs from multiple tables, places them in the appropriate task queue, assembles them into a Spark task, and sends it to the job executor at the appropriate time.

[0209] Job executor: provides multiple ways to execute jobs. One is to send HTTP requests to the Spark task platform (single tasks and periodic tasks), one is to call the spark-submit command (single tasks), and one is to start a scheduled task in the service and call the spark-submit command.

[0210] Job submit: The Spark JAR package supports multi-table execution by passing parameters. Each task contains several tables and is submitted asynchronously.

[0211] Among them, periodic tasks involve using the job editor to modify table information, the input parameters of the Spark execution jar, and modify resource configuration. One-time tasks only require submission.

[0212] Job reporter: Spark-submitted tasks require task execution details, including table information, task start time, duration, and task results. These details are then processed and recorded via HTTP requests or recorded in a database using Java Database Connectivity (JDBC). These details are then received by the Job Receiver and graphically displayed, allowing users to identify execution trends. Failure alerts are generated and sent to users or operations staff, and can be viewed on the display interface. The Spark task execution code is "Actions.rewriteDataFiles().execute();."

[0213] 6. After executing the tasks in the task queue, the generated task (Job) execution information is sent to the message queue

[0214] 7. The message queue sends task execution information to the display platform (Show), which displays event statistics and task execution information. The display platform processes the event information and displays it in a specific dimension on components such as Grafana and Telegraf (a data collection tool), displaying current table information and table historical status. Grafana is an open-source data visualization tool developed in the Go language that can perform data detection and statistics, and has an alerting function.

[0215] 8. The task indicator system can store task execution information, resource information and other information persistently in the database based on the task execution status in the task queue for subsequent analysis.

[0216] These automated data operations services have no impact on existing data lake usage and support concurrent operations with select, insert, update, and delete. By using these automated data operations services, you can reduce small files in your data cluster system, optimize data layout, and accelerate queries. They also reduce orphan files, data snapshots, and expired data files, increasing available storage space.

[0217] To sum up, the technical solution provided by the embodiment of the present application determines the storage data management conditions corresponding to the target data cluster by receiving the storage management configuration information determined by the information configuration operation on the storage data management page, and judges the cluster operation information based on the conditions, thereby automatically executing the corresponding data processing tasks and sending the task execution data to the second device, so that the second device displays the task execution information within the data cluster on a preset page based on the received task execution data, realizing data operation and maintenance automation, reducing data operation and maintenance workload, improving data operation and maintenance efficiency and data storage efficiency within the data cluster, and also increasing the amount of information displayed in the data operation and maintenance results, thereby improving user experience.

[0218] Please refer to Figure 10 , which shows the interactive process of the data processing method provided by an embodiment of the present application Figure 1 .

[0219] Step 1001: The second device displays a storage data management page.

[0220] Step 1002: The second device determines storage management configuration information corresponding to the target data cluster in response to an information configuration operation received on the storage data management page.

[0221] Step 1003: The second device sends storage management configuration information to the first device.

[0222] Correspondingly, the first device receives the storage management configuration information corresponding to the target data cluster.

[0223] Step 1004: The first device obtains cluster operation information corresponding to the target data cluster.

[0224] In an exemplary embodiment, as Figure 11 As shown, the above step 1004 also includes the following steps (1009-1010): Figure 11 The interactive process of the data processing method provided by one embodiment of the present application is shown Figure 2 .

[0225] Step 1009: The first device adaptively determines management configuration prompt information for the target data cluster based on the cluster operation information.

[0226] Step 1010: The first device sends management configuration prompt information to the second device, so that the storage data management page displays the management configuration prompt information.

[0227] Correspondingly, the second device receives the management configuration prompt information. Figure 11 As shown, after the above step 1001, the first device side also includes the following steps (1011-1012).

[0228] Step 1011: The second device displays management configuration prompt information on the storage data management page.

[0229] Step 1012: The second device determines to store the management configuration information in response to a confirmation operation on the management configuration prompt information.

[0230] Step 1005: The first device determines the storage data management conditions corresponding to the target data cluster based on the storage management configuration information.

[0231] Step 1006 : When the cluster operation information satisfies the storage data management condition, the first device executes the data processing task for the target data cluster and obtains task execution data corresponding to the data processing task.

[0232] In an exemplary embodiment, as Figure 11 As shown, the implementation process of the above step 1006 includes the following sub-steps (1006a~1006d).

[0233] Step 1006a: When the data operation event information satisfies the target storage data management condition, the first device determines at least one data table corresponding to the target data processing task.

[0234] Step 1006b: The first device creates a data table operation task corresponding to at least one data table.

[0235] Step 1006c: The first device distributes the data table operation task corresponding to the at least one data table to at least one task queue, and generates a data table operation task package.

[0236] Step 1006d: The first device executes the data table operation task in the data table operation task package to obtain task execution data.

[0237] Step 1007 : The first device sends the task execution data to the second device, so that the second device displays the task execution information corresponding to the target data cluster in a preset page based on the task execution data.

[0238] Correspondingly, the second device receives task execution data corresponding to the data processing task.

[0239] Step 1008 : The second device displays the task execution information corresponding to the target data cluster on a preset page based on the task execution data.

[0240] The above steps have been described above. This embodiment describes the interaction process between the first device and the second device, and the introduction of each step will not be repeated here.

[0241] The following is an overall introduction to the above data processing interaction process in combination with data operation and maintenance scenarios. Figure 12 , which exemplifies an overall flow chart of an automated data operation and maintenance service.

[0242] 1. Users can configure whether to enable this service in the front-end interface. Optionally, they can configure whether to report data in the data organization format (Iceberg tale property). Reporting is enabled by default. Iceberg is an open table format for large analytical data sets.

[0243] 2. Based on the reported address configuration results, the kernel write information is reported to the Message Queue (MQ), the reference configuration information generation component (advisor), and the display platform. The advisor generates management configuration prompts based on the reported information and sends them to the display platform for display on the front-end page. This kernel write information includes, but is not limited to, event information corresponding to operations such as commit and create.

[0244] 3. The above-mentioned reporting information, storage management configuration information (table config), historical task execution time and other data can be simultaneously transmitted to the task information receiving component (Job receiver).

[0245] 4. If the information received by the task information receiving component meets the task conditions, namely the aforementioned storage data management conditions, a data table operation task (Task) corresponding to the individual data tables is generated according to the established rules. Each task for each data table is then submitted to the job scheduler. Because Apache Spark takes a long time to start up, the task scheduler can assemble multiple data tables into task packages, generating unified Spark tasks, which are then submitted to the data computing engine. Apache Spark is a fast and versatile computing engine designed specifically for large-scale data processing.

[0246] 5. The job resource scheduler allocates appropriate computing resources to spark tasks, such as memory and processing cores.

[0247] 6. The task execution component (Job executor) calls the data calculation engine to execute the task.

[0248] 7. The job reporter component reports the task execution information of each data table, persists it to the database on the service side, and sends it to the display platform for display on the front-end configuration page.

[0249] With this automated data operations service, users no longer need to submit multiple Spark or Flink tasks to complete data operations. Instead, they simply configure service parameters on the front-end interface, and the back-end service automatically executes. Furthermore, given that the resources specified in user-submitted tasks are often suboptimal, the task resource scheduler dynamically allocates the most appropriate computing resources for each task based on table status, reducing resource waste. It also rationally schedules tasks based on cluster resources to prevent task failures.

[0250] For SQL queries, average query speed can be increased by 20%, and some single-point queries can be increased by 60%. The number of data files and metadata files is reduced, which can reduce the pressure on the cluster master node (namenode).

[0251] To sum up, the technical solution provided by the embodiment of the present application can determine the storage management configuration information corresponding to the target data cluster through the information configuration operation received on the storage data management page of the second device; the second device sends the storage management configuration information to the first device, so that the first device can determine the storage data management conditions based on the received storage management configuration information and judge the cluster operation information based on the conditions, thereby automatically executing the corresponding data processing tasks and sending the task execution data to the second device. The second device can display the task execution information within the data cluster on a preset page based on the received task execution data, thereby realizing data operation and maintenance automation, reducing data operation and maintenance workload, improving data operation and maintenance efficiency and data storage efficiency within the data cluster, and also increasing the amount of information displayed in the data operation and maintenance results, thereby improving user experience.

[0252] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0253] Please refer to Figure 13 , which shows a block diagram of a data processing device provided by an embodiment of the present application Figure 1 . The device has the function of implementing the above-mentioned data processing method, and the function can be implemented by hardware or by hardware executing corresponding software. The device can be a computer device or can be set in a computer device. The device 1300 can include: a management page display module 1310, a configuration information determination module 1320, a configuration information sending module 1330, a task data receiving module 1340, and a task information display module 1350.

[0254] A management page display module 1310 is used to display a storage data management page;

[0255] A configuration information determination module 1320 is configured to determine storage management configuration information corresponding to a target data cluster in response to an information configuration operation received on the storage data management page, wherein the storage management configuration information is used to determine storage data management conditions corresponding to the target data cluster;

[0256] a configuration information sending module 1330 configured to send the storage management configuration information so that the first device executes the data processing task for the target data cluster if the cluster operation information corresponding to the target data cluster satisfies the storage data management condition, where the storage data management condition refers to a triggering condition for the data processing task;

[0257] The task data receiving module 1340 is used to receive the task execution data corresponding to the data processing task;

[0258] The task information display module 1350 is configured to display the task execution information corresponding to the target data cluster in a preset page based on the task execution data.

[0259] In an exemplary embodiment, the apparatus 1300 further includes: a prompt information receiving module.

[0260] The prompt information receiving module is configured to receive management configuration prompt information corresponding to the target data cluster, wherein the management configuration prompt information is management configuration information adaptively determined based on the cluster operation information.

[0261] The management page display module 1310 is further configured to display the management configuration prompt information on the storage data management page.

[0262] The configuration information determining module 1320 is further configured to determine the storage management configuration information in response to a confirmation operation on the management configuration prompt information.

[0263] In an exemplary embodiment, the storage management configuration information includes file merging configuration information, the data processing task includes a file merging task, and the task information display module 1350 includes: a file merging information display unit.

[0264] The file merging information display unit is configured to display the file merging information in the target data cluster on the preset page based on the file merging data corresponding to the file merging task.

[0265] The file merging data is task execution data obtained by executing the file merging task when the cluster operation information satisfies the file merging condition corresponding to the file merging configuration information. The file merging information is used to characterize the file merging situation in the target data cluster.

[0266] In an exemplary embodiment, the storage management configuration information includes file cleaning configuration information, the data processing task includes a file cleaning task, and the task information display module 1350 includes: a file cleaning information display unit.

[0267] A file cleaning information display unit, configured to display file cleaning information in the target data cluster on the preset page based on the file cleaning data corresponding to the file cleaning task;

[0268] Among them, the file cleaning data is the task execution data obtained by executing the file cleaning task when the cluster operation information meets the file cleaning conditions corresponding to the file cleaning configuration information. The file cleaning information is used to characterize the file cleaning status in the target data cluster.

[0269] In an exemplary embodiment, the storage management configuration information includes data table configuration information, the data processing task includes a data table adjustment task, and the task information display module 1350 includes: a data table adjustment information display unit.

[0270] The data table adjustment information display unit is configured to display the data table adjustment information in the target data cluster on the preset page based on the data table adjustment data corresponding to the data table adjustment task.

[0271] The data table adjustment data is task execution data obtained by executing the data table adjustment task when the cluster operation information meets the data table adjustment conditions corresponding to the data table configuration information. The data table adjustment information is used to characterize the data table adjustment status in the target data cluster.

[0272] To sum up, the technical solution provided by the embodiment of the present application determines the storage management configuration information corresponding to the target data cluster and sends the storage management configuration information by receiving the information configuration operation on the storage data management page, so that the first device can determine the storage data management conditions based on the received storage management configuration information and judge the cluster operation information based on the conditions, thereby automatically executing the corresponding data processing tasks and sending the task execution data to the local device. The local device can display the task execution information within the data cluster on a preset page based on the received task execution data, thereby realizing data operation and maintenance automation, reducing data operation and maintenance workload, improving data operation and maintenance efficiency and data storage efficiency within the data cluster, and also increasing the amount of information displayed in the data operation and maintenance results, thereby improving user experience.

[0273] Please refer to Figure 14 , which shows a block diagram of a data processing device provided by an embodiment of the present application Figure 2. The device has the function of implementing the above-mentioned data processing method, and the function can be implemented by hardware or by hardware executing corresponding software. The device can be a computer device or can be set in a computer device. The device 1400 can include: an operation information acquisition module 1410, a configuration information receiving module 1420, a management condition determination module 1430, a data task execution module 1440, and a task data sending module 1450.

[0274] The operation information acquisition module 1410 is used to obtain cluster operation information corresponding to the target data cluster.

[0275] The configuration information receiving module 1420 is configured to receive storage management configuration information corresponding to the target data cluster, where the storage management configuration information is configuration information determined by the second device according to the information configuration operation received on the storage data management page.

[0276] The management condition determination module 1430 is configured to determine the storage data management condition corresponding to the target data cluster based on the storage management configuration information.

[0277] The data task execution module 1440 is configured to execute the data processing task for the target data cluster and obtain task execution data corresponding to the data processing task when the cluster operation information satisfies the storage data management condition.

[0278] The task data sending module 1450 is configured to send the task execution data so that the second device displays the task execution information corresponding to the target data cluster on a preset page based on the task execution data.

[0279] In an exemplary embodiment, the apparatus 1400 further includes: a prompt information determining module and a prompt information sending module.

[0280] The prompt information determination module is used to adaptively determine management configuration prompt information for the target data cluster based on the cluster operation information.

[0281] The prompt information sending module is used to send the management configuration prompt information so that the storage data management page displays the management configuration prompt information.

[0282] In an exemplary embodiment, the cluster operation information includes data operation event information, and the data operation event information refers to the information corresponding to the data table read and write operation events in the target data cluster; the data task execution module 1440 includes: a data table determination unit, a task creation unit, a task package generation unit, and a task package execution unit.

[0283] A data table determination unit is configured to determine at least one data table corresponding to a target data processing task if the data operation event information satisfies a target storage data management condition. The target storage data management condition is any storage data management condition, and the target data processing task is a data processing task corresponding to the target storage data management condition.

[0284] A task creation unit, configured to create a data table operation task corresponding to the at least one data table, wherein the target data processing task includes the data table operation task corresponding to the at least one data table;

[0285] A task package generating unit, configured to allocate the data table operation task corresponding to the at least one data table to at least one task queue, and generate a data table operation task package;

[0286] The task package execution unit is used to execute the data table operation task in the data table operation task package to obtain the task execution data.

[0287] In an exemplary embodiment, the storage data management condition includes at least one of a file merging condition, a file cleaning condition, and a data table adjustment condition, and the data processing task includes at least one of a file merging task corresponding to the file merging condition, a file cleaning task corresponding to the file cleaning condition, and a data table adjustment task corresponding to the data table adjustment condition.

[0288] In an exemplary embodiment, the data task execution module 1440 further includes: an information acquisition unit and a computing resource determination unit.

[0289] The information acquisition unit is configured to acquire resource load information corresponding to the target data cluster and data table attribute information corresponding to the at least one data table.

[0290] A computing resource determining unit is configured to determine the computing resource corresponding to the data table operation task based on the resource load information and the data table attribute information.

[0291] The task package execution unit is specifically configured to execute the data table operation tasks in the data table operation task package based on the computing resources corresponding to the data table operation tasks in the data table operation task package to obtain the task execution data.

[0292] To sum up, the technical solution provided by the embodiment of the present application determines the storage data management conditions corresponding to the target data cluster by receiving the storage management configuration information determined by the information configuration operation on the storage data management page, and judges the cluster operation information based on the conditions, thereby automatically executing the corresponding data processing tasks and sending the task execution data to the second device, so that the second device displays the task execution information within the data cluster on a preset page based on the received task execution data, realizing data operation and maintenance automation, reducing data operation and maintenance workload, improving data operation and maintenance efficiency and data storage efficiency within the data cluster, and also increasing the amount of information displayed in the data operation and maintenance results, thereby improving user experience.

[0293] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0294] Please refer to Figure 15 , which shows the structural frame of a computer device provided by an embodiment of the present application Figure 1 The computer device may be a second device. The computer device is used to implement the data processing method on the second device side provided in the above embodiment. Specifically:

[0295] Typically, the computer device 1500 includes a processor 1501 and a memory 1502 .

[0296] The processor 1501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1501 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1501 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1501 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0297] The memory 1502 may include one or more computer-readable storage media, which may be non-transitory. The memory 1502 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1502 is used to store at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, at least one program, code set, or instruction set is configured to be executed by one or more processors to implement the above-mentioned data processing method.

[0298] In some embodiments, computer device 1500 may optionally include a peripheral device interface 1503 and at least one peripheral device. Processor 1501, memory 1502, and peripheral device interface 1503 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 1503 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 1504, a touchscreen display 1505, a camera assembly 1506, an audio circuit 1507, a positioning assembly 1508, and a power supply 1509.

[0299] Those skilled in the art will understand that Figure 15The structure shown in the figure does not constitute a limitation on the computer device 1500, and the computer device 1500 may include more or fewer components than shown in the figure, or combine some components, or adopt a different component arrangement.

[0300] Please refer to Figure 16 , which shows the structural frame of a computer device provided by an embodiment of the present application Figure 2 The computer device may be a first device, such as a server, for executing the data processing method on the first device side. Specifically:

[0301] Computer device 1600 includes a central processing unit (CPU) 1601, a system memory 1604 including a random access memory (RAM) 1602 and a read-only memory (ROM) 1603, and a system bus 1605 connecting system memory 1604 and CPU 1601. Computer device 1600 also includes a basic input / output system (I / O system) 1606 that facilitates information transfer between various components within the computer, and a mass storage device 1607 for storing an operating system 1613, application programs 1614, and other program modules 1615.

[0302] The basic input / output system 1606 includes a display 1608 for displaying information and an input device 1609, such as a mouse and keyboard, for user input. Both the display 1608 and the input device 1609 are connected to the central processing unit 1601 via an input / output controller 1610 connected to the system bus 1605. The basic input / output system 1606 may also include an input / output controller 1610 for receiving and processing input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1610 also provides output to a display screen, printer, or other types of output devices.

[0303] The mass storage device 1607 is connected to the central processing unit 1601 via a mass storage controller (not shown) connected to the system bus 1605. The mass storage device 1607 and its associated computer-readable media provide non-volatile storage for the computer device 1600. In other words, the mass storage device 1607 may include computer-readable media (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0304] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above-mentioned ones. The above-mentioned system memory 1604 and mass storage device 1607 can be collectively referred to as memory.

[0305] According to various embodiments of the present application, the computer device 1600 may also be connected to a remote computer on a network such as the Internet for operation. That is, the computer device 1600 may be connected to a network 1612 via a network interface unit 1611 connected to the system bus 1605, or the network interface unit 1611 may be used to connect to other types of networks or remote computer systems (not shown).

[0306] The memory also includes a computer program, which is stored in the memory and configured to be executed by one or more processors to implement the above-mentioned data processing method on the first device side.

[0307] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. When the at least one instruction, the at least one program, the code set or the instruction set is executed by a processor, the above-mentioned data processing method is implemented.

[0308] Optionally, the computer-readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or an optical disk, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0309] In an exemplary embodiment, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described data processing method.

[0310] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application do not limit this.

[0311] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

[0312] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

Claims

1. A data processing method, characterized in that: The method comprises: Display the storage data management page; In response to an information configuration operation received on the storage data management page, determining storage management configuration information corresponding to a target data cluster, wherein the storage management configuration information is used to determine at least one storage data management condition corresponding to the target data cluster; Sending the storage management configuration information so that the first device executes a target data processing task for the target data cluster if the cluster operation information corresponding to the target data cluster satisfies a target storage data management condition, wherein the at least one storage data management condition refers to a trigger condition of at least one data processing task, the target storage data management condition is any storage data management condition, and the target data processing task is a data processing task corresponding to the target storage data management condition; Receiving task execution data corresponding to the target data processing task; Based on the task execution data, task execution information corresponding to the target data cluster is displayed on a preset page.

2. The method according to claim 1, characterized in that The method further comprises: Receive management configuration prompt information corresponding to the target data cluster; wherein the management configuration prompt information is management configuration information adaptively determined based on the cluster operation information; Displaying the management configuration prompt information on the storage data management page; In response to a confirmation operation on the management configuration prompt information, the storage management configuration information is determined.

3. The method according to claim 1, characterized in that The storage management configuration information includes file merging configuration information, the data processing task includes a file merging task, and the displaying of the task execution information corresponding to the target data cluster on a preset page based on the task execution data includes: Based on the file merging data corresponding to the file merging task, displaying the file merging information in the target data cluster on the preset page; The file merging data is task execution data obtained by executing the file merging task when the cluster operation information satisfies the file merging condition corresponding to the file merging configuration information. The file merging information is used to characterize the file merging situation in the target data cluster.

4. The method according to claim 1, wherein The storage management configuration information includes file cleaning configuration information, the data processing task includes a file cleaning task, and the displaying of the task execution information corresponding to the target data cluster on a preset page based on the task execution data includes: Based on the file cleaning data corresponding to the file cleaning task, displaying the file cleaning information in the target data cluster on the preset page; Among them, the file cleaning data is the task execution data obtained by executing the file cleaning task when the cluster operation information meets the file cleaning conditions corresponding to the file cleaning configuration information. The file cleaning information is used to characterize the file cleaning status in the target data cluster.

5. The method according to claim 1, wherein The storage management configuration information includes data table configuration information, the data processing task includes a data table adjustment task, and the displaying of the task execution information corresponding to the target data cluster on a preset page based on the task execution data includes: Based on the data table adjustment data corresponding to the data table adjustment task, displaying the data table adjustment information in the target data cluster on the preset page; The data table adjustment data is task execution data obtained by executing the data table adjustment task when the cluster operation information meets the data table adjustment conditions corresponding to the data table configuration information. The data table adjustment information is used to characterize the data table adjustment status in the target data cluster.

6. A data processing method, characterized in that: The method comprises: Get the cluster operation information corresponding to the target data cluster; Receive storage management configuration information corresponding to the target data cluster, where the storage management configuration information is configuration information determined by the second device according to an information configuration operation received on a storage data management page; Determining at least one storage data management condition corresponding to the target data cluster based on the storage management configuration information; When the cluster operation information satisfies the target storage data management condition, executing the target data processing task for the target data cluster and obtaining task execution data corresponding to the target data processing task; the at least one storage data management condition refers to a trigger condition of at least one data processing task, the target storage data management condition is any storage data management condition, and the target data processing task is a data processing task corresponding to the target storage data management condition; The task execution data is sent so that the second device displays task execution information corresponding to the target data cluster in a preset page based on the task execution data.

7. The method according to claim 6, characterized in that After obtaining the cluster operation information corresponding to the target data cluster, the method further includes: Adaptively determining management configuration prompt information for the target data cluster based on the cluster operation information; The management configuration prompt information is sent so that the storage data management page displays the management configuration prompt information.

8. The method according to claim 6, characterized in that The cluster operation information includes data operation event information, and the data operation event information refers to information corresponding to data table read and write operation events in the target data cluster; When the cluster operation information satisfies the target storage data management condition, executing the target data processing task for the target data cluster and obtaining task execution data corresponding to the target data processing task include: In a case where the data operation event information satisfies a target storage data management condition, determining at least one data table corresponding to the target data processing task; Creating a data table operation task corresponding to the at least one data table, wherein the target data processing task includes the data table operation task corresponding to the at least one data table; Allocating a data table operation task corresponding to the at least one data table to at least one task queue to generate a data table operation task package; The data table operation task in the data table operation task package is executed to obtain the task execution data.

9. The method according to claim 8, characterized in that The storage data management conditions include at least one of a file merging condition, a file cleaning condition, and a data table adjustment condition; the data processing tasks include at least one of a file merging task corresponding to the file merging condition, a file cleaning task corresponding to the file cleaning condition, and a data table adjustment task corresponding to the data table adjustment condition.

10. The method according to claim 8, characterized in that Before executing the data table operation task in the data table operation task package and obtaining the task execution data, the method further includes: Acquire resource load information corresponding to the target data cluster and data table attribute information corresponding to the at least one data table; Determining computing resources corresponding to the data table operation task based on the resource load information and the data table attribute information; The executing of the data table operation task in the data table operation task package includes: The data table operation tasks in the data table operation task package are executed based on the computing resources corresponding to the data table operation tasks in the data table operation task package to obtain the task execution data.

11. A data processing device, characterized in that: The device comprises: Management page display module, used to display the storage data management page; a configuration information determining module, configured to determine storage management configuration information corresponding to a target data cluster in response to an information configuration operation received on the storage data management page, wherein the storage management configuration information is used to determine at least one storage data management condition corresponding to the target data cluster; a configuration information sending module, configured to send the storage management configuration information so that the first device executes a target data processing task for the target data cluster when the cluster operation information corresponding to the target data cluster satisfies a target storage data management condition, wherein the at least one storage data management condition refers to a trigger condition of at least one data processing task, the target storage data management condition is any storage data management condition, and the target data processing task is a data processing task corresponding to the target storage data management condition; A task data receiving module, configured to receive task execution data corresponding to the target data processing task; The task information display module is used to display the task execution information corresponding to the target data cluster in a preset page based on the task execution data.

12. The device according to claim 11, characterized in that The device also includes a prompt information receiving module; A prompt information receiving module, configured to receive management configuration prompt information corresponding to the target data cluster; wherein the management configuration prompt information is management configuration information adaptively determined based on the cluster operation information; The management page display module is further configured to display the management configuration prompt information on the storage data management page; The configuration information determination module is further configured to determine the storage management configuration information in response to a confirmation operation on the management configuration prompt information.

13. The device according to claim 11, characterized in that The storage management configuration information includes file merging configuration information, the data processing task includes a file merging task, and the task information display module includes: a file merging information display unit; A file merging information display unit, configured to display the file merging information in the target data cluster on the preset page based on the file merging data corresponding to the file merging task; The file merging data is task execution data obtained by executing the file merging task when the cluster operation information satisfies the file merging condition corresponding to the file merging configuration information. The file merging information is used to characterize the file merging situation in the target data cluster.

14. The device according to claim 11, characterized in that The storage management configuration information includes file cleaning configuration information, the data processing task includes a file cleaning task, and the task information display module includes: a file cleaning information display unit; A file cleaning information display unit, configured to display file cleaning information in the target data cluster on the preset page based on the file cleaning data corresponding to the file cleaning task; Among them, the file cleaning data is the task execution data obtained by executing the file cleaning task when the cluster operation information meets the file cleaning conditions corresponding to the file cleaning configuration information. The file cleaning information is used to characterize the file cleaning status in the target data cluster.

15. The device according to claim 11, characterized in that The storage management configuration information includes data table configuration information, the data processing task includes a data table adjustment task, and the task information display module includes: a data table adjustment information display unit; a data table adjustment information display unit, configured to display the data table adjustment information in the target data cluster on the preset page based on the data table adjustment data corresponding to the data table adjustment task; The data table adjustment data is task execution data obtained by executing the data table adjustment task when the cluster operation information meets the data table adjustment conditions corresponding to the data table configuration information. The data table adjustment information is used to characterize the data table adjustment status in the target data cluster.

16. A data processing device, characterized in that: The device comprises: The operation information acquisition module is used to obtain the cluster operation information corresponding to the target data cluster; a configuration information receiving module, configured to receive storage management configuration information corresponding to a target data cluster, wherein the storage management configuration information is configuration information determined by the second device according to an information configuration operation received on a storage data management page; a management condition determination module, configured to determine at least one storage data management condition corresponding to the target data cluster based on the storage management configuration information; a data task execution module, configured to execute a target data processing task for the target data cluster and obtain task execution data corresponding to the target data processing task if the cluster operation information satisfies a target storage data management condition; the at least one storage data management condition refers to a trigger condition of at least one data processing task, the target storage data management condition is any storage data management condition, and the target data processing task is a data processing task corresponding to the target storage data management condition; The task data sending module is used to send the task execution data so that the second device displays the task execution information corresponding to the target data cluster in a preset page based on the task execution data.

17. The device according to claim 16, characterized in that The device further comprises: a prompt information determining module and a prompt information sending module; a prompt information determination module, configured to adaptively determine management configuration prompt information for the target data cluster based on the cluster operation information; The prompt information sending module is used to send the management configuration prompt information so that the storage data management page displays the management configuration prompt information.

18. The device according to claim 16, characterized in that The cluster operation information includes data operation event information, and the data operation event information refers to information corresponding to data table read and write operation events in the target data cluster; The data task execution module includes: a data table determination unit, a task creation unit, a task package generation unit, and a task package execution unit; a data table determining unit, configured to determine at least one data table corresponding to a target data processing task if the data operation event information satisfies a target storage data management condition; A task creation unit, configured to create a data table operation task corresponding to the at least one data table, wherein the target data processing task includes the data table operation task corresponding to the at least one data table; A task package generating unit, configured to allocate the data table operation task corresponding to the at least one data table to at least one task queue, and generate a data table operation task package; The task package execution unit is used to execute the data table operation task in the data table operation task package to obtain the task execution data.

19. The device according to claim 18, characterized in that The storage data management conditions include at least one of a file merging condition, a file cleaning condition, and a data table adjustment condition; the data processing tasks include at least one of a file merging task corresponding to the file merging condition, a file cleaning task corresponding to the file cleaning condition, and a data table adjustment task corresponding to the data table adjustment condition.

20. The device according to claim 18, characterized in that The data task execution module further includes: an information acquisition unit and a computing resource determination unit; an information acquisition unit, configured to acquire resource load information corresponding to the target data cluster and data table attribute information corresponding to the at least one data table; a computing resource determining unit, configured to determine the computing resource corresponding to the data table operation task based on the resource load information and the data table attribute information; The task package execution unit is specifically configured to execute the data table operation tasks in the data table operation task package based on the computing resources corresponding to the data table operation tasks in the data table operation task package to obtain the task execution data.

21. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the data processing method according to any one of claims 1 to 5, or to implement the data processing method according to any one of claims 6 to 10.

22. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the data processing method according to any one of claims 1 to 5, or to implement the data processing method according to any one of claims 6 to 10.

23. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes to implement the data processing method according to any one of claims 1 to 5, or implements the data processing method according to any one of claims 6 to 10.

Images