Data management method, device and computing equipment

Abstract

The present application provides a data management method, apparatus, and computing device, the method comprising: the computing device obtaining task information and table information corresponding to a target data table, wherein the task information includes the processor occupancy time for generating the target data table; the table information includes the storage space occupied by the target data table and the number of dependent tasks, the number of dependent tasks indicating the number of tables using the target data table; the computing device determines the health of the target data table based on the task information and table information corresponding to the target data table, and determines whether to optimize the target data table based on the health of the target data table. By obtaining the log data of the file system and the metadata of each data table, and then determining the above-mentioned task information and table information of each data table, and then determining the health of each table based on the task information and table information of each data table, and then determining whether to optimize the data table, the difficulty and cost of data management in the file system can be reduced, and the efficiency of data management can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data management method, apparatus, and computing device. Background Art

[0002] As enterprises continue to expand, they need to collect, store, and manage more and more data. Because data requires continuous maintenance and governance, traditional data governance requires high-level engineers. Both the direction and effectiveness of governance rely heavily on the engineers' skills and experience. This makes it difficult for most enterprises to effectively manage data and effectively utilize their data. Therefore, how to implement data governance is a technical issue that needs to be addressed urgently. Summary of the Invention

[0003] The present application discloses a data management method, apparatus, and computing device. By using the log data of the file system and the metadata of each data table, the health of each table is determined and then whether the data table should be optimized. This method can reduce the difficulty and cost of data management in the file system and improve the efficiency of data management.

[0004] In a first aspect, the present application provides a data management method, comprising:

[0005] Acquire task information corresponding to the target data table, the task information including the processor occupancy time when generating the target data table;

[0006] Acquire table information of the target data table, the table information including storage space occupied by the target data table and the number of dependent tasks, where the number of dependent tasks is the number of other data tables generated based on the target data table;

[0007] Determine the health of the target data table according to the task information corresponding to the target data table and the table information;

[0008] The target data table is optimized based on the health of the target data table.

[0009] In one possible implementation, obtaining task information corresponding to the target data table includes: obtaining log data of the file system, and obtaining the time occupied by each subtask of multiple subtasks used to generate the target data table from the log data according to the identifier of the target data table, wherein the target data table is obtained after executing one or more tasks, and each task includes one or more subtasks; determining the processor occupancy time of the target data table based on the time occupied by each subtask.

[0010] In one possible implementation, the task information also includes a time span corresponding to the target data table, and the time span indicates a time span of the source data used when generating the target data table; obtaining the task information corresponding to the target data table also includes: obtaining from the log data the table identifier of the data table read by each of the multiple subtasks when generating the target data table, to obtain multiple table identifiers; obtaining the time span of the data recorded in the data table corresponding to each table identifier according to the multiple table identifiers, to obtain multiple time spans, wherein each table corresponds to a time span; and determining the time span corresponding to the target data table according to the multiple time spans.

[0011] In one possible implementation, the task information also includes the amount of data read and written to the target data table. The amount of data read refers to the amount of data read from the source data table by one or more subtasks when generating the target data table; the amount of data written refers to the amount of data written to the target data table by one or more subtasks when generating the target data table; the task information corresponding to the target data table is obtained by obtaining the amount of data read from the source data table and the amount of data written to the target data table when generating the target data table from the log data corresponding to the target data table; the source data table refers to other data tables read by one or more subtasks executed when generating the target data table; the amount of data read and the amount of data written can represent the number of input / output requests when generating the target data table, that is, the consumption of communication resources.

[0012] In one possible implementation, obtaining the table information of the target data table includes: obtaining metadata of the target data table, and obtaining the storage space occupied by the target data table from the metadata; obtaining one or more subtasks for reading the target data table from the metadata, determining the number of data tables generated after executing the one or more subtasks, and using the number of data tables generated after executing the one or more subtasks as the number of dependent tasks.

[0013] In a possible implementation, the table information also includes an update frequency; obtaining the table information of the target data table further includes: obtaining the number of updates of the target data table within a preset time period; and determining the update frequency based on the preset time period and the number of updates.

[0014] In one possible implementation, determining the health of the target data table based on the task information and the table information corresponding to the target data table includes: normalizing the data included in the task information and the table information to obtain normalized data; and inputting the normalized data into a linear regression model to obtain the health of the target data table.

[0015] In one possible implementation, the optimizing the target data table based on the health of the target data table includes: when the health is less than a preset threshold, determining to optimize the target data table; when the health is greater than or equal to the preset threshold, determining not to optimize the target data table; or, inputting the health and the normalized data into a logistic regression model, and outputting a result of whether to optimize the target data table, wherein the value output by the logistic regression model is 0 or 1, 0 indicating that the target data table needs to be optimized, and 1 indicating that the target data table does not need to be optimized.

[0016] In a second aspect, the present application provides a data management device, the device comprising:

[0017] an acquiring unit, configured to acquire task information corresponding to a target data table, wherein the task information includes a processor occupancy time for generating the target data table;

[0018] Acquire table information of the target data table, the table information including storage space occupied by the target data table and the number of dependent tasks, the number of dependent tasks indicating the number of tables using data in the target data table;

[0019] The processing unit is configured to determine the health of the target data table according to the task information corresponding to the target data table and the table information, and determine whether to optimize the target data table based on the health of the target data table.

[0020] In a third aspect, the present application provides a computing device comprising a processor and a memory, wherein the processor and the memory may be interconnected via a bus or may be integrated together. The processor executes a computer program stored in the memory to implement the method described in the first aspect or any possible implementation of the first aspect.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a program or instruction. When the above program or instruction is run on a computer device, the computer device can execute the method described in the first aspect or any possible implementation of the first aspect.

[0022] By obtaining the log data of the file system and the metadata of each data table, the above-mentioned task information corresponding to each data table is obtained based on the log data, and the above-mentioned table information of each data table is obtained based on the metadata of each data table. Then, based on the task information and table information of each data table, the health of each table is determined and whether the data table should be optimized is determined. This can objectively evaluate the data, reduce the dependence on the experience and level of engineers in data management, reduce the difficulty and cost of data management in the file system, and improve the efficiency of data management. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 This is a system diagram provided by an embodiment of the present application;

[0025] Figure 2 This is a flow chart of a data management method provided in an embodiment of the present application;

[0026] Figure 3 This is a schematic diagram of the structure of a data management device provided in an embodiment of the present application;

[0027] Figure 4 It is a structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0029] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0030] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0031] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0032] In specific implementations, the devices described in the embodiments of the present invention include, but are not limited to, other portable devices such as mobile phones, laptop computers, or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or touch pads). It should also be understood that in some embodiments, the device is not a portable communication device, but rather a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or touch pads).

[0033] In the following discussion, devices including a display and a touch-sensitive surface are described. However, it should be understood that a device may include one or more other physical user interface devices such as a physical keyboard, mouse, and / or joystick.

[0034] In the embodiment of the present invention, the computing device may include various terminal devices such as mobile phones, tablet computers, personal digital assistants (PDAs), mobile Internet devices (MIDs), laptop computers, smart wearable devices (such as smart watches and smart bracelets), etc., and the embodiment of the present invention is not limited thereto. The computing device supports various human-computer interaction methods, and the method of obtaining information and instructions can be contact or contactless. Among them, the contact method can be that the user inputs information and instructions to the device through physical buttons, or can be that the information and instructions are input through virtual buttons and / or touch buttons, or can be that the user inputs information and instructions through single-point or multi-point control of the touch device screen, for example, the user inputs instructions to the device through a physical keyboard. The contactless method can be that the device obtains information and instructions by recognizing user gestures or voice, or the terminal obtains information and instructions by tracking and locating the movement of the user's eyes or head. For example, the user can activate the voice recognition function, and the terminal collects the user's voice for processing and recognition to obtain instructions. It can be understood that the above examples are all used as examples and cannot be understood as specific limitations.

[0035] like Figure 1 As shown, Figure 1 This is a diagram of the system architecture for the data management method provided in this application. This system architecture includes a NameNode and multiple DataNodes, with a master-slave architecture between the NameNode and the DataNodes. The NameNode acts as the master server (Master), managing the file system's namespace, file system metadata, and client access to files. The DataNode provides slave services for storing and managing data. Data is stored in the form of data tables on the DataNodes, which consist of one or more physical servers.

[0036] In the embodiment of the present application, data calculations on the file system are implemented using Hive on MapReduce. Data calculations (such as summing, aggregating, and grouping data) are performed using MapReduce. A MapReduce process is called a job. Typically, a user submits a task to the NameNode through a client. The task is split into multiple subtasks, i.e., jobs. After the file system executes the task, it generates a data table, stores the data table on the DataNode, and returns it to the client. That is, the generation of a data table requires the execution of one or more tasks, and a task includes one or more jobs.

[0037] In order to solve the above problems in the data management process, this application provides a data management method that can reduce the difficulty and cost of data management in the file system and improve the efficiency of data management. Figure 2 , Figure 2 This is a flow chart of a data management method provided by an embodiment of the present invention. Figure 2 As shown, the data management method includes S201 to S203.

[0038] S201 : Acquire task information corresponding to a target data table, where the task information includes a processor occupancy time when generating the target data table.

[0039] Among them, the method can be executed by a computing device such as a physical server in the above-mentioned NameNode, or a virtual machine or container in the physical server. In the embodiment of this application, a computing device is used as an example.

[0040] In an embodiment of the present application, the above-mentioned task information includes the processor occupancy time when generating the target data table, and the above-mentioned target data table is any data table stored in the DataNode of the file system. The file system stores the log data of the file system, and the log data includes the time the processor is occupied when executing each subtask. The computing device obtains the log data of the file system, and obtains the time each subtask of the multiple subtasks for generating the target data table occupies the processor from the log data according to the identifier of the target data table, wherein the target data table is generated after executing the task submitted by the user, and each task includes one or more subtasks; then the computing device determines the processor occupancy time of the target data table according to the time each subtask occupies the processor.

[0041] Data calculations in the file system are performed using MapReduce. This means that the process of generating a data table includes executing one or more MapReduce processes. A MapReduce process is called a job, and the log data includes the processor occupancy time of each of the one or more jobs executed when generating each data table. For a target data table, the log data corresponding to the target data table is obtained based on the target data table's identifier. Furthermore, the processor occupancy time of each of the multiple jobs executed when generating the target data table is obtained. By adding together the processor occupancy times of each job, the processor occupancy time corresponding to the target data table is obtained. The processor occupancy time can represent the computing resource consumption when generating the target data table.

[0042] It should be noted that if a job occupies multiple processors, the sum of the duration each processor is occupied is used as the processor occupation duration for that job. For example, if the process of generating the target data table includes three jobs, job1, job2, and job3, and if job1 occupies two processors, each occupying 100 milliseconds, then the processor occupation duration for that job is 200 milliseconds; job2 occupies one processor, which occupies 200 milliseconds; and job3 occupies four processors, each occupying 90 milliseconds, then the processor occupation duration for generating the target data table is 760 milliseconds.

[0043] In a possible implementation, the task information also includes the time span corresponding to the target data table, which refers to the time span of the source data used when generating the target data table. After the computing device obtains the log data corresponding to the target data table, it obtains the table identifier of the data table read by each of the multiple subtasks mentioned above when generating the target data table from the log data corresponding to the target data table, and obtains multiple table identifiers; based on the multiple table identifiers, it obtains the data table corresponding to each table identifier, and the data table corresponding to the table identifier includes the above source data, and then the computing device obtains the time span of the data recorded in each data table, and obtains multiple time spans, wherein each table corresponds to a time span; finally, the time span corresponding to the target data table is determined based on the multiple time spans. Usually, a data table is obtained by calculation, combination, etc. based on the data of other data tables. In the embodiment of the present application, the other data tables used to generate a data table are referred to as the source data tables of the data table, that is, the source data table includes the source data used when generating the target data table. The target data table may be generated from one or more source data tables. The log data corresponding to the target data table records the table identifiers of the source data tables read by each subtask when generating the target data table. The computing device queries the time span of the corresponding data table based on each table identifier, obtaining multiple time spans, one for each source data table. The computing device obtains the start and end times of each of these multiple time spans and uses the earliest start time and the latest end time as the time span of the target data table, or uses the difference between the latest end time and the earliest start time as the time span of the target data table.

[0044] For example, when a quarterly financial report is needed, four source data tables are required, including an income table, a personnel expenditure table, an equipment purchase table, and an equipment maintenance expenditure table. The time span can be 3 months or the start and end time of the quarter.

[0045] In a possible implementation, the above-mentioned task information also includes the amount of data read and the amount of data written to the target data table. The computing device can also obtain the amount of data read from the above-mentioned source data table and the amount of data written to the target data table when generating the target data table from the log data corresponding to the target data table. Among them, the amount of data read refers to the amount of data read from the source data table by one or more subtasks when generating the target data table; the amount of data written refers to the amount of data written to the target data table by one or more subtasks when generating the target data table. In the embodiment of the present application, a subtask is a job, so the amount of data read refers to the amount of data read from the source data table by multiple jobs when generating the target data table; the amount of data written refers to the amount of data written to the target data table by multiple jobs when generating the target data table. Among them, the amount of data read and the amount of data written can characterize the number of input / output (I / O) requests when generating the target data table, that is, the consumption of communication resources.

[0046] It should be understood that other information corresponding to the target data table can also be directly obtained from the log data of the system file or obtained through calculation, and this application will not go into details one by one.

[0047] S202: Acquire table information of the target data table, where the table information includes storage space occupied by the target data table and the number of dependent tasks, where the number of dependent tasks is the number of data tables generated based on the target data table.

[0048] In an embodiment of the present application, the table information includes the storage space occupied by the target data table and the number of dependent tasks. The number of dependent tasks is the number of other data tables generated using the data in the target data table; that is, the data of the tables generated using the data in the target data table as the source data table. The file system includes metadata for each data table. The computing device obtains the metadata corresponding to the target data table based on the target data table and obtains the storage space occupied by the target data table from the target data. The computing device also obtains one or more subtasks that read the target data table when executed from the metadata corresponding to the target data table, determines the number of data tables generated after executing the one or more subtasks, and uses the number of data tables generated after executing the one or more subtasks as the number of dependent tasks. In one implementation, the metadata of the target data table also records the task identifier of the job that reads the target data table. The computing device obtains the task identifier of the one or more jobs that read the target data table from the metadata corresponding to the target data table, determines the number of data tables generated after executing the one or more jobs, and uses the number of data tables generated after executing the one or more subtasks as the number of dependent tasks for the target data table. For example, after the target data table is generated, a total of 8 jobs read the data in the target data table during execution until the current moment. After the execution of these 8 jobs, 5 data tables are generated. Then the task dependency number of the target data table is 5.

[0049] It should be understood that the above-mentioned task dependency number may also be the number of jobs that read the data in the target data table during execution after the target data table is generated and before the current moment, and this embodiment of the application does not make a specific limitation.

[0050] In one possible implementation, the table information also includes an update frequency. When determining whether a target data table requires optimization, the computing device can obtain the number of updates to the target data table within a preset time period and then determine the target data table's update frequency based on the preset time period and the number of updates. In one implementation, the metadata corresponding to a data table also includes the update times of multiple updates to the data table. The computing device obtains multiple update times from the metadata corresponding to the target data table and determines the target data table's update frequency based on the earliest and latest update times. For example, if the earliest of the multiple update times in the metadata corresponding to the target data table is 8:00 AM and the latest is 8:00 PM, and there are 10 updates within this time period, the target data table's update frequency is 1.2 hours. Alternatively, the computing device obtains the number of updates within a preset time period before the current time and determines the target data table's update frequency based on the preset time period and the number of updates. For example, if the preset time period is 3 days and the computing device obtains from the metadata that the target data table has been updated 8 times within 72 hours before the current time, the target data table's update frequency is 9 hours. The above-mentioned preset duration can be 1 hour, one day, 10 days, etc. The embodiment of the present application does not impose any specific restrictions on the preset duration.

[0051] S203: Determine the health of the target data table according to the task information and the table information corresponding to the target data table.

[0052] According to the method of S201 above, task information of the target data table can be obtained. This task information includes the processor usage time when generating the target data table; it can also include the time span corresponding to the target data table, the amount of data read and written when generating the target task table, etc. According to the method of S202 above, table information of the target data table can be obtained. This table information includes the amount of storage space occupied by the target data table and the number of dependent tasks of the target data table; it can also include the update frequency of the target data table, etc.

[0053] After obtaining the task information and table information of the target data table, each data included in the task information and each data segment included in the table information are normalized to obtain the normalized data corresponding to each data. The normalized data is then input into a preset linear regression model to output the health of the target data table. The linear regression model can be a simple linear regression model or a multiple linear regression model.

[0054] In one possible implementation, the normalized data of the processor occupancy time when generating the target data table, the size of the storage space occupied by the target data table, and the number of dependent tasks of the target data table are input into a preset linear regression model, and the linear regression model outputs the health of the target data table.

[0055] In one possible implementation, the above-mentioned task information also includes the time span corresponding to the target data table, the data reading amount and the data writing amount when generating the target task table; the above-mentioned table information also includes the update frequency. After normalizing any one or more of the time span corresponding to the target data table, the data reading amount when generating the target task table, the data writing amount when generating the target task table or the update frequency, the data are input into the preset linear regression model together with the normalized data of the processor occupancy time, the size of the storage space occupied by the target data table and the number of dependent tasks of the target data table to obtain the health of the target data table.

[0056] S204: Optimize the target data table based on the health of the target data table.

[0057] In one possible implementation, after obtaining the health of the target data table through the linear regression model, if the health of the target data table is less than a preset threshold, it is determined to optimize the target data table; if the health of the target data table is greater than or equal to the preset threshold, it is determined not to optimize the target data table.

[0058] In one possible implementation, the health of the target data table and the normalized data are input into a logistic regression model, which outputs a result indicating whether the target data table should be optimized. The logistic regression model outputs a value of 0 or 1, where 0 indicates that the target data table needs to be optimized and 1 indicates that the target data table does not need to be optimized.

[0059] The computing device can determine whether a data table in the file system needs to be optimized through the above method. The computing device can traverse each data table in the file system at a preset time interval to determine whether each data table needs to be optimized, and then output the name and location information of the data table that needs to be optimized to prompt the administrator of the file system that there is a data table that needs to be optimized.

[0060] In one possible implementation, the computing device can provide optimization instructions based on the time span of the target data table. For example, if the target data table only requires business data from the previous month, but the source data table only contains business data from the previous three months, the computing device can provide guidance on how to optimize computing resources.

[0061] By implementing the data management method of the present application, whether the data table needs to be optimized is evaluated based on the relevant information of the data table extracted from the log data and metadata. This can objectively evaluate the data, reduce the dependence on the experience and level of engineers in data management, reduce the difficulty and cost of data management in the file system, and improve the efficiency of data management.

[0062] It should be noted that, for the above method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the order of the actions described. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present invention.

[0063] Other reasonable step combinations that can be thought of by those skilled in the art based on the above description also fall within the scope of protection of the present invention. Secondly, those skilled in the art should also be familiar with that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present invention.

[0064] Combined with the above Figure 1 and Figure 2 The data management method provided by the embodiment of the present application is described in detail. Figure 4 , introduces the data management device and computing equipment provided in the embodiments of the present application.

[0065] See Figure 3 , Figure 3 3 is a structural diagram of a data management device provided in an embodiment of the present application. The data management device 300 includes an acquisition unit 310 and a processing unit 320, wherein:

[0066] An acquisition unit 310 is used to acquire task information corresponding to a target data table, the task information including the processor occupancy time for generating the target data table; and to acquire table information of the target data table, the table information including the storage space occupied by the target data table and the number of dependent tasks, wherein the number of dependent tasks indicates the number of tables using the data in the target data table; that is, the target data table is used as a source data table, and the data of the table is generated using the data in the target data table.

[0067] The processing unit is configured to determine the health of the target data table according to the task information corresponding to the target data table and the table information, and determine whether to optimize the target data table based on the health of the target data table.

[0068] In one possible implementation, the acquisition unit 310 acquires task information corresponding to the target data table, specifically by acquiring log data from the file system and, based on an identifier of the target data table, obtaining from the log data the processor time occupied by each of multiple subtasks used to generate the target data table. The target data table is generated after executing a task submitted by a user, and each task includes one or more subtasks. The processing unit 320 is further configured to determine the processor occupancy duration of the target data table based on the processor time occupied by each of the subtasks.

[0069] In one possible implementation, the above-mentioned task information also includes a time span corresponding to the target data table, and the time span indicates the time span of the source data used when generating the target data table; the above-mentioned acquisition unit 310 obtains the task information corresponding to the target data table, specifically including: obtaining from the above-mentioned log data the table identifier of the data table read by each subtask in the multiple subtasks when generating the target data table, to obtain multiple table identifiers; obtaining the time span of the data recorded in the data table corresponding to each table identifier according to the multiple table identifiers, to obtain multiple time spans, wherein each table corresponds to a time span; and determining the time span corresponding to the target data table according to the multiple time spans.

[0070] The above-mentioned processing unit 320 determines the health of the target data table based on the task information and table information corresponding to the target data table, specifically including: normalizing the data included in the task information and table information corresponding to the above-mentioned target data table to obtain normalized data; then inputting the normalized data into the linear regression model to obtain the health of the target data table.

[0071] In one possible implementation, the processing unit 320 determines whether to optimize the target data table based on the health of the target data table, specifically including: when the health of the target data table is less than a preset threshold, determining to optimize the target data table; when the health is greater than or equal to the preset threshold, determining not to optimize the target data table; or, inputting the health and the normalized data into a logistic regression model, and outputting a result of whether to optimize the target data table.

[0072] Specifically, the operations performed by the data management device 300 can refer to the above Figure 2 The related operations performed by the computing device in the illustrated method embodiment are not described in detail here.

[0073] See also Figure 4 , Figure 4 4 is a schematic diagram of a computing device provided in an embodiment of the present application. The computing device 400 includes: a processor 410, a communication interface 420, and a memory 430. The processor 410, the communication interface 420, and the memory 430 are interconnected via a bus 440.

[0074] The specific implementation of the various operations performed by the processor 410 can refer to the specific operations of the computing device in the above method embodiment. For example, the processor 410 is used to perform the above Figure 2 The related operations described in S201 to S203 are not repeated here.

[0075] The processor 410 can have a variety of specific implementation forms. For example, the processor 410 can be a central processing unit (CPU) or a graphics processing unit (GPU). The processor 410 can also be a single-core processor or a multi-core processor. The processor 410 can be a combination of a CPU and a hardware chip. The above-mentioned hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The above-mentioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The processor 410 can also be implemented solely using a logic device with built-in processing logic, such as an FPGA or a digital signal processor (DSP).

[0076] Communication interface 420 can be a wired interface or a wireless interface for communicating with other modules or devices. Memory 430 can be a non-volatile memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Memory 430 can also be a volatile memory, which can be a random access memory (RAM) used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0077] The memory 430 can be used to store program codes and data, so that the processor 410 can call the program codes stored in the memory 430 to execute the operation steps of implementing data management in the above method embodiment. In addition, the computing device 400 may include a Figure 4 Show more or fewer components, or configure components differently.

[0078] The bus 440 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus 440 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0079] Optionally, the computing device 400 may further include an input / output interface 450 , to which an input / output device is connected for receiving input information and outputting operation results.

[0080] Specifically, the specific implementation of the various operations performed by the computing device 400 can refer to the specific operations performed by the computing device in the above method embodiment, which will not be repeated here.

[0081] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a processor, the method steps in the above method embodiment can be implemented. The specific implementation of the processor of the computer-readable storage medium in executing the above method steps can refer to the specific operations of the above method embodiment, which will not be repeated here.

[0082] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0083] The above embodiments can be implemented in whole or in part through software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0084] The steps in the method of the embodiment of the present application can be adjusted in order, combined or deleted according to actual needs; the modules in the device of the embodiment of the present application can be divided, combined or deleted according to actual needs.

[0085] The above is a detailed introduction to the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data management method, characterized in that: include: Obtaining task information corresponding to a target data table, the task information including a processor occupation time when generating the target data table, a time span corresponding to the target data table, a data read amount of the target data table, and a data write amount of the target data table, wherein the time span indicates a time span of source data used when generating the target data table, the source data being data in a source data table used to generate the target data table, the data read amount of the target data table being an amount of data read from the source data table by one or more subtasks when generating the target data table, and the data write amount of the target data table being an amount of data written to the target table by one or more subtasks when generating the target data table; The obtaining of task information corresponding to the target data table includes: obtaining log data of a file system, and obtaining, from the log data according to an identifier of the target data table, a processor occupancy time of each of a plurality of subtasks used to generate the target data table, wherein the target data table is generated after executing a task submitted by a user, each task includes one or more subtasks, and determining the processor occupancy time of the target data table according to the processor occupancy time of each subtask; The acquiring task information corresponding to the target data table further includes: acquiring, from the log data, a table identifier of a data table read by each of the multiple subtasks when generating the target data table, to obtain multiple table identifiers; acquiring, based on the multiple table identifiers, a time span of data recorded in the data table corresponding to each table identifier, to obtain multiple time spans, wherein each table corresponds to a time span; and determining the time span corresponding to the target data table based on the multiple time spans; The acquiring task information corresponding to the target data table further includes: when generating the target data table from the log data corresponding to the target data table, using the amount of data read from the source data table as the data read amount of the target data table, and using the amount of data written to the target data table as the data write amount of the target data table; Acquiring table information of the target data table, the table information including storage space occupied by the target data table and the number of dependent tasks, the number of dependent tasks being the number of data tables generated based on the target data table, the table information also including an update frequency, the acquiring the table information of the target data table further comprising: acquiring the number of updates of the target data table within a preset duration, and determining the update frequency based on the preset duration and the number of updates; Determining the health of the target data table according to the task information and the table information corresponding to the target data table, including: inputting normalized data of the time span corresponding to the target data table, the amount of data read from the target data table, the amount of data written to the target data table, and the update frequency, together with normalized data of the processor occupancy time, the size of the storage space occupied by the target data table, and the number of dependent tasks of the target data table, into a preset linear regression model to obtain the health of the target data table; The target data table is optimized based on the health of the target data table.

2. The method according to claim 1, characterized in that The acquiring of the table information of the target data table includes: Obtaining metadata of the target data table, and obtaining storage space occupied by the target data table from the metadata; One or more subtasks for reading the target data table are obtained from the metadata, the number of data tables generated after executing the one or more subtasks is determined, and the number of data tables generated after executing the one or more subtasks is used as the number of dependent tasks.

3. The method according to claim 1 or 2, characterized in that The determining the health of the target data table according to the task information and the table information corresponding to the target data table includes: Normalizing the data included in the task information and the table information to obtain normalized data; The normalized data is input into a linear regression model to obtain the health of the target data table.

4. The method according to claim 3, characterized in that The optimizing the target data table based on the health of the target data table includes: When the health level is less than a preset threshold, determining to optimize the target data table; or, The health degree and the normalized data are input into a logistic regression model, and when an output of the logistic regression model is a preset value, it is determined to optimize the target data table.

5. A data management device, configured to execute the method according to any one of claims 1 to 4, characterized in that: The device comprises: an acquiring unit, configured to acquire task information corresponding to a target data table, wherein the task information includes a processor occupancy time for generating the target data table; Acquire table information of the target data table, the table information including storage space occupied by the target data table and the number of dependent tasks, the number of dependent tasks indicating the number of tables generated based on the target data table; a processing unit, configured to determine the health of the target data table according to the task information corresponding to the target data table and the table information; When the health of the target data table is less than a preset threshold, it is determined to optimize the target data table.

6. A computing device, characterized in that Including memory and processor: The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory, so that the computing device performs the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The method comprises a program or an instruction, and when the program or the instruction is executed on a computer device, the method according to any one of claims 1 to 4 is executed.

Images