Data processing method and device, server and computer readable storage medium

Abstract

The application provides a data processing method and device, a server and a computer readable storage medium, and belongs to the technical field of data processing. The method comprises the following steps: obtaining usage information of at least one target data index at at least one data consumer; determining target data value information of the at least one target data index based on the usage information and data processing information of the at least one target data index; and deleting data corresponding to a target data index that does not meet a first preset condition from a corresponding database based on the target data value information. This can ensure that data with low usage value and high resource consumption is deleted, so that the data does not need to be processed during data processing, thereby reducing the calculation amount during data processing, ensuring the accuracy of the calculation result, and improving the data processing efficiency and accuracy.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, server, and computer-readable storage medium. Background Technology

[0002] Internet services generate massive amounts of data when providing services to users. As these services evolve, this data is continuously processed, resulting in a constant accumulation of new data. With this accumulation, both storage and computing costs increase.

[0003] Currently, all data generated by internet services is stored, which may result in servers storing a lot of invalid or expired data. During subsequent data processing, this data will increase the amount of computation and affect the data processing, leading to errors in the results obtained based on this data, thus resulting in low data processing efficiency and accuracy. Summary of the Invention

[0004] This application provides a data processing method, apparatus, server, and computer-readable storage medium, which can improve data processing efficiency and accuracy. The technical solution is as follows:

[0005] On the one hand, a data processing method is provided, which includes:

[0006] Obtain usage information of at least one target data metric in at least one data consumer, the usage information being used to indicate the data usage corresponding to the target data metric;

[0007] Based on the usage information and the data processing information of the at least one target data indicator, the target data value information of the at least one target data indicator is determined. The data processing information is used to indicate the process of obtaining the data corresponding to the target data indicator based on at least one raw data.

[0008] Based on the target data value information, delete the data corresponding to the target data indicator whose target data value information does not meet the first preset condition from the corresponding database.

[0009] On one hand, a data processing apparatus is provided, the apparatus comprising:

[0010] The acquisition module is used to acquire usage information of at least one target data indicator in at least one data consumer, and the usage information is used to indicate the data usage corresponding to the target data indicator.

[0011] An information determination module is used to determine the target data value information of the at least one target data indicator based on the usage information and the data processing information of the at least one target data indicator. The data processing information is used to indicate the process of obtaining the data corresponding to the target data indicator based on at least one raw data.

[0012] The deletion module is used to delete data corresponding to target data indicators that do not meet the first preset condition from the corresponding database based on the target data value information.

[0013] In one possible implementation, the determining unit is configured to determine a data processing procedure corresponding to the target data indicator based on data lineage, wherein the data lineage is used to indicate the generation and being generated relationship between data; based on the data processing procedure, determine at least one original data indicator corresponding to the target data indicator; and determine the computational cost information based on the data volume of the original data corresponding to the at least one original data indicator and the data processing procedure.

[0014] In one possible implementation, the third determining submodule is used to perform a weighted calculation on the first data value information and the second data value information based on the weights corresponding to the first data value information and the second data value information, to obtain the target data value information of the at least one target data indicator.

[0015] In one possible implementation, the device further includes:

[0016] The indicator determination module is used to determine at least one target data indicator corresponding to the target data based on target data from at least one data consumer.

[0017] In one possible implementation, the indicator determination module is used to obtain the data keyword corresponding to the target data, and in the database, match the data keyword with at least one data field name, and determine at least one data field name that satisfies the second preset condition as the at least one target data indicator.

[0018] In one possible implementation, the device further includes:

[0019] The indicator storage module is used to store any one of the at least one target data indicators in the indicator data service system if the target data indicator is not stored in the indicator data service system. The indicator data service system is used to store data indicators and the relationships between data indicators.

[0020] In one possible implementation, the device further includes:

[0021] The matching module is used to match the data type of any target data metric with the data types already existing in the database.

[0022] The data volume determination module is used to determine the target data volume corresponding to the target data indicator based on the current remaining capacity of the database and the data volume corresponding to the target data indicator if there is a data type in the database with a similarity greater than a preset threshold to the data type of the target data indicator. The target data volume is used to indicate the amount of data that needs to be stored.

[0023] The data storage module is used to store the data corresponding to the target data volume into the database.

[0024] In one possible implementation, the device further includes:

[0025] The adjustment module is used to adjust the first preset condition based on the current remaining capacity of the database.

[0026] On one hand, a server is provided, which includes one or more processors and one or more memories, wherein at least one piece of program code is stored in the one or more memories, the program code being loaded and executed by the one or more processors to implement the operations performed by the data processing method.

[0027] On the one hand, a computer-readable storage medium is provided, which stores at least one piece of program code that is loaded and executed by a processor to implement the operations performed by the data processing method.

[0028] On the one hand, an application product is provided, which stores at least one piece of program code, which is loaded and executed by the processor of an electronic device to implement the above-mentioned data processing method.

[0029] The solution provided in this application obtains usage information of at least one target data indicator from at least one data consumer. Based on the usage information and data processing information of at least one target data indicator, it determines the target data value information of at least one target data indicator. Since the target data value information can indicate the user's usage needs for the target data indicator and the consumption of the target data indicator, it can delete data corresponding to the target data indicator whose target data value information does not meet the first preset condition from the corresponding database based on the target data value information. This ensures that data with low usage value and high resource consumption is deleted, so there is no need to process these data during data processing, reducing the amount of computation during data processing, and also ensuring the accuracy of the calculation results, thereby improving the efficiency and accuracy of data processing. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of the implementation environment of a data processing method provided in an embodiment of this application;

[0032] Figure 2 This is a flowchart of a data processing method provided in an embodiment of this application;

[0033] Figure 3 This is a flowchart of a data processing method provided in an embodiment of this application;

[0034] Figure 4 This is an architecture diagram of an indicator data service system provided in an embodiment of this application;

[0035] Figure 5 This is an architecture diagram of an indicator data service system provided in an embodiment of this application;

[0036] Figure 6 This is an overall flowchart of a data processing method provided in an embodiment of this application;

[0037] Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0038] Figure 8 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0040] The following is an explanation of the relevant technical terms and terminology used in this application:

[0041] Data as a Service (DaaS) offers data as a commodity to any organization or individual with a need. Through integrated data access, users can move data more quickly without having to consider the underlying data source. If users require slightly different data structures or need to access data in specific locations, DaaS can meet their needs very quickly with minimal changes.

[0042] Metrics: Metrics, commonly used in data analysis, are essentially measures. They range from monitoring and evaluating the status of business processes to measuring the performance of a specific functional module or the effectiveness of one's own activities. From an operational perspective, a good metric should possess four characteristics:

[0043] (1) It has value from a business perspective;

[0044] (2) It can measure the true state of business operations;

[0045] (3) Simple and executable;

[0046] (4) Everyone agrees.

[0047] From a technical perspective, a good metric needs to have four characteristics:

[0048] (1) Easy to collect and quick to measure;

[0049] (2) High accuracy;

[0050] (3) It can be decomposed in multiple dimensions;

[0051] (4) From a single data source.

[0052] In a more technical sense, the most basic metric is a valuable field or record in a data entity (table) (such as the amount paid in a transaction). The next level of metrics is another data metric calculated based on one or more fields (such as the total revenue of a supermarket in a day). And then there are more advanced types of data measurement.

[0053] Disparate data source: In a broad sense, it refers to multiple data sources with different data structures, access methods, and formats.

[0054] Data lineage: refers to the chain of data generation, that is, how the data came about and what processes and stages it went through.

[0055] Data consumers: refers to the applications that ultimately use the data, such as reports, visualization products, business decision-making systems, etc.

[0056] Figure 1 This is a schematic diagram illustrating the implementation environment of a data processing method provided in this application embodiment. See also... Figure 1 The implementation environment includes: terminal 101 and server 102.

[0057] Terminal 101 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal 101 and server 102 can be connected directly or indirectly via wired or wireless communication, which is not limited herein. Terminal 101 can provide users with various functional services based on the various Internet services provided by server 102. Users can use terminal 101 to search, browse, chat, watch videos, etc. During the process of searching, browsing, chatting, and watching videos, users can generate a large amount of data. Terminal 101 can obtain the data generated by users and send this data to server 102.

[0058] Terminal 101 can refer to one of a plurality of terminals; this embodiment uses terminal 101 as an example. Those skilled in the art will understand that the number of terminals can be more or less. For example, there may be only one terminal, or there may be dozens or hundreds, or even more. This embodiment does not limit the number of terminals or the type of device.

[0059] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Server 102 and terminal 101 can be directly or indirectly connected via wired or wireless communication, which is not limited herein. Server 102 can receive data sent by terminal 101, and server 102 can also maintain multiple databases to store the received data. The database can store data in the form of data tables, and optionally, other forms of data storage can also be used, which is not limited herein. Server 102 can obtain the data required for data analysis from the database to output based on this data. Optionally, the number of the above-mentioned servers can be more or less, which is not limited in this embodiment. Of course, server 102 can also include other functional servers to provide more comprehensive and diversified services.

[0060] Figure 2 This is a flowchart of a data processing method provided in an embodiment of this application. See also... Figure 2 The method includes:

[0061] 201. The server obtains usage information of at least one target data metric in at least one data consumer, and the usage information is used to indicate the data usage corresponding to the target data metric.

[0062] It should be noted that the data consumer is the object that uses the data, such as reports, visualization products, and business decision-making systems. Optionally, the data consumer can also be other types, and this application embodiment does not limit this. The data usage can be the frequency or number of times the data is used, and optionally, the data usage can also be other types, and this application embodiment does not limit this.

[0063] 202. The server determines the target data value information of the at least one target data indicator based on the usage information and the data processing information of the at least one target data indicator. The data processing information is used to indicate the process of obtaining the data corresponding to the target data indicator based on at least one raw data.

[0064] It should be noted that the data processing information includes computational cost information and storage cost information. The computational cost information indicates which raw data were processed from the target data indicator, and what computational rules were used to process these raw data to obtain the data corresponding to the target data indicator. The storage cost information indicates the cost required to store the raw data. Since different indicators correspond to different processing procedures and raw data, different indicators have different data processing information. The specific content of the data processing information is not limited in the embodiments of this application.

[0065] Usage information for target data metrics can indicate the frequency and number of times the data metric is used, while data processing information for target data metrics can indicate the complexity and cost of calculating the data metric. The target data value information determined based on usage information and data processing information can more comprehensively represent the importance of the data metric and improve the accuracy of the determined data value information.

[0066] 203. Based on the target data value information, the server deletes the data corresponding to the target data indicator whose target data value information does not meet the first preset condition from the corresponding database.

[0067] It should be noted that the corresponding database is the database that stores the data corresponding to the target data value information. By deleting data corresponding to the target data indicators that do not meet the first preset condition from the database, we can avoid these data occupying storage space, ensure that there is sufficient storage space in the database to store subsequent data, and reduce the storage pressure on the server, thereby improving the server's processing speed.

[0068] The solution provided in this application, by determining the target data value information of the target data indicator, can indicate the user's demand for the target data indicator and the consumption required to obtain it. Therefore, deletion can be performed based on the target data indicator value information, ensuring that data with low usage value and high resource consumption is deleted. This eliminates the need to process such data during data processing, reduces the computational load during data processing, ensures the accuracy of the calculation results, and improves the efficiency and accuracy of data processing.

[0069] Figure 3 This is a flowchart of a data processing method provided in an embodiment of this application. See also... Figure 3 The method includes:

[0070] 301. The server determines at least one target data indicator corresponding to the target data based on target data from at least one data consumer.

[0071] It should be noted that the data consumer can define the required data metrics and process the raw data to obtain the data corresponding to the required data metrics. Then, the consumer can send the data corresponding to the required data metrics to the server. The server can receive this data, use it as target data, and determine at least one target data metric based on this target data.

[0072] In one possible implementation, the server can obtain the data keyword corresponding to the target data, match the data keyword with at least one data field name in the database, and determine the at least one data field name that meets the second preset condition as the at least one target data indicator.

[0073] The second preset condition can be that the similarity between the data keyword and the data field name is greater than a preset threshold. Optionally, the second preset condition can also be other conditions, which are not limited in this embodiment.

[0074] It should be noted that the server can provide various services, such as indicator data services, and thus the solution provided in this application embodiment can be implemented through the indicator server system. This indicator data service system can process data from different data sources and processed by different business teams to obtain multiple data indicators. It can then store the data lineage of each data indicator and the data corresponding to that indicator with the data corresponding to other data indicators. This data lineage is used to indicate the generation and being generated relationship between data. For example, if the data corresponding to data indicator A is obtained by adding the data corresponding to data indicator B and the data corresponding to data indicator C, then it can be determined that the data corresponding to data indicator A has a data lineage with the data corresponding to data indicator B and the data corresponding to data indicator C. This indicator data service system can also detect the usage of each data indicator on different data consumers, record the data consumption behavior of each data indicator on different data consumers, and based on the recorded data consumption behavior, integrate the usage information of each data indicator's corresponding data on different data consumers to obtain the usage information of each data indicator. The indicator data service system can monitor data usage without requiring data producers to track data consumption, thus isolating data producers and consumers, improving process efficiency, and consequently, data processing efficiency.

[0075] The architecture of this indicator data service system can be found in [reference needed]. Figure 4 , Figure 4 This is an architecture diagram of an indicator data service system provided in this application embodiment. Data from data sources 401, 402, and 403 can be processed at a unified indicator data service 404 to obtain multiple data indicators. Various data consumers, such as applications 405, 406, and 407, can obtain data indicators from the unified indicator data service 404. The indicator data service 404 can detect the usage of the data corresponding to the data indicators at each data consumer, and then determine the data value information of the data through data usage and data lineage.

[0076] It should be noted that the specific architecture of this indicator data service system can be found in [link to relevant documentation]. Figure 5 , Figure 5This is an architecture diagram of an indicator data server system provided in this application embodiment. Data from different data sources can be stored in different data tables, such as transaction data storage table 501, user data storage table 502, and behavior data storage table 503. The server can process the data in transaction data storage table 501 in step 504 to obtain transaction indicator 507, process the data in user data storage table 502 in step 505 to obtain user count indicator 508, and process the data in behavior data storage table 503 in step 506 to obtain behavior statistics indicator 509. The indicator service 510 can store transaction indicator 507, user count indicator 508, and behavior statistics indicator 509. The indicator service 510 can also record the usage of each data indicator on different data consumers, such as consumers 511, 512, and 513, such as the number of times and frequency of indicator consumption, so as to determine the data value information of the data corresponding to each data indicator based on the number of times and frequency of indicator consumption.

[0077] It should be noted that, for any one of the at least one target data indicator, if that target data indicator is not stored in the indicator data service system, then that target data indicator will be stored in the indicator data service system. This system stores data indicators and the relationships between them. By storing the target data indicators in the indicator data service system, the target data indicator can be directly retrieved from the indicator server system the next time data corresponding to it is received, without needing to process the data again to obtain the target data indicator. This reduces the server's processing load, increases server processing speed, and improves data processing efficiency.

[0078] After storing any target data metric in the metric data service system, the server can also store the target data corresponding to that metric in the database. For example, the server can match the data type of the target data metric with existing data types in the database. If a data type with a similarity greater than a preset threshold exists in the database, the server determines the target data volume corresponding to the target data metric based on the current remaining capacity of the database and the amount of data corresponding to the target data metric. This target data volume indicates the amount of data to be stored, and the data corresponding to this target data volume is stored in the database. Storing only a portion of the data of the same or similar type already stored in the database avoids excessive storage space usage, thereby reducing the server's storage pressure and improving server processing speed.

[0079] 302. The server obtains usage information of at least one target data metric in at least one data consumer, and the usage information is used to indicate the data usage corresponding to the target data metric.

[0080] In this system, a single data metric can correspond to multiple data consumers; that is, a single data metric can be used by multiple different data consumers. If a data metric is used by any one data consumer, the metric data service system can store the usage relationship between that data metric and that particular data consumer, so that the server can determine the data consumer using the data metric based on the metric's identifier.

[0081] It should be noted that the server can identify the data consumers who have used the target data indicator in the indicator data system based on the indicator identifier of the target data indicator. Then, it can obtain usage information of at least one target data indicator from the indicator data service system across different data consumers, so as to evaluate the data value of the target data corresponding to the target data indicator based on this usage information. This usage information can be the frequency and number of times the data is used; optionally, it can also be other types, which are not limited in this embodiment.

[0082] 303. Based on the usage information, the server determines the first data value information of the at least one target data indicator, the first data value information being used to indicate the data importance corresponding to the target data indicator.

[0083] It should be noted that the server can set different first data value information for different numerical ranges based on the value range to which the usage information belongs. For example, if the usage information is the frequency or number of times the data is used, the server can divide the data into different ranges based on the frequency or number of times the data is used, and then set different first data value information for different numerical ranges. Taking dividing the data ranges based on the number of times the data is used as an example, for data used no more than 1,000 times, the first data value information can be set to 10; for data used more than 1,000 times but not more than 2,000 times, the first data value information can be set to 20, and so on. For data used more than 1,000 times but not more than 10,000 times, the first data value information can be set to 100, thus obtaining the first data value information corresponding to each numerical range.

[0084] In one possible implementation, the server can determine the numerical range corresponding to the acquired usage information based on the usage information, and then match the data range with the corresponding first data value information to determine the first data value information corresponding to the usage information, which is used as the first data value information of the at least one target data indicator.

[0085] By using information, the primary data value information of the target data indicator can be determined, that is, the data value information of the target data itself corresponding to the target data indicator. This primary data value information can represent the user's usage needs, thus reflecting the importance of the target data and providing a reference for determining whether the target data needs to be deleted.

[0086] 304. The server obtains the storage cost information of the raw data corresponding to the target data indicator.

[0087] It should be noted that the storage cost information can be the amount of data, that is, the storage space occupied by the data. The server can set different storage cost information for different numerical ranges based on the data amount; the larger the data amount, the higher the storage cost, and the smaller the data amount, the lower the storage cost. Optionally, the storage cost information can also be other types, which are not limited in this embodiment.

[0088] 305. The server obtains information on the computational cost incurred in processing the raw data corresponding to the target data indicator.

[0089] In one possible implementation, the server can determine the data processing procedure corresponding to the target data indicator based on the data lineage, where the data lineage is used to indicate the generation and being generated relationship between data. Based on the data processing procedure, at least one original data indicator corresponding to the target data indicator is determined, and the computational cost information for obtaining the target data corresponding to the target data indicator by processing the original data corresponding to the at least one original data indicator is determined.

[0090] The computational cost information can be the CPU utilization rate during the processing of the raw data corresponding to the at least one raw data indicator. The server can set different computational cost information for different CPU utilization ranges; a higher CPU utilization rate results in a higher computational cost, and a lower CPU utilization rate results in a lower computational cost. Optionally, the computational cost information can also be of other types, which are not limited in this embodiment.

[0091] 306. Based on the storage cost information and the computing cost information, the server determines the second data value information of the at least one target data indicator.

[0092] In one possible implementation, the server can directly add the storage cost information and the computation cost information to obtain the second data value information of at least one target data indicator.

[0093] In another possible implementation, the server can set different weights for the storage cost information and the computation cost information, and then perform a weighted calculation on the storage cost information and the computation cost information based on the weights corresponding to the storage cost information and the computation cost information to obtain the second data value information of the at least one target data indicator.

[0094] The weights of the storage cost information and the computation cost information can both be any values ​​greater than or equal to 0 and less than or equal to 1, and this application embodiment does not limit them.

[0095] 307. The server determines the target data value information of the at least one target data indicator based on the first data value information and the second data value information.

[0096] In one possible implementation, the server can directly add the first data value information and the second data value information to obtain the target data value information of at least one target data indicator.

[0097] In another possible implementation, the server can set different weights for the first data value information and the second data value information, and then perform a weighted calculation on the first data value information and the second data value information based on the weights corresponding to the first data value information and the second data value information to obtain the target data value information of the at least one target data indicator.

[0098] The weights of the first data value information and the second data value information can both be any values ​​greater than or equal to 0 and less than or equal to 1, and this application embodiment does not limit them.

[0099] It should be noted that, for steps 303, 306 and 307 above, after the server determines the first data value information, the second data value information and the target data value information, it can also perform normalization processing on the first data value information, the second data value information and the target data value information, etc., and this application embodiment does not limit this.

[0100] 308. Based on the target data value information, the server deletes the data corresponding to the target data indicator whose target data value information does not meet the first preset condition from the corresponding database.

[0101] The first preset condition can be that the value of the data information is greater than a preset threshold. Optionally, the first preset condition can also be other conditions, which are not limited in this embodiment. The preset threshold can be any value, which is also not limited in this embodiment.

[0102] It should be noted that the server can also adjust the first preset condition based on the current remaining capacity of the database. Specifically, the server can check the current remaining capacity of the database every first preset time interval. If the current remaining capacity of the database is large, the server can adjust the value of the preset threshold to a smaller value; if the current remaining capacity of the database is small, the server can adjust the value of the preset threshold to a larger value. The first preset time interval can be any value, and this embodiment does not limit it.

[0103] Furthermore, the server can determine the data value information of data from different data consumers already stored in the database at second preset intervals, and then delete data from the database whose data value information does not meet the first preset condition. This second preset interval can be any value, and this embodiment does not limit it. By determining the data value information of the data in the database at second preset intervals, invalid or expired data in the database can be cleaned up in a timely manner, reducing the storage pressure on the server, ensuring that subsequently generated valid data can be stored normally, and also improving the server's processing speed and data processing speed.

[0104] It should be noted that the processes of steps 301 to 308 above can be found in [reference needed]. Figure 6 , Figure 6 This is an overall flowchart of a data processing method provided in an embodiment of this application. The data consumer can generate target data in step 601. The server can check whether the target data exists in the indicator service system in step 602. If the target data does not exist in the indicator service system, the business team can process and produce the target data in step 603. The server can determine the target data indicator based on the received target data in step 301. Then, in step 302, the usage information of the target data indicator in step 604 is obtained. Thus, in step 303, the determination of the first data value information in step 605 is achieved. Through steps 304 to 307, the determination of the target data value information in step 606 is achieved. Finally, in step 308, the data management in step 308 is achieved. The specific process can be found in steps 301 to 308 above, and will not be repeated here.

[0105] It should be noted that the solution provided in this application embodiment can be applied to various data consumption scenarios, such as reports, visualized data products, business decision-making systems, and operational analysis, and can directly manage data without manual intervention, thereby reducing manpower and improving data processing efficiency. Optionally, the solution provided in this application embodiment can also be applied to other more data consumption scenarios, and this application embodiment does not limit the application scenario.

[0106] The solution provided in this application, by determining the target data value information of the target data indicator, can indicate the user's demand for the target data indicator and the consumption required to obtain it. Therefore, deletion can be performed based on this target data indicator value information, ensuring that data with low usage value and high resource consumption is deleted. This eliminates the need to process such data during data processing, reducing the computational load and ensuring the accuracy of calculation results, thus improving data processing efficiency and accuracy. Deleting low-value, high-resource-consumption data from the database also reduces the storage and computational costs of invalid and expired data, reducing the storage and computational pressure on the server. Furthermore, the indicator data service system allows for unified management of data from heterogeneous data sources and monitoring of data usage. This eliminates the need for data producers to statistically analyze data consumption, reducing data processing costs. Moreover, the indicator data service system can isolate data producers and consumers, improving process efficiency and ultimately enhancing data processing efficiency.

[0107] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0108] Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. See also... Figure 7 The device includes:

[0109] The acquisition module 701 is used to acquire usage information of at least one target data indicator in at least one data consumer, and the usage information is used to indicate the data usage corresponding to the target data indicator.

[0110] Information determination module 702 is used to determine the target data value information of the at least one target data indicator based on the usage information and the data processing information of the at least one target data indicator. The data processing information is used to indicate the process of processing the data corresponding to the target data indicator based on at least one raw data.

[0111] The deletion module 703 is used to delete data corresponding to target data indicators whose target data value information does not meet the first preset condition from the corresponding database based on the target data value information.

[0112] The apparatus provided in this application determines the target data value information of the target data indicator. Since the target data value information can indicate the user's demand for the target data indicator and the consumption required to obtain the target data indicator, deletion can be performed based on the target data indicator value information. This ensures that data with low usage value and high resource consumption is deleted, so that these data do not need to be processed during data processing, reducing the amount of computation during data processing, and also ensuring the accuracy of the calculation results, thereby improving the efficiency and accuracy of data processing.

[0113] In one possible implementation, the information determination module 702 includes a first determination submodule, a second determination submodule, and a third determination submodule;

[0114] The first determining submodule is used to determine first data value information of the at least one target data indicator based on the usage information, and the first data value information is used to indicate the data importance corresponding to the target data indicator;

[0115] The second determining submodule is used to determine the second data value information of the at least one target data indicator based on the data processing information of the at least one target data indicator. The second data value information is used to indicate the data processing cost of the data.

[0116] The third determining submodule is used to determine the target data value information of the at least one target data indicator based on the first data value information and the second data value information.

[0117] In one possible implementation, the second determining submodule includes an acquisition unit and a determining unit;

[0118] The acquisition unit is used to acquire the storage cost information of the original data corresponding to the target data indicator, and to acquire the computing cost information consumed in processing the original data corresponding to the target data indicator.

[0119] The determining unit is used to determine the second data value information of the at least one target data indicator based on the storage cost information and the computing cost information.

[0120] In one possible implementation, the determining unit is configured to determine a data processing procedure corresponding to the target data indicator based on data lineage, wherein the data lineage is used to indicate the generation and being generated relationship between data; based on the data processing procedure, determine at least one original data indicator corresponding to the target data indicator; and determine the computational cost information based on the data volume of the original data corresponding to the at least one original data indicator and the data processing procedure.

[0121] In one possible implementation, the third determining submodule is used to perform a weighted calculation on the first data value information and the second data value information based on the weights corresponding to the first data value information and the second data value information, to obtain the target data value information of the at least one target data indicator.

[0122] In one possible implementation, the device further includes:

[0123] The indicator determination module is used to determine at least one target data indicator corresponding to the target data based on target data from at least one data consumer.

[0124] In one possible implementation, the indicator determination module is used to obtain the data keyword corresponding to the target data, and in the database, match the data keyword with at least one data field name, and determine at least one data field name that satisfies the second preset condition as the at least one target data indicator.

[0125] In one possible implementation, the device further includes:

[0126] The indicator storage module is used to store any one of the at least one target data indicators in the indicator data service system if the target data indicator is not stored in the indicator data service system. The indicator data service system is used to store data indicators and the relationships between data indicators.

[0127] In one possible implementation, the device further includes:

[0128] The matching module is used to match the data type of any target data metric with the data types already existing in the database.

[0129] The data volume determination module is used to determine the target data volume corresponding to the target data indicator based on the current remaining capacity of the database and the data volume corresponding to the target data indicator if there is a data type in the database with a similarity greater than a preset threshold to the data type of the target data indicator. The target data volume is used to indicate the amount of data that needs to be stored.

[0130] The data storage module is used to store the data corresponding to the target data volume into the database.

[0131] In one possible implementation, the device further includes:

[0132] The adjustment module is used to adjust the first preset condition based on the current remaining capacity of the database.

[0133] It should be noted that the data processing apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when processing data. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the server can be divided into different functional modules to complete all or part of the functions described above. In addition, the data processing apparatus and data processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0134] Figure 8 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 800 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 801 and one or more memories 802. Each memory 802 stores at least one line of program code, which is loaded and executed by the one or more processors 801 to implement the methods provided in the various method embodiments described above. Of course, the server 800 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 800 may also include other components for implementing device functions, which will not be elaborated upon here.

[0135] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including program code that can be executed by a processor to perform the data processing method described above. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0136] In an exemplary embodiment, a computer program product is also provided, including one or more lines of program code that can be executed by a server's processor to perform the method steps of the data processing method provided in the above embodiments.

[0137] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program or program code related to hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0138] The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain usage information of at least one target data metric in at least one data consumer, wherein the usage information is used to indicate the data usage corresponding to the target data metric, and the data usage includes the frequency or number of times the target data metric is used, wherein the target data corresponding to the target data metric is obtained based on at least one raw data processing. Based on the usage information, a first data value information for the at least one target data indicator is determined, wherein the first data value information is used to indicate the data importance corresponding to the target data indicator. Based on the data processing information of the at least one target data indicator, a second data value information of the at least one target data indicator is determined. The data processing information is used to indicate the process of obtaining the target data corresponding to the target data indicator based on at least one raw data. The second data value information is used to indicate the data processing cost of the data. Based on the first data value information and the second data value information, the target data value information of the at least one target data indicator is determined; Based on the target data value information, delete the target data corresponding to the target data indicator whose target data value information does not meet the first preset condition from the corresponding database. The first preset condition includes the value of the target data value information being greater than a preset threshold.

2. The method according to claim 1, characterized in that, The step of determining the second data value information of the at least one target data indicator based on the data processing information of the at least one target data indicator includes: Obtain the storage cost information of the original data corresponding to the target data indicator; Obtain information on the computational cost consumed in processing the raw data corresponding to the target data indicator; Based on the storage cost information and the computing cost information, the second data value information of the at least one target data indicator is determined.

3. The method according to claim 2, characterized in that, The information on the computational cost consumed in processing the raw data corresponding to the target data indicator includes: Based on data lineage, determine the data processing procedure corresponding to the target data indicator, wherein the data lineage is used to indicate the relationship between data generation and being generated; Based on the data processing procedure, at least one original data indicator corresponding to the target data indicator is determined; The calculation cost information is determined based on the amount of raw data corresponding to the at least one raw data indicator and the data processing procedure.

4. The method according to claim 1, characterized in that, The determination of the target data value information for the at least one target data indicator based on the first data value information and the second data value information includes: Based on the weights corresponding to the first data value information and the second data value information, the first data value information and the second data value information are weighted and calculated to obtain the target data value information of the at least one target data indicator.

5. The method according to claim 1, characterized in that, Before obtaining usage information of at least one target data metric from at least one data consumer, the method further includes: Based on target data from at least one data consumer, at least one target data indicator is determined corresponding to the target data.

6. The method according to claim 5, characterized in that, The step of determining at least one target data indicator corresponding to the target data based on target data from at least one data consumer includes: Obtain the data keywords corresponding to the target data; In the database, the data keywords are matched against at least one data field name; At least one data field name that meets the second preset condition is determined as the at least one target data indicator.

7. The method according to claim 5, characterized in that, After determining at least one target data indicator corresponding to the target data based on target data from at least one data consumer, the method further includes: For any one of the at least one target data indicator, if the target data indicator is not stored in the indicator data service system, then the target data indicator is stored in the indicator data service system, which is used to store data indicators and the relationships between data indicators.

8. The method according to claim 7, characterized in that, After storing any one of the target data indicators into the indicator data service system, the method further includes: Based on the data type of any of the target data indicators, match it with the data types already existing in the database; If there is a data type in the database that has a similarity to the data type of the target data indicator that is greater than a preset threshold, then based on the current remaining capacity of the database and the amount of data corresponding to the target data indicator, the target data amount is determined, and the target data amount is used to indicate the amount of data that needs to be stored. The data corresponding to the target data volume is stored in the database.

9. The method according to claim 1, characterized in that, After deleting data corresponding to target data indicators whose target data value information does not meet the first preset condition from the corresponding database based on the target data value information, the method further includes: The first preset condition is adjusted based on the current remaining capacity of the database.

10. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire usage information of at least one target data indicator in at least one data consumer. The usage information is used to indicate the data usage corresponding to the target data indicator. The data usage includes the frequency or number of times the target data indicator is used. The target data corresponding to the target data indicator is obtained based on at least one raw data. The first determining submodule in the information determining module is used to determine the first data value information of the at least one target data indicator based on the usage information, wherein the first data value information is used to indicate the data importance corresponding to the target data indicator; The second determining submodule in the information determining module is used to determine the second data value information of the at least one target data indicator based on the data processing information of the at least one target data indicator. The data processing information is used to indicate the process of obtaining the target data corresponding to the target data indicator based on at least one original data. The second data value information is used to indicate the data processing cost of the data. The third determining submodule in the information determining module is used to determine the target data value information of the at least one target data indicator based on the first data value information and the second data value information. The deletion module is used to delete target data corresponding to target data indicators whose target data value information does not meet a first preset condition from the corresponding database based on the target data value information. The first preset condition includes the value of the target data value information being greater than a preset threshold.

11. The apparatus according to claim 10, characterized in that, The second determining submodule includes an acquisition unit and a determining unit; The acquisition unit is used to acquire storage cost information of the original data corresponding to the target data indicator, and to acquire computation cost information consumed in processing the original data corresponding to the target data indicator. The determining unit is used to determine the second data value information of the at least one target data indicator based on the storage cost information and the computing cost information.

12. The apparatus according to claim 11, characterized in that, The acquisition unit is used for: Based on data lineage, determine the data processing procedure corresponding to the target data indicator, wherein the data lineage is used to indicate the relationship between data generation and being generated; Based on the data processing procedure, at least one original data indicator corresponding to the target data indicator is determined; The calculation cost information is determined based on the amount of raw data corresponding to the at least one raw data indicator and the data processing procedure.

13. The apparatus according to claim 10, characterized in that, The third determining submodule is used for: Based on the weights corresponding to the first data value information and the second data value information, the first data value information and the second data value information are weighted and calculated to obtain the target data value information of the at least one target data indicator.

14. The apparatus according to claim 10, characterized in that, The device further includes: The indicator determination module is used to determine at least one target data indicator corresponding to the target data based on target data from at least one data consumer.

15. The apparatus according to claim 14, characterized in that, The indicator determination module is used for: Obtain the data keywords corresponding to the target data; In the database, the data keywords are matched against at least one data field name; At least one data field name that meets the second preset condition is determined as the at least one target data indicator.

16. The apparatus according to claim 14, characterized in that, The device further includes: The indicator storage module is used to store any one of the at least one target data indicators in the indicator data service system if the target data indicator is not stored in the indicator data service system. The indicator data service system is used to store data indicators and the relationships between data indicators.

17. The apparatus according to claim 16, characterized in that, The device further includes: The matching module is used to match the data type of any target data indicator with the data types already existing in the database. The data volume determination module is used to determine the target data volume corresponding to the target data indicator based on the current remaining capacity of the database and the data volume corresponding to the target data indicator if there is a data type in the database with a similarity greater than a preset threshold to the data type of the target data indicator. The target data volume is used to indicate the amount of data that needs to be stored. The data storage module is used to store the data corresponding to the target data volume into the database.

18. The apparatus according to claim 10, characterized in that, The device further includes: The adjustment module is used to adjust the first preset condition based on the current remaining capacity of the database.

19. A server, characterized in that, The server includes one or more processors and one or more memories, wherein at least one piece of program code is stored in the one or more memories, the program code being loaded and executed by the one or more processors to perform the operations performed by the data processing method as described in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one piece of program code, which is loaded and executed by a processor to perform the operations of the data processing method as described in any one of claims 1 to 9.

21. An application product, characterized in that, The application product stores at least one piece of program code, which is loaded and executed by the processor of the electronic device to perform the operations performed by the data processing method as described in any one of claims 1 to 9.

Images