Method, device and system for determining cold and hot data

Abstract

The application provides a method, device and system for determining cold and hot data. The method comprises: obtaining M pieces of data development logic execution statements, M being a positive integer greater than or equal to 2; parsing the M pieces of data development logic execution statements to obtain N correlation weight values, the N correlation weight values corresponding to N data tables one by one, the N data tables being different data tables in K data tables, the K data tables being data tables obtained after parsing the M pieces of data development logic execution statements, the ith correlation weight value being used for indicating the number of data tables in the K data tables which have a correlation relationship with the ith data table in the N data tables, N and K being positive integers greater than or equal to 1, K being less than or equal to N, and i = 1, 2, 3, …, N; and determining cold and hot data in the N data tables according to the N correlation weight values. The method can effectively reduce system overhead while ensuring that the determination result of the cold and hot data has high accuracy.

Description

Technical Field

[0001] This application relates to the field of storage technology, and more specifically, to a method, apparatus, and system for determining hot and cold data. Background Technology

[0002] With the rapid development and widespread application of technologies such as cloud computing, data is experiencing explosive growth. However, the value of data is not uniform. Therefore, it is necessary to effectively distinguish between hot and cold data in massive amounts of data (e.g., data in a data lake), and formulate better storage strategies based on this distinction to save system memory.

[0003] In existing technologies, the hotness or coldness of data is determined by accessing the underlying data engine and based on the frequency of data access. However, the above methods suffer from high system overhead and inaccurate judgment when determining hot or cold data. Summary of the Invention

[0004] This application provides a method, apparatus, and system for determining hot and cold data. The method can effectively reduce system overhead while ensuring high accuracy in determining the hot and cold data.

[0005] In a first aspect, a method for determining hot and cold data is provided, characterized in that the method includes:

[0006] To retrieve M data points, execute the logic statement where M is a positive integer greater than or equal to 2.

[0007] Parsing the M data development logic execution statements yields N association weight values, each corresponding to one of the N data tables. These N data tables are distinct from the K data tables, which are the data tables obtained after parsing the M data development logic execution statements. The i-th association weight value among the N association weight values ​​indicates the number of data tables in the K data tables that are related to the i-th data table in the N data tables. N and K are positive integers greater than or equal to 1, and K is less than or equal to N, i = 1, 2, 3, ..., N.

[0008] Based on these N correlation weight values, determine the hot and cold data in these N data tables.

[0009] The data platform for retrieving M data development logic execution statements is not specifically limited. For example, the data platform can be a data lake, data warehouse, or data middle platform, etc.

[0010] In the above technical solution, the acquired data is directly parsed using the logic execution statements. Based on the parsing results, the relationships between various data tables in the data platform are determined. The relationship weights of the corresponding data tables are then determined, and the hot / cold attributes of the data tables are determined based on these weights. This method avoids the need to access the underlying data engine as required by existing technologies, effectively reducing system overhead while ensuring high accuracy in determining hot / cold data.

[0011] In conjunction with the first aspect, in certain implementations of the first aspect, the execution statements of the M data development logic are parsed to obtain N related weight values, including:

[0012] Based on the parameter information of the M data development logic execution statements, the M data development logic execution statements are parsed to obtain the data table corresponding to each data development logic execution statement;

[0013] Count the number of data tables that are related to the i-th data table in the data table corresponding to each data development logic execution statement;

[0014] The association weight value of the i-th data table is determined based on the number of data tables that are related to the i-th data table.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the parameter information of the M data development logic execution statements includes at least the following information: the operation type information of the M data development logic execution statements and the operation parameter information of the data development logic execution statements.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the N association weight values ​​are determined based on the number of data tables that are related to the i-th data table, including:

[0017] If the number of data tables that are related to the i-th data table in the K data tables is P, then the i-th association weight value is determined to be equal to P, where P is an integer greater than or equal to zero.

[0018] In the above technical solution, the association weight value of each data table can be determined based on the established association relationships between the data tables.

[0019] In conjunction with the first aspect, in certain implementations of the first aspect, the hot and cold data in the N data tables are determined based on the N association weight values, including:

[0020] If the i-th association weight value is greater than or equal to the preset association weight value, then the data table corresponding to the i-th association weight value is determined to contain hot data, i = 1, 2, 3, ..., N;

[0021] If the i-th association weight value is less than the preset association weight value, then the data table corresponding to the i-th association weight value is determined to contain cold data.

[0022] The preset association weight value can be set according to the specific application scenario, and this application embodiment does not make specific limitations on it.

[0023] In the above technical solution, by comparing the preset association weight value with the association weight value of each data table, the hot and cold attributes of each data table can be determined quickly and easily.

[0024] In conjunction with the first aspect, the execution statements for the M data development logic differ in some implementations of the first aspect.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, the M data development logic execution statements include structured query statements (SQL).

[0026] In a second aspect, an apparatus for determining hot and cold data is provided, the apparatus comprising a module or unit for performing the method described in the first aspect and any implementation thereof.

[0027] Thirdly, an apparatus for determining hot and cold data is provided, the apparatus comprising: a memory for storing a program; and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method of the first aspect and any implementation thereof.

[0028] Fourthly, a computer-readable medium is provided that stores program code for execution by a device, the program code including methods for performing the first aspect and any implementation thereof.

[0029] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the methods described in the first aspect and any implementation thereof.

[0030] In a sixth aspect, a chip is provided, the chip including a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes the methods in the first aspect and any implementation thereof.

[0031] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in the first aspect and any implementation thereof.

[0032] The aforementioned chip can be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0033] In a seventh aspect, a system is provided, comprising the apparatus for determining hot and cold data as described in the second aspect and / or the device for determining hot and cold data as described in the third aspect. Attached Figure Description

[0034] Figure 1 This is a schematic flowchart of a method 100 for determining hot and cold data provided in an embodiment of this application.

[0035] Figure 2 This is a schematic flowchart of another method 200 for determining hot and cold data provided in the embodiments of this application.

[0036] Figure 3 This is a schematic structural diagram of a device 3000 for determining hot and cold data provided in an embodiment of this application.

[0037] Figure 4 This is a schematic structural diagram of a device 4000 for determining hot and cold data provided in an embodiment of this application.

[0038] Figure 5 This is a structural schematic diagram of a system 5000 provided in an embodiment of this application. Detailed Implementation

[0039] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0040] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0041] In this application, the terms "first," "second," and "third" are used to distinguish identical or similar items that have essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "third," nor are they limited in quantity or execution order.

[0042] This application will present various aspects, embodiments, or features relating to systems that may include multiple devices, components, modules, etc. It should be understood and appreciated that individual systems may include additional devices, components, modules, etc., and / or may not include all the devices, components, modules, etc. discussed in conjunction with the accompanying drawings. Furthermore, combinations of these approaches are also possible.

[0043] Furthermore, in the embodiments of this application, the words "exemplary," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplary" is intended to present the concept in a concrete manner.

[0044] In the embodiments of this application, "corresponding" and "corresponding" can sometimes be used interchangeably. It should be noted that when the distinction is not emphasized, their intended meanings are consistent.

[0045] In the embodiments of this application, sometimes the subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0046] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0047] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0048] The relevant technologies of this application are described below:

[0049] To facilitate understanding, before describing the method for determining hot and cold data provided in the embodiments of this application, the relevant terms involved in the embodiments of this application will be briefly introduced first.

[0050] 1. Hot and cold data

[0051] Hot and cold data refer to data that is accessed at different frequencies. Generally speaking, data that is accessed frequently is called hot data, and data that is accessed infrequently is called cold data.

[0052] 2. Structured Query Language (SQL)

[0053] SQL is a database query and programming language used to access, query, update, and manage relational database systems.

[0054] SQL is a high-level, non-procedural programming language that allows users to work on high-level data structures. It does not require users to specify how data is stored, nor does it require users to understand the specific data storage methods. Therefore, different database systems with completely different underlying structures can use the same Structured Query Language as an interface for data input and management. SQL can be nested, which gives it great flexibility and powerful functionality.

[0055] 3. Data Lake

[0056] A data lake is a large repository for storing a wide variety of raw data within an enterprise. The data in a data lake can be accessed, processed, analyzed, and transmitted.

[0057] The data in a data lake is raw data obtained from multiple data sources within an enterprise, and for different purposes, the same raw data may have multiple copies that conform to specific internal model formats. Therefore, the data processed in a data lake can be of any type, from structured data to completely unstructured data.

[0058] 4. Data warehouse (DW or DWH)

[0059] A data warehouse is a strategic collection of all types of data that supports decision-making processes at all levels within an enterprise. It is a single data store created for analytical reporting and decision support purposes. It provides guidance for business process improvement, monitoring of time, cost, quality, and control for enterprises that require business intelligence.

[0060] 5. Data Platform

[0061] A data platform refers to the use of data technology to collect, compute, store, and process massive amounts of data, while unifying standards and definitions. After unifying the data, the data platform creates standardized data, which is then stored to form a big data asset layer, thereby providing efficient services to customers.

[0062] 6. Relational database (RD)

[0063] A relational database is a database that uses a relational model to organize data. It stores data in rows and columns for user convenience. These rows and columns are called tables, and a set of tables makes up the database. Users retrieve data from the database through queries, which are executable codes used to define specific areas within the database. The relational model can be simply understood as a two-dimensional table model, and a relational database is a data organization composed of two-dimensional tables and the relationships between them. Common relational databases include Oracle and SQL Server.

[0064] In existing technologies, the hotness or coldness of data is determined by accessing the underlying data engine and based on the frequency of data access. For example, to improve the management efficiency of data lakes (e.g., the intelligent data lake operation platform DAYU), it is usually necessary to determine which data in the data lake is considered cold data and which is considered hot data during the data lake governance process. This method determines the hotness or coldness of data in the data lake by recording the frequency of data access, but this method relies on interfaces provided by the underlying data engine, which has the problems of high system overhead and insufficient accuracy.

[0065] This application provides a method for determining hot and cold data. This method effectively reduces system overhead while ensuring high accuracy in determining the hot and cold data.

[0066] Below, in conjunction with Figure 1 The method for determining hot and cold data provided in the embodiments of this application will be described in detail.

[0067] Figure 1 This is a schematic flowchart of a method 100 for determining hot and cold data provided in an embodiment of this application.

[0068] like Figure 1 As shown, method 100 may include steps 110 to 130. Steps 110 to 130 will be described in detail below.

[0069] Step 110: Obtain M data development logic execution statements, where M is a positive integer greater than or equal to 2.

[0070] In this embodiment of the application, the data platform for obtaining M data development logic execution statements is not specifically limited.

[0071] For example, the data platform may be a data lake, that is, M pieces of data development logic execution statements are obtained from the data lake. For example, the data platform may be a data warehouse, that is, M pieces of data development logic execution statements may also be obtained from the data warehouse. For example, the data platform may further be a data middle platform, that is, M pieces of data development logic execution statements may also be obtained from the data middle platform.

[0072] Optionally, in some implementation manners, the M pieces of data development logic execution statements include SQL. That is, the M pieces of data development logic execution statements may be SQL statements.

[0073] Optionally, in other implementation manners, the M pieces of data development logic execution statements may also be other types of database languages. For example, in another example, the M pieces of data development logic execution statements may also be Oracle statements.

[0074] For example, normally, the statements stored in a data lake are SQL statements. When M pieces of data development logic execution statements are obtained from the data lake, it means that M SQL statements are obtained from the data lake.

[0075] In the embodiments of the present application, no specific limitation is imposed on the types of the M pieces of data development logic execution statements.

[0076] For example, when the M pieces of data development logic execution statements are SQL statements, the types of the M SQL statements may include, but are not limited to the following types: query type (select), insert type (insert), update type (update), delete type (delete), etc.

[0077] For example, in an example, when one SQL statement among the M SQL statements is of select type, it can be expressed as: select cno,cname from courses where credit=3, and this statement can be understood as finding the course numbers and course names of courses with 3 credits. For example, in an example, when one SQL statement among the M SQL statements is of insert type, it can be expressed as: insert into infos values('s100102','Lin Chong','Male',22,2,to_date('2020-5-9 06:30:10','yyyy-mm-dd hh24:mi:ss')).

[0078] Optional. In some implementations, the M data development logic execution statements obtained above are not identical. This means that the fields in the data table corresponding to each of the M data development logic execution statements are not entirely the same.

[0079] For example, when the above data development logic executes SQL statements, three SQL statements are retrieved from the data lake, which will be denoted as SQL#1, SQL#2, and SQL#3 for ease of description. If parsing SQL#1 reveals the following fields in the corresponding table: `id` and `name`, and parsing SQL#2 reveals the following fields: `id`, `age`, and `name`, then SQL#1 and SQL#2 can be considered different. If parsing SQL#1 reveals the `id` field, and parsing SQL#2 reveals the `name` field, then SQL#1 and SQL#3 can also be considered different. It is understandable that if parsing SQL#1 reveals the `id` field, and parsing SQL#2 reveals the `id` field, then SQL#1 and SQL#2 can be considered the same.

[0080] Optionally, the following steps may be included before step 110:

[0081] Retrieve K data development logic execution statements, where the K data development logic execution statements are not identical, and K is a positive integer greater than or equal to 1.

[0082] Select M distinct data development logic execution statements from the K data development logic execution statements.

[0083] Step 120: Parse the M data development logic execution statements to obtain N association weight values. The N association weight values ​​correspond one-to-one with the N data tables. The N data tables are different from the K data tables. The K data tables are the data tables obtained after parsing the M data development logic execution statements. The i-th association weight value among the N association weight values ​​is used to indicate the number of data tables in the K data tables that have an association relationship with the i-th data table in the N data tables. N and K are positive integers greater than or equal to 1, and K is less than or equal to N, i = 1, 2, 3, ..., N;

[0084] The above N correlation weight values ​​correspond one-to-one with N data tables. This means that the i-th correlation weight value among the N values ​​corresponds to the i-th data table. In other words, the i-th correlation weight value is the correlation weight value of the corresponding i-th data table.

[0085] The data tables in the above K data tables that are related to the i-th data table can be understood as data tables in the K data tables that have one or more of the same fields as the i-th data table.

[0086] For example, in one scenario, when the above M data development logic execution statements are SQL statements, parsing the M SQL statements yields three data tables, denoted as Table 1, Table 2, and Table 3. Table 1 only includes the `id` field, Table 2 includes both `id` and `age` fields, and Table 3 only includes the `name` field. In this case, Table 2 can be considered to be the data table related to Table 1, Table 1 to Table 2, and no data table related to Table 3.

[0087] In this embodiment of the application, parsing M data development logic execution statements and N associated weight values ​​may include the following steps:

[0088] Based on the parameter information of the M data development logic execution statements, the M data development logic execution statements are parsed to obtain the data table corresponding to each data development logic execution statement;

[0089] Count the number of data tables that are related to the i-th data table in the data table corresponding to each data development logic execution statement;

[0090] The i-th association weight value is determined based on the number of data tables that are related to the i-th data table.

[0091] The parameter information of the aforementioned M data development logic execution statements may include at least the following: operation type information and operation parameter information of the M data development logic execution statements. It should be understood that the parameter information of the data development logic execution statements is not specifically limited in this embodiment. For example, the parameter information of the data development logic execution statements may also include parameter information of other data development logic execution statements besides the operation type information and operation parameter information.

[0092] Optionally, in other implementations, parsing the M data development logic execution statements and the N associated weight values ​​may include the following steps:

[0093] Parse the first data development logic execution statement among the M data development logic execution statements to obtain the data table corresponding to the first data development logic execution statement;

[0094] Parse the p-th data development logic execution statement among the M data development logic execution statements to obtain the data table corresponding to the p-th data development logic execution statement, where p = 2, 3, 4, ..., M;

[0095] Based on the data tables corresponding to the first data development logic execution statement to the p-th data development logic execution statement, determine the association weight value of each data table in the data table corresponding to the p-th data development logic execution statement, and update the association weight value of each data table in the data tables corresponding to the first data development logic execution statement to the (p-1)-th data development logic execution statements.

[0096] In this embodiment of the application, the operation type information and operation parameter information of the data development logic execution statement are not specifically limited.

[0097] In one example, when the execution statements for the above M data development logic are SQL statements, the SQL operation type information can include, but is not limited to, the following types: query type, insert type, update type, or delete type. For example, the SQL operation parameter information can include, but is not limited to, the following types: fields of the data table referenced by the SQL, the data table referenced by the SQL, the database referenced by the SQL, and the SQL statement type, etc.

[0098] In this embodiment of the application, determining N association weight values ​​based on the number of data tables that are associated with the i-th data table may include the following steps:

[0099] If the number of data tables that are related to the i-th data table is P, then the i-th association weight value is determined to be equal to P, where P is an integer greater than or equal to zero.

[0100] For example, if the number of tables in K data tables that are related to the i-th data table is 3, then the association weight value of the i-th table is determined to be 3. Conversely, if the number of tables in K data tables that are related to the i-th table is 1, then the association weight value of the i-th table is determined to be 1.

[0101] Step 130: Determine the hot and cold data in the N data tables based on the N associated weight values.

[0102] In this embodiment of the application, determining hot and cold data in N data tables based on N correlation weight values ​​may include the following steps:

[0103] If the i-th association weight value is greater than or equal to the preset association weight value, then the data table corresponding to the i-th association weight value is determined to contain hot data, i = 1, 2, 3, ..., N;

[0104] If the i-th association weight value is less than the preset association weight value, then the data table corresponding to the i-th association weight value is determined to contain cold data.

[0105] The preset association weight value can be set according to the specific application scenario, and this application embodiment does not impose specific limitations on it. For example, when the above M data development logic execution statements are SQL statements, after parsing the M SQL statements, 20 data tables are obtained, and 10 of these 20 data tables are different. In this case, the preset association weight value can be set to 3, 4, or 5, etc.

[0106] It should be understood that the above Figure 1 This is for illustrative purposes only and does not constitute any limitation on the embodiments of this application. For example, the above SQL statements can also be replaced with other statements that store data tables in a database (e.g., Oracle statements).

[0107] The method for determining hot and cold data provided in this application avoids the need to access the underlying data engine as required in existing technologies. This method directly parses the data development logic execution statements obtained from the data platform, determines the relationships between various data tables in the data platform based on the parsing results, determines the association weight values ​​of the corresponding data tables based on the association relationships, and determines the hot and cold attributes of the data tables based on the association weight values. This method effectively reduces system overhead while ensuring high accuracy in determining hot and cold data.

[0108] Below, in conjunction with Figure 2 Taking the execution statement of data development logic as an example, this application introduces a specific embodiment of determining hot and cold data.

[0109] Figure 2 This is a schematic flowchart of another method 200 for determining hot and cold data provided in the embodiments of this application.

[0110] like Figure 2 As shown, method 200 may include steps 210 to 240. Steps 210 to 240 are described in detail below. It should be understood that... Figure 2 This is for illustrative purposes only and does not constitute any limitation on this application. For example, the SQL statement in method 200 can also be replaced with other types of statements that store data tables.

[0111] Step 210: The SQL collector retrieves M distinct SQL statements from the data lake, where M is a positive integer greater than or equal to 2.

[0112] Among them, M different SQL statements can be understood as the data table fields corresponding to each of the M SQL statements not being completely the same.

[0113] Step 220: Store the M SQL statements using a relational database.

[0114] In one example, the relational database mentioned above could be a SQL Server database.

[0115] Step 230: Use an SQL parser to parse the M SQL statements stored in the relational database to obtain K data tables. The K data tables include N different data tables, where N and K are positive integers greater than or equal to 1, and K is less than or equal to N.

[0116] The specific analysis method is the same as that in step 120 of method 100 above, and will not be described in detail here.

[0117] Step 240: Use a graph database to process the obtained K data tables to obtain a first record table. The first record table is a mesh table used to record the relationships between the various data tables in the above N different data tables and the corresponding N relationship weight values.

[0118] The process of determining the relationships between the various data tables in the N distinct data tables and the corresponding relationship weight values ​​is the same as the method in step 120 of method 100 above, and will not be described in detail here.

[0119] In the above technical solution, a network-structured table relationship (i.e., the first record table mentioned above) is finally recorded in the graph database. Based on the established network structure table, the degree of association between each data table in the data lake can be determined. According to the degree of association and a preset threshold, the hot or cold attribute (i.e., cold data or hot data) of each data table in the data lake can be further determined.

[0120] In one example, if the i-th association weight value recorded in the first record table is greater than or equal to the preset association weight value, then the data table corresponding to the i-th association weight value is determined to contain hot data, i = 1, 2, 3, ..., N;

[0121] If the i-th association weight value is less than the preset association weight value, then the data table corresponding to the i-th association weight value is determined to contain cold data.

[0122] The preset association weight value can be set according to the specific application scenario, and this application embodiment does not impose a specific limitation on it. For example, after parsing M SQL statements, 20 data tables are obtained, and 10 of these 20 data tables are different. In this case, the preset association weight value can be set to 3, 4, or 5, etc.

[0123] Optionally, the following operations may be included after steps 210 to 240 above:

[0124] The popularity server is used to publicly announce the hot / cold attributes of each of the N distinct data tables identified above.

[0125] The parsing server uses the hot / cold attributes mentioned above to obtain the hot / cold status of N different data tables.

[0126] The above text combined Figure 1 and Figure 2 This application describes in detail the method for determining hot and cold data provided in this application. Below, in conjunction with... Figures 3 to 5 This application provides a detailed description of the apparatus, device, and system for determining hot and cold data. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be found in the foregoing method embodiments.

[0127] Figure 3 This is a schematic structural diagram of a device 3000 for determining hot and cold data provided in this application.

[0128] like Figure 3 As shown, the device 3000 includes an acquisition unit 3001 and a processing unit 3002.

[0129] Acquisition unit 3001 is used to acquire M data development logic execution statements, where M is a positive integer greater than or equal to 2;

[0130] Processing unit 3002 is used to parse the M data development logic execution statements to obtain N association weight values. The N association weight values ​​correspond one-to-one with N data tables. The N data tables are different from the K data tables. The K data tables are the data tables obtained after parsing the M data development logic execution statements. The i-th association weight value among the N association weight values ​​is used to indicate the number of data tables in the K data tables that have an association relationship with the i-th data table in the N data tables. N and K are positive integers greater than or equal to 1, and K is less than or equal to N, i = 1, 2, 3, ..., N;

[0131] The processing unit 3002 is also used to determine the hot and cold data in the N data tables based on the N associated weight values.

[0132] Optionally, in some implementations, the processing unit 3002 is also used to perform the following operations:

[0133] Based on the parameter information of the M data development logic execution statements, the M data development logic execution statements are parsed to obtain the data table corresponding to each data development logic execution statement;

[0134] Count the number of data tables that are related to the i-th data table in the data table corresponding to each data development logic execution statement;

[0135] The association weight value of the i-th data table is determined based on the number of data tables that are related to the i-th data table.

[0136] Optionally, in some implementations, the parameter information of the M data development logic execution statements includes at least the following: the operation type information of the M data development logic execution statements and the operation parameter information of the data development logic execution statements.

[0137] Optionally, in some implementations, the processing unit 3002 is also used to perform the following operations:

[0138] If the number of data tables that are related to the i-th data table in the K data tables is P, then the i-th association weight value is determined to be equal to P, where P is an integer greater than or equal to zero.

[0139] Optionally, in some implementations, the processing unit 3002 is also used to perform the following operations:

[0140] If the i-th association weight value is greater than or equal to the preset association weight value, then the data table corresponding to the i-th association weight value is determined to contain hot data, i = 1, 2, 3, ..., N;

[0141] If the i-th association weight value is less than the preset association weight value, then the data table corresponding to the i-th association weight value is determined to contain cold data.

[0142] Optionally, in some implementations, the execution statements for these M data development logics are different.

[0143] Optionally, in some implementations, the M data development logic execution statements include structured query statements (SQL).

[0144] Below, in conjunction with Figure 4 This will be illustrated using a device that determines hot and cold data, including a transceiver, processor, and memory.

[0145] Figure 4 This is a schematic structural diagram of a device 4000 for determining hot and cold data provided in this application. Figure 4 As shown, the device 4000 includes a transceiver 4010, a processor 4020, and a memory 4030. The transceiver 4010, processor 4020, and memory 4030 communicate with each other via internal connections to transmit control and / or data signals. The memory 4030 stores computer programs, and the processor 4010 retrieves and runs the computer programs from the memory 4030 to control the transceiver 4020 to transmit and receive signals.

[0146] Specifically, the functions of transceiver 4010 are... Figure 3 The specific functions of the acquisition unit 3001 shown are as follows, and will not be described in detail here.

[0147] Specifically, the functions of the processor 4020 are... Figure 3 The specific functions of the processing unit 3002 shown are as follows, and will not be described in detail here.

[0148] In this embodiment of the application, the device 4000 for determining hot and cold data should include a processor 4020. Optionally, in some implementations, the device 4000 for determining hot and cold data may also include a transceiver 4010.

[0149] Optionally, in some implementations, the device 4000 for determining hot and cold data may also include a memory 4030.

[0150] Figure 5 This is a structural schematic diagram of a system 5000 provided in an embodiment of this application. For example... Figure 5 As shown, the system 5000 includes: the device 3000 for determining hot and cold data as described above and / or the device 4000 for determining hot and cold data.

[0151] This application provides a computer program product that, when run on a device 3000 for determining hot and cold data, enables the device 3000 to execute method 100 and / or method 200 in the above-described method embodiments.

[0152] Those skilled in the art will recognize that the method steps and units described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0153] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be found in the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0154] In the several embodiments provided in this application, the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, or it may be an electrical, mechanical, or other form of connection.

[0155] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0156] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0157] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0158] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0159] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer program instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0160] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0161] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0162] Furthermore, the term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. The term "at least one" in this application can represent "one" and "two or more." For example, at least one of A, B, and C can represent: A existing alone, B existing alone, C existing alone, A and B existing simultaneously, A and C existing simultaneously, C and B existing simultaneously, and A, B, and C existing simultaneously.

[0163] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for determining hot and cold data, characterized in that, The method includes: To retrieve M data points, execute the logic statement where M is a positive integer greater than or equal to 2. The M data development logic execution statements are parsed to obtain N association weight values. The N association weight values ​​correspond one-to-one with N data tables. The N data tables are different from K data tables. The K data tables are data tables obtained after parsing the M data development logic execution statements. The i-th association weight value among the N association weight values ​​is used to indicate the number of data tables in the K data tables that have an association relationship with the i-th data table in the N data tables. N and K are positive integers greater than or equal to 1, and K is less than or equal to N, i=1,2,3,…,N; Based on the N correlation weight values, determine the hot and cold data in the N data tables; The process of parsing the M data development logic execution statements yields N related weight values, including: The M data development logic execution statements are parsed based on their parameter information to obtain the data table corresponding to each statement. Count the number of data tables that are related to the i-th data table in the data table corresponding to each data development logic execution statement; The i-th association weight value is determined based on the number of data tables that are associated with the i-th data table.

2. The method according to claim 1, characterized in that, The parameter information of the M data development logic execution statements includes at least the following: the operation type information of the M data development logic execution statements and the operation parameter information of the data development logic execution statements.

3. The method according to claim 1, characterized in that, The step of determining the N association weight values ​​based on the number of data tables that are associated with the i-th data table includes: If the number of data tables in the K data tables that are related to the i-th data table is P, then the i-th association weight value is determined to be equal to P, where P is an integer greater than or equal to zero.

4. The method according to any one of claims 1-3, characterized in that, The step of determining the hot and cold data in the N data tables based on the N association weight values ​​includes: If the i-th association weight value is greater than or equal to the preset association weight value, then the data table corresponding to the i-th association weight value is determined to contain hot data, i=1,2,3,…,N; If the i-th association weight value is less than the preset association weight value, then the data table corresponding to the i-th association weight value is determined to contain cold data.

5. The method according to any one of claims 1-3, characterized in that, The execution statements for the M data development logics are different.

6. The method according to any one of claims 1-3, characterized in that, The M data development logic execution statements include structured query statements (SQL).

7. An apparatus for determining hot and cold data, characterized in that, The device includes a processor and a memory, the memory being used to store computer execution instructions, and the processor being used to read the computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, Includes a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 6.

9. A chip system, characterized in that, It includes at least one processor and an interface; the at least one processor is configured to invoke and run a computer program to cause the chip system to perform the method as described in any one of claims 1 to 6.

Images