Statement execution method and device

Abstract

The embodiments of the present specification provide a statement execution method and device, wherein the statement execution method is applied to a data computing engine, including: receiving a data operation statement submitted to the data computing engine; obtaining a hotspot data structure identifier and a target operator matched by the hotspot data structure identifier by parsing the data operation statement, and constructing a hotspot key-value pair based on the hotspot data structure identifier and the target operator; when executing the data operation statement to a sub-operation statement carrying the hotspot data structure identifier, accessing the data storage space associated with the data computing engine according to the hotspot key-value pair; determining the hotspot data structure corresponding to the hotspot data structure identifier based on the access result, and continuing to execute the data operation statement based on the hotspot data structure.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of computer technology, and more particularly to a statement execution method and device. Background Art

[0002] With the development of computer technology, the requirements for improving data processing efficiency and resource utilization are getting higher and higher. Especially in the scenario where the data computing engine processes data, how to improve data processing efficiency and reduce resource consumption has always been an issue of concern. In the prior art, the data computing engine generates a large amount of intermediate data during the data processing process, and the intermediate data will be released after being used; in this process, if the intermediate data of a table is referenced multiple times, it will be calculated multiple times, which will not only occupy more computing resources, but also cause the problem of low data processing efficiency. Although the computing engine uses the cache method, the cache mechanism used by most computing engines requires manual debugging by the user, which requires the user to be familiar with the underlying original of the computing engine to be able to achieve it, and requires high professionalism. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention

[0003] In view of this, an embodiment of this specification provides a statement execution method. One or more embodiments of this specification also relate to a statement execution device, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a statement execution method is provided, which is applied to a data computing engine, and includes:

[0005] receiving a data operation statement submitted to the data computing engine;

[0006] Obtaining a hotspot data structure identifier and a target operator matching the hotspot data structure identifier by parsing the data operation statement, and constructing a hotspot key-value pair based on the hotspot data structure identifier and the target operator;

[0007] When executing the data operation statement to the sub-operation statement carrying the hotspot data structure identifier, accessing the data storage space associated with the data calculation engine according to the hotspot key-value pair;

[0008] The hot data structure corresponding to the hot data structure identifier is determined according to the access result, and the data operation statement is continued to be executed based on the hot data structure.

[0009] According to a second aspect of an embodiment of this specification, a statement execution device is provided, which is applied to a data computing engine, including:

[0010] A receiving module, configured to receive a data operation statement submitted to the data computing engine;

[0011] A parsing module, configured to obtain a hotspot data structure identifier and a target operator matching the hotspot data structure identifier by parsing the data operation statement, and construct a hotspot key-value pair based on the hotspot data structure identifier and the target operator;

[0012] An access module, configured to access the data storage space associated with the data computing engine according to the hotspot key-value pair when executing the data operation statement to the sub-operation statement carrying the hotspot data structure identifier;

[0013] The determination module is configured to determine the hot data structure corresponding to the hot data structure identifier according to the access result, and continue to execute the data operation statement based on the hot data structure.

[0014] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0015] Memory and processor;

[0016] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above statement execution method are implemented.

[0017] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned statement execution method are implemented.

[0018] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions, which implement the steps of the above statement execution method when executed by a processor.

[0019] The statement execution method provided in this embodiment, in order to improve the data processing efficiency during statement execution and reduce the professional requirements at the same time, can first obtain the hot data structure identifier and the target operator matching the hot data structure identifier by parsing the data operation statement after receiving the data operation statement submitted to the data calculation engine, and at this time, a hot key-value pair can be constructed based on the hot data structure identifier and the target operator; on this basis, when the data operation statement is executed to the sub-operation statement carrying the hot data structure identifier, it is explained that the intermediate data required to be used by the sub-operation statement is the hot data, and in this scenario, in order to improve the data processing efficiency, the hot data will be stored in the data storage space, so the data storage space associated with the data calculation engine can be accessed according to the hot key-value pair; the hot data structure corresponding to the hot data structure identifier is determined according to the access result, and after obtaining the hot data structure, the data operation statement can be continued to be executed based on the hot data structure. It can be seen from this that the statement execution method provided in this embodiment can first cache the intermediate data associated with the hot data in the data storage space, so that when the intermediate data is used again, it can be directly read and used from the data storage space, thereby improving the data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a schematic diagram of a statement execution method provided by an embodiment of this specification;

[0021] Figure 2 is a flow chart of a statement execution method provided by an embodiment of this specification;

[0022] Figure 3 It is a schematic diagram of caching intermediate data in a statement execution method provided by an embodiment of this specification;

[0023] Figure 4 is a flow chart of a processing process of a statement execution method provided by an embodiment of this specification;

[0024] Figure 5 It is a structural diagram of a statement execution device provided by an embodiment of this specification;

[0025] Figure 6 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0026] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0027] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0028] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0029] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0030] First, the terms involved in one or more embodiments of this specification are explained.

[0031] RDD (Resilient Distributed Datasets): Resilient distributed datasets is an abstract concept of distributed memory. RDD provides a highly restricted shared memory model, that is, RDD is a collection of read-only record partitions and can only be created by performing certain transformation operations (such as map, join, and group by) on other RDDs. These restrictions make the cost of implementing fault tolerance very low.

[0032] Spark: is a fast and general computing engine designed for large-scale data processing. Spark uses in-memory computing technology to store data in memory, which increases computing speed.

[0033] Flink: It is a powerful distributed big data processing engine. Flink supports stateful computing on finite and infinite data streams. It can be deployed in various cluster environments and quickly compute data of various sizes.

[0034] In this specification, a statement execution method is provided. This specification also relates to a statement execution device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0035] See also Figure 1 The schematic diagram shown, the statement execution method provided by this embodiment, in order to improve the data processing efficiency during statement execution and reduce the professional requirements at the same time, can first obtain the hot data structure identifier and the target operator matching the hot data structure identifier by parsing the data operation statement after receiving the data operation statement submitted to the data calculation engine, and at this time, the hot key-value pair can be constructed based on the hot data structure identifier and the target operator; on this basis, when the data operation statement is executed to the sub-operation statement carrying the hot data structure identifier, it is explained that the intermediate data required to be used by the sub-operation statement is the hot data, and in this scenario, in order to improve the data processing efficiency, the hot data will be stored in the data storage space, so the data storage space associated with the data calculation engine can be accessed according to the hot key-value pair; the hot data structure corresponding to the hot data structure identifier is determined according to the access result, and after obtaining the hot data structure, the data operation statement can be continued to be executed based on the hot data structure. It can be seen from this that the statement execution method provided by this embodiment can first cache the intermediate data associated with the hot data in the data storage space, so that when the intermediate data is used again, it can be directly read and used from the data storage space, thereby improving the data processing efficiency.

[0036] See also Figure 2 , Figure 2 A flowchart of a statement execution method provided according to an embodiment of the present specification is shown. The method is applied to a data computing engine and specifically includes the following steps.

[0037] Step S202: receiving a data operation statement submitted to the data computing engine.

[0038] The statement execution method provided in this embodiment is applied to a data computing engine, wherein the data computing engine may be a Spark computing engine or a Flink data computing engine, and this embodiment does not make any limitation here. The statement execution method provided in this embodiment can cache the hot intermediate data involved in the data operation statement in advance to the data storage space when the data computing engine executes insert, update or delete operations, so that other statements can read and use the intermediate data from the data storage space when using it, thereby effectively improving data processing efficiency and reducing the consumption of computing resources.

[0039] Specifically, data operation statements refer to SQL statements submitted when the data computing engine processes data. Based on this, the tables used multiple times are extracted from the SQL logic execution plan of the data computing engine and mapped with the action operators in the data computing engine, so that the hot table names can be found, and then they can be converted into RDD and cached, so as to support the reuse of intermediate data and improve the data processing efficiency of the data computing engine.

[0040] In actual applications, the statements that the data computing engine needs to execute are usually triggered by the client, and after the execution is completed, the execution results need to be fed back to the client. Therefore, the specific implementation method is as follows:

[0041] Receive a statement editing instruction submitted by a client node included in the data computing engine, determine a data operation statement according to the statement editing instruction, and then execute steps S204 to S208. Thereafter, generate data operation information according to the execution result of the data operation statement, and send the data operation information to the client node.

[0042] Specifically, the client node refers to the node corresponding to the client that has an interactive relationship with the data computing engine. It can enable the user to submit a statement editing instruction through the node, and generate a data operation statement corresponding to the user's needs according to the statement editing instruction, so as to be handed over to the data computing engine for subsequent data processing operations. Correspondingly, the data operation information specifically refers to the information that the data computing engine feeds back to the client node after executing the data operation statement. For example, the results of adding, deleting, modifying and checking data.

[0043] Based on this, when a statement editing instruction submitted by a client node included in the data computing engine is received, it means that the user corresponding to the client node needs to perform data operations through the data computing engine, such as adding, deleting, modifying, and checking the target data, so the data operation statement can be determined according to the statement editing instruction first; then steps S204 to S208 are executed. After that, after the data operation statement is executed, data operation information can be generated according to the execution result of the data operation statement, and the data operation information can be sent to the client node to inform the user of the data operation result.

[0044] Step S204, obtaining the hotspot data structure identifier and the target operator matching the hotspot data structure identifier by parsing the data operation statement, and constructing a hotspot key-value pair based on the hotspot data structure identifier and the target operator.

[0045] Specifically, after receiving the data operation statement submitted to the data computing engine, further, in order to improve data processing efficiency, the data computing engine will cache the hot intermediate data involved according to the SQL logical execution plan, and support other statements that need to use the intermediate data, and can directly read and use it from the data storage space, thereby improving data processing efficiency. Under this mechanism, considering that the data operation statement may involve multiple data calculations, and the data required to be used each time is different, in order to be able to execute the statement, the cached intermediate data can be read from the data storage space for use, and the hot data structure identifier and the target operator matching the hot data structure identifier can be obtained by parsing the data operation statement. On this basis, a hot key-value pair can be constructed based on the hot data structure identifier and the target operator, so that the hot data corresponding to the hot data structure can be read in the data storage space in combination with the hot key-value pair, thereby improving data processing efficiency.

[0046] Among them, the hotspot data structure identifier specifically refers to the identification information of the associated hotspot data structure in the data operation statement, which can be the setting identifier corresponding to the hotspot data structure, or the table name corresponding to the hotspot data structure. This embodiment does not make any limitation here. Correspondingly, the target operator specifically refers to the operator in the data operation statement that needs to use the hotspot data structure identifier to continue executing the statement, which can be operators such as group by, count, join, etc. This embodiment does not make any limitation here. Correspondingly, the hotspot key-value pair specifically refers to the Map key-value pair constructed according to the hotspot data structure identifier and the target operator, which can be read from the data storage space and used when the hotspot data structure corresponding to the hotspot data structure identifier is needed in the future. And by establishing this key-value pair, the blood relationship between the mapping identifier and the operator can be reflected to reflect the correlation between the two, so as to facilitate the subsequent execution of the operator to read the data structure for use.

[0047] Furthermore, when parsing data operation statements, since data operation statements are SQL statements that need to be executed by the data computing engine, they may involve multiple data tables and use multiple operators, and the intermediate data stored in the data storage space is generally a hot data structure, it is necessary to obtain the data structure identifier and the operator, and then select the hot data structure identifier and the target operator to use. In this embodiment, the specific implementation method is as follows:

[0048] The data operation statement is parsed to obtain at least one data structure identifier and at least one operator; a hot data structure identifier is screened from the at least one data structure identifier, and a target operator matching the hot data structure identifier is screened from the at least one operator.

[0049] Specifically, the data structure identifier specifically refers to the identifier corresponding to the data structure involved in the data operation statement obtained after parsing the data operation statement. Correspondingly, the operator specifically refers to the operator involved when executing the data operation statement, which is used to calculate the data corresponding to the data structure identifier.

[0050] Based on this, after obtaining the data operation statement, considering that the data operation statement may involve multiple data structure identifiers and operators, and in order to be able to read the reusable data structure from the data storage space in the future, the data operation statement can be parsed first, and at least one data structure identifier and at least one operator can be obtained according to the parsing result; on this basis, the hot data structure identifier can be screened from the at least one data structure identifier, and the target operator matching the hot data structure identifier can be screened from the at least one operator. In order to subsequently combine the data structure identifier and the target operator to construct a hot key-value pair, so that it can be used to read the data structure in the data processing stage.

[0051] On this basis, when selecting a hot data structure identifier from at least one data structure identifier and selecting a target operator from at least one operator, the number of references of the data structure can be combined to determine, and then the operator associated with the data structure identifier can be selected as the target operator. In this embodiment, the specific implementation method is as follows:

[0052] Determine the number of references corresponding to each data structure identifier; determine the hot data structure identifier in the at least one data structure identifier by comparing the number of references corresponding to each data structure identifier; and select an operator having an execution relationship with the hot data structure identifier as a target operator in the at least one operator.

[0053] Specifically, the number of citations refers to the number of times a data structure identifier is cited. This number can indicate whether the data structure corresponding to the data structure identifier is hot data. It can be determined by comparing it with a citation number threshold. The threshold can be set according to actual needs, and this embodiment does not make any limitation here.

[0054] Based on this, after parsing the data operation statement, at least one data structure identifier and at least one operator can be obtained. On this basis, the number of references corresponding to each data structure identifier can be determined first; thereafter, the number of references corresponding to each data structure identifier can be compared, and the data structure identifier with the highest number of references can be selected as the hotspot data structure identifier according to the comparison result; after the hotspot data structure identifier is determined, since the execution of the operator requires the use of different data structure identifiers, an operator having an execution relationship with the hotspot data structure identifier can be selected from at least one operator as the target operator according to the hotspot data structure identifier, so that the data structure identifier and the target operator can be combined to construct a hotspot key-value pair in the subsequent data processing stage, so that it can be used to read the data structure.

[0055] In addition, in addition to determining the hot data structure identifier according to the number of citations, the citation score can also be calculated by combining the number of citations and a preset weight, and the hot data structure identifier can be obtained by comparing the citation scores. In practical applications, it can be set according to actual needs, and this embodiment does not make any limitation here.

[0056] For example, after receiving the SQL statement submitted by the client, the data computing engine Spark can first parse the SQL statement. According to the parsing results, multiple table names {table a, table b, table c} and multiple action operators {operator 1, operator 2 and operator 3} involved in the SQL statement can be obtained; since each operator needs to use the table name to complete the operation on the data in the data table corresponding to the table name when executing, the blood relationship between each operator and each table name can be determined first. The table with a later blood relationship indicates that the calculation process is more complicated and the weight is greater; determine that operator 1 is associated with table a, operator 2 is associated with table b, operator 3 is associated with table c, and the weight of the table. On this basis, by comparing the number of times table a, table b and table c are used respectively and multiplying the weight of the table in the SQL statement, it is determined that after weighting, table b>table a>table c, so it is determined that table b has the largest number of uses. At this time, table b and operator 2 can be combined to form a Map object key-value pair; or, when the weighted value of table b, table a and table c is >=3, even if it is not the maximum value table, it needs to be cached. The cached data table can be read from the data storage space based on the key-value pair and used directly, thereby improving data processing efficiency.

[0057] In summary, by filtering the hot data structure identifier in combination with the number of references corresponding to the data structure identifier, it can be ensured that the filtered hot data structure identifier corresponds to the hot data, thereby supporting the subsequent direct use of the hot data structure from the data storage space, thereby improving data processing efficiency.

[0058] Step S206, when executing the data operation statement to the sub-operation statement carrying the hotspot data structure identifier, access the data storage space associated with the data calculation engine according to the hotspot key-value pair.

[0059] Step S208, determining the hot data structure corresponding to the hot data structure identifier according to the access result, and continuing to execute the data operation statement based on the hot data structure.

[0060] Specifically, after obtaining the hot key-value pair as described above, the data operation statement can be executed at this time. When the data operation statement is executed to the sub-operation statement carrying the hot data structure identifier, it means that the hot data structure corresponding to the hot data structure identifier will be used at this time. Since the previous data calculation engine has cached the hot data structure, it can be reused to support other statements. Therefore, when the currently executed data operation statement is executed, the data storage space associated with the data calculation engine can be accessed in combination with the hot key-value pair obtained above. After determining the hot data structure corresponding to the hot data structure identifier based on the access result, the hot data structure can be used to continue executing the data operation statement, thereby completing the data processing corresponding to the data operation statement.

[0061] Among them, the sub-operation statement specifically refers to the statement in the data operation statement that carries the hotspot data structure identifier and is currently being executed. Correspondingly, the data storage space specifically refers to the storage space used by the data computing engine to store intermediate data, which can be memory space or disk space. In actual applications, it can be set according to the computing power of the data computing engine. This embodiment does not make any restrictions here. Correspondingly, the hotspot data structure specifically refers to the hotspot data corresponding to the hotspot data structure identifier. For example, if the hotspot data structure identifier is a table name, then the hotspot data structure is the data table corresponding to the table name.

[0062] Furthermore, if the hot data structure matching the hot data structure identifier is stored in the data storage space, it can be directly read and reused. If it does not exist, the hot data can be cached in the data storage space to support reuse of other data operation statements. In this embodiment, the specific implementation method is as follows:

[0063] When it is determined based on the access result that there is a data structure matching the hotspot key-value pair in the data storage space, the data structure matching the hotspot key-value pair is read in the data storage space; the data structure matching the hotspot key-value pair is used as the hotspot data structure, and the step of continuing to execute the data operation statement based on the hotspot data structure is performed.

[0064] When it is determined based on the access result that there is no data structure matching the hotspot key-value pair in the data storage space, the hotspot data structure to be stored is determined by executing a sub-operation statement carrying the hotspot data structure identifier; the hotspot data structure to be stored is converted into a resilient distributed data set, and the resilient distributed data set is stored in the data storage space; based on the storage result, the hotspot data structure to be stored is used as a hotspot data structure, and the step of continuing to execute the data operation statement based on the hotspot data structure is performed.

[0065] Specifically, the hot data structure to be stored refers to a data structure that can be cached in the data storage space to support the execution and use of other statements. Correspondingly, the elastic distributed dataset refers to the RDD obtained after the hot data structure to be stored is converted.

[0066] Based on this, when it is determined according to the access result that there is a data structure matching the hotspot key-value pair in the data storage space, it means that the hotspot data structure has been cached before, and when the hotspot data structure is needed for the currently executed data operation statement, the data structure matching the hotspot key-value pair can be directly read in the data storage space; the data structure matching the hotspot key-value pair is used as the hotspot data structure, and the steps of continuing to execute the data operation statement based on the hotspot data structure are executed.

[0067] When it is determined based on the access result that there is no data structure matching the hotspot key-value pair in the data storage space, it means that the hotspot data structure corresponding to the data operation statement has not been cached. In order to support the use of the hotspot data structure when other statements are executed, the hotspot data structure corresponding to the hotspot data structure identifier can be cached at this stage. Therefore, the hotspot data structure to be stored can be determined by executing a sub-operation statement carrying the hotspot data structure identifier; thereafter, the hotspot data structure to be stored can be converted into an elastic distributed data set, and the elastic distributed data set can be stored in the data storage space; thereafter, the hotspot data structure to be stored can be used as a hotspot data structure according to the storage result, and the step of continuing to execute the data operation statement based on the hotspot data structure can be performed.

[0068] Continuing with the above example, after obtaining a Map object key-value pair formed by Table b and Operator 2, you can execute the SQL statement. When the statement involving the table name corresponding to Operator 2 (Table b) is executed, if the data table corresponding to Table b has been pre-cached in the disk corresponding to the data computing engine Spark, then the data table corresponding to Table b can be read from the disk based on the Map object key-value pair to continue executing the SQL statement, thereby completing operations such as adding, deleting, modifying, and querying the target data.

[0069] If the data table corresponding to table b has not been pre-cached in the disk corresponding to the data computing engine Spark, and the data table corresponding to table b is also a hot data table, in order to support other SQL statements to be directly read and used from the disk to avoid wasting time for further calculation, the data table corresponding to table b can be cached to the disk corresponding to the data computing engine Spark, so that it can be reused when other statements are executed to avoid recalculation. SQL statements can continue to be executed while caching. In this way, operations such as adding, deleting, modifying, and querying the target data can be completed.

[0070] In summary, by storing the hot data structure in the data storage space of the data computing engine, it can be ensured that when the data computing engine executes a statement, the cached data that needs to be used can be read from the data storage space, thereby avoiding repeated calculations and achieving the purpose of improving data processing efficiency.

[0071] In addition, after the execution of the data operation statement is completed, in order to avoid occupying the data storage space, the data storage space can also be released. In this embodiment, the specific implementation method is as follows:

[0072] When the data storage space reaches a preset storage threshold and the data operation statement is executed, the data storage space is released by deleting the hotspot data structure; or, the storage duration corresponding to each hotspot data structure to be applied in the data storage space is determined, and the hotspot data structure to be deleted is determined by comparing the storage duration corresponding to each hotspot data structure to be applied, and the data storage space is released by deleting the hotspot data structure to be deleted; or, the data storage space is released at preset time intervals.

[0073] Based on this, when the data operation statement is executed, considering that the storage resources of the data storage space are limited, in order to avoid occupying the space for a long time and affecting the cache of other hot data, the data storage space can be managed in different ways. On the one hand, when the data storage space reaches the preset storage threshold and the data operation statement is executed, it means that the data operation statement can no longer be used, so the data storage space can be released by deleting the hot data structure. On the second hand, the storage duration corresponding to each hot data structure to be applied in the data storage space can also be determined, and by comparing the storage duration corresponding to each hot data structure to be applied, the hot data structure with the longest duration is selected as the hot data structure to be deleted, and the data storage space is released by deleting the hot data structure to be deleted. On the third hand, the data storage space can also be released according to the preset time interval.

[0074] In actual applications, the method of releasing data storage space can be selected according to actual needs, and this embodiment does not make any limitation here.

[0075] The statement execution method provided in this embodiment is as follows: Figure 3 As shown, in order to improve the data processing efficiency when the statement is executed and reduce the professional requirements at the same time, after receiving the data operation statement submitted to the data calculation engine, the hot data structure identifier and the target operator matching the hot data structure identifier can be obtained by parsing the data operation statement. At this time, a hot key-value pair can be constructed based on the hot data structure identifier and the target operator; on this basis, when the data operation statement is executed to the sub-operation statement carrying the hot data structure identifier, it is explained that the intermediate data required to be used by the sub-operation statement is hot data, and in this scenario, in order to improve the data processing efficiency, the hot data will be stored in the data storage space, so the data storage space associated with the data calculation engine can be accessed according to the hot key-value pair; the hot data structure corresponding to the hot data structure identifier is determined according to the access result, and after obtaining the hot data structure, the data operation statement can be continued to be executed based on the hot data structure. It can be seen from this that the statement execution method provided in this embodiment can first cache the intermediate data associated with the hot data in the data storage space, so that when the intermediate data is used again, it can be directly read and used from the data storage space, thereby improving the data processing efficiency.

[0076] The following combination Figure 4 , taking the application of the statement execution method provided in this specification in the scenario where a data computing engine adds, deletes, and modifies data as an example, the statement execution method is further described. Figure 4 A processing flow chart of a statement execution method provided in one embodiment of the present specification is shown. The method is applied to a data computing engine and specifically includes the following steps.

[0077] Step S402: receiving a data operation statement submitted to the data computing engine.

[0078] Step S404: parse the data operation statement to obtain at least one data structure identifier and at least one operator.

[0079] Step S406, determining the number of references corresponding to each data structure identifier, and determining a hot data structure identifier in at least one data structure identifier by comparing the number of references corresponding to each data structure identifier.

[0080] Step S408: Select an operator having an execution relationship with the hotspot data structure identifier from at least one operator as a target operator, and construct a hotspot key-value pair based on the hotspot data structure identifier and the target operator.

[0081] Step S410, when executing a data operation statement to a sub-operation statement carrying a hotspot data structure identifier, access the data storage space associated with the data calculation engine according to the hotspot key-value pair.

[0082] Step S412: when it is determined according to the access result that there is a data structure matching the hotspot key-value pair in the data storage space, the data structure matching the hotspot key-value pair is read in the data storage space.

[0083] Step S414: Use the data structure matching the hotspot key-value pairs as the hotspot data structure, and continue to execute data operation statements based on the hotspot data structure.

[0084] Step S416, when it is determined according to the access result that there is no data structure matching the hotspot key-value pair in the data storage space, determine the hotspot data structure to be stored by executing a sub-operation statement carrying the hotspot data structure identifier;

[0085] Step S418, converting the hotspot data structure to be stored into a resilient distributed data set, and storing the resilient distributed data set into the data storage space.

[0086] Step S420: The hotspot data structure to be stored is used as the hotspot data structure according to the storage result, and the data operation statement is continued to be executed based on the hotspot data structure.

[0087] The statement execution method provided in this embodiment, in order to improve the data processing efficiency during statement execution and reduce the professional requirements at the same time, can first obtain the hot data structure identifier and the target operator matching the hot data structure identifier by parsing the data operation statement after receiving the data operation statement submitted to the data calculation engine, and at this time, a hot key-value pair can be constructed based on the hot data structure identifier and the target operator; on this basis, when the data operation statement is executed to the sub-operation statement carrying the hot data structure identifier, it is explained that the intermediate data required to be used by the sub-operation statement is the hot data, and in this scenario, in order to improve the data processing efficiency, the hot data will be stored in the data storage space, so the data storage space associated with the data calculation engine can be accessed according to the hot key-value pair; the hot data structure corresponding to the hot data structure identifier is determined according to the access result, and after obtaining the hot data structure, the data operation statement can be continued to be executed based on the hot data structure. It can be seen from this that the statement execution method provided in this embodiment can first cache the intermediate data associated with the hot data in the data storage space, so that when the intermediate data is used again, it can be directly read and used from the data storage space, thereby improving the data processing efficiency.

[0088] Corresponding to the above method embodiment, this specification also provides a statement execution device embodiment, Figure 5 FIG. 1 shows a schematic diagram of the structure of a statement execution device provided by an embodiment of the present specification. Figure 5 As shown, the device is applied to a data computing engine, comprising:

[0089] A receiving module 502 is configured to receive a data operation statement submitted to the data computing engine;

[0090] The parsing module 504 is configured to obtain a hotspot data structure identifier and a target operator matching the hotspot data structure identifier by parsing the data operation statement, and construct a hotspot key-value pair based on the hotspot data structure identifier and the target operator;

[0091] The access module 506 is configured to access the data storage space associated with the data computing engine according to the hotspot key-value pair when executing the data operation statement to the sub-operation statement carrying the hotspot data structure identifier;

[0092] The determination module 508 is configured to determine the hot data structure corresponding to the hot data structure identifier according to the access result, and continue to execute the data operation statement based on the hot data structure.

[0093] In an optional embodiment, the parsing module 504 is further configured to:

[0094] The data operation statement is parsed to obtain at least one data structure identifier and at least one operator; a hot data structure identifier is screened from the at least one data structure identifier, and a target operator matching the hot data structure identifier is screened from the at least one operator.

[0095] In an optional embodiment, the parsing module 504 is further configured to:

[0096] Determine the number of references corresponding to each data structure identifier; determine the hot data structure identifier in the at least one data structure identifier by comparing the number of references corresponding to each data structure identifier; and select an operator having an execution relationship with the hot data structure identifier as a target operator in the at least one operator.

[0097] In an optional embodiment, the determining module 508 is further configured to:

[0098] When it is determined based on the access result that there is a data structure matching the hotspot key-value pair in the data storage space, the data structure matching the hotspot key-value pair is read in the data storage space; the data structure matching the hotspot key-value pair is used as the hotspot data structure, and the step of continuing to execute the data operation statement based on the hotspot data structure is performed.

[0099] In an optional embodiment, the determining module 508 is further configured to:

[0100] When it is determined based on the access result that there is no data structure matching the hotspot key-value pair in the data storage space, the hotspot data structure to be stored is determined by executing a sub-operation statement carrying the hotspot data structure identifier; the hotspot data structure to be stored is converted into a resilient distributed data set, and the resilient distributed data set is stored in the data storage space; based on the storage result, the hotspot data structure to be stored is used as a hotspot data structure, and the step of continuing to execute the data operation statement based on the hotspot data structure is performed.

[0101] In an optional embodiment, the device further comprises:

[0102] The space release module is configured to release the data storage space by deleting the hot data structure when the data storage space reaches a preset storage threshold and the data operation statement is executed; or, determine the storage time corresponding to each hot data structure to be applied in the data storage space, compare the storage time corresponding to each hot data structure to be applied to determine the hot data structure to be deleted, and release the data storage space by deleting the hot data structure to be deleted; or, release the data storage space at preset time intervals.

[0103] In an optional embodiment, the receiving module 502 is further configured to:

[0104] Receive statement editing instructions submitted by a client node included in the data computing engine, and determine data operation statements based on the statement editing instructions; wherein, after the step of continuing to execute the data operation statement based on the hot data structure is executed, it also includes: generating data operation information based on the execution result of the data operation statement, and sending the data operation information to the client node.

[0105] The statement execution device provided in this embodiment, in order to improve the data processing efficiency during statement execution and reduce the professional requirements at the same time, can first obtain the hot data structure identifier and the target operator matched by the hot data structure identifier by parsing the data operation statement after receiving the data operation statement submitted to the data calculation engine, and at this time, a hot key-value pair can be constructed based on the hot data structure identifier and the target operator; on this basis, when the data operation statement is executed to the sub-operation statement carrying the hot data structure identifier, it is explained that the intermediate data required to be used by the sub-operation statement is the hot data, and in this scenario, in order to improve the data processing efficiency, the hot data will be stored in the data storage space, so the data storage space associated with the data calculation engine can be accessed according to the hot key-value pair; the hot data structure corresponding to the hot data structure identifier is determined according to the access result, and after obtaining the hot data structure, the data operation statement can be continued to be executed based on the hot data structure. It can be seen from this that the statement execution method provided in this embodiment can first cache the intermediate data associated with the hot data in the data storage space, so that when the intermediate data is used again, it can be directly read and used from the data storage space, thereby improving the data processing efficiency.

[0106] The above is a schematic scheme of a statement execution device of this embodiment. It should be noted that the technical scheme of the statement execution device and the technical scheme of the above statement execution method belong to the same concept, and the details not described in detail in the technical scheme of the statement execution device can be referred to the description of the technical scheme of the above statement execution method.

[0107] Figure 6 The block diagram of a computing device 600 according to an embodiment of the present specification is shown. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and the database 650 is used to store data.

[0108] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0109] In one embodiment of the present specification, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 6 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0110] The computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 600 may also be a mobile or stationary server.

[0111] The processor 620 is used to execute the following computer executable instructions, which implement the steps of the above statement execution method when executed by the processor.

[0112] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above statement execution method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above statement execution method.

[0113] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above statement execution method when executed by a processor.

[0114] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above statement execution method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above statement execution method.

[0115] An embodiment of the present specification also provides a computer program product, including a computer program or instructions, which implements the steps of the above statement execution method when executed by a processor.

[0116] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above statement execution method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above statement execution method.

[0117] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0119] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0120] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0121] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and use this specification well.

Claims

1. A statement execution method, characterized in that: Applied to data computing engines, including: receiving a data operation statement submitted to the data computing engine; Obtain at least one data structure identifier and at least one operator by parsing the data operation statement, filter a hot data structure identifier in the at least one data structure identifier according to the number of references, filter a target operator matching the hot data structure identifier in the at least one operator, and construct a hot key-value pair based on the hot data structure identifier and the target operator; When executing the data operation statement to the sub-operation statement carrying the hotspot data structure identifier, accessing the data storage space associated with the data calculation engine according to the hotspot key-value pair; The hot data structure corresponding to the hot data structure identifier is determined according to the access result, and the data operation statement is continued to be executed based on the hot data structure.

2. The statement execution method according to claim 1, characterized in that: The filtering of the hot data structure identifier in the at least one data structure identifier according to the number of references, and the filtering of the target operator matching the hot data structure identifier in the at least one operator, include: Determine the number of references corresponding to each data structure identifier; Determine a hotspot data structure identifier from the at least one data structure identifier by comparing the number of references corresponding to each data structure identifier; An operator having an execution relationship with the hotspot data structure identifier is selected from the at least one operator as a target operator.

3. The statement execution method according to claim 1, characterized in that: The step of determining the hotspot data structure corresponding to the hotspot data structure identifier according to the access result, and continuing to execute the data operation statement based on the hotspot data structure, includes: When it is determined according to the access result that there is a data structure matching the hotspot key-value pair in the data storage space, reading the data structure matching the hotspot key-value pair in the data storage space; The data structure matching the hotspot key-value pair is used as a hotspot data structure, and the step of continuing to execute the data operation statement based on the hotspot data structure is performed.

4. The statement execution method according to claim 3, characterized in that: The step of determining the hotspot data structure corresponding to the hotspot data structure identifier according to the access result, and continuing to execute the data operation statement based on the hotspot data structure, includes: When it is determined according to the access result that there is no data structure matching the hotspot key-value pair in the data storage space, determining the hotspot data structure to be stored by executing a sub-operation statement carrying the hotspot data structure identifier; Converting the hotspot data structure to be stored into a resilient distributed data set, and storing the resilient distributed data set in the data storage space; According to the storage result, the hotspot data structure to be stored is used as a hotspot data structure, and the step of continuing to execute the data operation statement based on the hotspot data structure is performed.

5. The statement execution method according to any one of claims 1 to 4, characterized in that: After the step of continuing to execute the data operation statement based on the hot data structure is executed, the method further includes: When the data storage space reaches a preset storage threshold and the data operation statement is executed, the data storage space is released by deleting the hotspot data structure; or, Determine the storage duration corresponding to each hotspot data structure to be applied in the data storage space, compare the storage duration corresponding to each hotspot data structure to be applied to determine the hotspot data structure to be deleted, and release the data storage space by deleting the hotspot data structure to be deleted; or The data storage space is released at preset time intervals.

6. The statement execution method according to any one of claims 1 to 4, characterized in that: The receiving of a data operation statement submitted to the data computing engine includes: Receiving a statement editing instruction submitted by a client node included in the data computing engine, and determining a data operation statement according to the statement editing instruction; Wherein, after the step of continuing to execute the data operation statement based on the hot data structure is executed, the method further includes: Data operation information is generated according to the execution result of the data operation statement, and the data operation information is sent to the client node.

7. A statement execution device, characterized in that: Applied to data computing engines, including: A receiving module, configured to receive a data operation statement submitted to the data computing engine; A parsing module, configured to obtain at least one data structure identifier and at least one operator by parsing the data operation statement, filter a hot data structure identifier in the at least one data structure identifier according to the number of references, filter a target operator matching the hot data structure identifier in the at least one operator, and construct a hot key-value pair based on the hot data structure identifier and the target operator; An access module, configured to access the data storage space associated with the data computing engine according to the hotspot key-value pair when executing the data operation statement to the sub-operation statement carrying the hotspot data structure identifier; The determination module is configured to determine the hot data structure corresponding to the hot data structure identifier according to the access result, and continue to execute the data operation statement based on the hot data structure.

8. A computing device, characterized in that include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Images