Data link analysis method in Hive script and related device

Abstract

The invention discloses a data link analysis method in a Hive script and a related device, which can be applied to the field of big data or finance, a to-be-processed target Hive script is analyzed to obtain at least one processing statement, key information in the at least one processing statement is extracted to determine a hierarchical relationship and a venation relationship in the key information, and the data link analysis method and the related device are applied to the field of big data or finance. According to the method and the device, automatic sorting of the data links in the Hive script is realized, so that the target data link tree is generated according to the hierarchical relationship and the context relationship in the key information, the sorted data links are displayed in a visual form, and data traceability and problem investigation are facilitated according to the data link tree.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data link analysis method and related devices in a Hive script. Background Art

[0002] The data link refers to the data flow path from the source of the data to the final destination. When processing and presenting data in the big data field, a large number of HIVE scripts and tables are often created for complex processing queries. This complex processing will make the data link very complicated, which is not conducive to data tracing and problem troubleshooting based on the data link. Summary of the invention

[0003] In view of the above problems, this application provides a data link analysis method and related devices in Hive scripts, which can realize automatic combing of data links in Hive scripts and display the combed data links in a visual form, so as to facilitate data tracing and problem troubleshooting according to the data link tree. The specific scheme is as follows:

[0004] The first aspect of the present application provides a data link analysis method in a Hive script, comprising:

[0005] Parse the target Hive script to be processed to obtain at least one processing statement;

[0006] Extracting key information from the at least one processing statement, the key information including a relationship between tables and fields, a relationship between tables, and a relationship between fields;

[0007] Determine the hierarchical and contextual relationships among the key information;

[0008] A target data link tree is generated according to the hierarchical relationship and contextual relationship in the key information.

[0009] In a possible implementation, determining the hierarchical relationship and contextual relationship in the key information includes:

[0010] Deduplication of the key information;

[0011] Determine the hierarchical relationship according to the relationship between the tables in the key information;

[0012] The contextual relationship is determined based on the relationship between the table and the field and the relationship between the fields in the key information.

[0013] In a possible implementation, the data link analysis method in the Hive script further includes:

[0014] Saving the correspondence between the target Hive script and the target data link tree;

[0015] Determine whether a data link tree corresponding to the target Hive script already exists in a data link forest, where the data link forest includes data link trees corresponding to multiple Hive scripts;

[0016] If a data link tree corresponding to the target Hive script already exists in the data link forest, updating the data link forest according to the target data link tree;

[0017] If the data link tree corresponding to the target Hive script does not exist in the data link forest, the target data link tree is spliced ​​into the data link forest.

[0018] In a possible implementation, the data link analysis method in the Hive script further includes:

[0019] When an error occurs during the running of the target Hive script, obtain error information;

[0020] Determine error data according to the error information;

[0021] The node corresponding to the erroneous data is marked and displayed in the data link forest.

[0022] In a possible implementation, the data link analysis method in the Hive script further includes:

[0023] Obtaining the running parameters of each Hive script, wherein the running parameters include at least the data volume and the running time;

[0024] The running parameters of each Hive script are displayed in the data link forest.

[0025] In a possible implementation, the data link analysis method in the Hive script further includes:

[0026] The data link analysis method in the Hive script also includes:

[0027] Displaying the data link forest on a visual interface;

[0028] In response to a connection operation between nodes in the data link forest, determining a hierarchical relationship and a contextual relationship between the nodes, the nodes including table nodes and field nodes;

[0029] A Hive script corresponding to the connection operation is generated according to the hierarchical relationship and context relationship between the nodes.

[0030] A second aspect of the present application provides a data link analysis device in a Hive script, comprising:

[0031] A script parsing unit, used for parsing a target Hive script to be processed to obtain at least one processing statement;

[0032] an information extraction unit, configured to extract key information from the at least one processing statement, wherein the key information includes a relationship between tables and fields, a relationship between tables, and a relationship between fields;

[0033] A relationship combing unit, used to determine the hierarchical relationship and contextual relationship in the key information;

[0034] The link tree generation unit is used to generate a target data link tree according to the hierarchical relationship and context relationship in the key information.

[0035] A third aspect of the present application provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the data link analysis method in the Hive script of the first aspect or any implementation of the first aspect.

[0036] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0037] The memory is used to store computer programs;

[0038] The processor is used to execute the computer program so that the electronic device can implement the data link analysis method in the Hive script of the above-mentioned first aspect or any implementation method of the first aspect.

[0039] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the data link analysis method in the Hive script of the above-mentioned first aspect or any implementation of the first aspect.

[0040] By means of the above technical scheme, the present application provides a method and related devices for analyzing data links in Hive scripts, which parse the target Hive script to be processed to obtain at least one processing statement, and determine the hierarchical relationship and context relationship in the key information by extracting key information in at least one processing statement, thereby realizing automatic combing of the data links in the Hive script, thereby generating a target data link tree according to the hierarchical relationship and context relationship in the key information, and displaying the combed data links in a visual form, which is convenient for data tracing and problem troubleshooting according to the data link tree. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0042] Figure 1 A system architecture diagram provided for an embodiment of the present application;

[0043] Figure 2 A schematic diagram of the structure of a terminal 100 provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of the structure of a server 200 provided in an embodiment of the present application;

[0045] Figure 4 A flow chart of a data link analysis method in a Hive script provided in an embodiment of the present application;

[0046] Figure 5 A data list corresponding to the Hive script 1 provided in the embodiment of the present application;

[0047] Figure 6 A schematic diagram of the structure of a data link analysis device in a Hive script provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In the following description, in order to ensure the accuracy of the citation and the fluency of reading, the key technical terms, abbreviations or abbreviations involved in the text are summarized and explained as follows:

[0050] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0051] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0052] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0053] See also Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 In the example, a server is included, and the server 200 can provide the method provided in the embodiment of the present application for one or more terminals.

[0054] Among them, an application can be installed on the terminal 100, and the above application can provide an interface. The terminal 100 can receive relevant parameters entered by the user on the interface, such as receiving a target Hive script to be processed, and send the above parameters to the server 200. The server 200 can obtain processing results based on the received parameters, such as obtaining a target data link tree corresponding to the target Hive script to be processed, and return the processing results to the terminal 100.

[0055] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited to this.

[0056] Next describe Figure 1 The product form of the mid-terminal 100;

[0057] The terminal 100 in the embodiment of the present application may be a mobile phone, a tablet computer, a laptop computer, an ultra mobile personal computer (UMPC), a netbook, etc., and the embodiment of the present application does not impose any limitation on this.

[0058] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0059] refer to Figure 2As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0060] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the terminal 100. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0061] Among them, the input device 132 can receive input data and the like.

[0062] The display unit 140 may be used to display information input by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file displays, and / or playback of any multimedia files. In an embodiment of the present application, the display unit 140 may be used to display Hive scripts, data link interfaces, processing results, and the like.

[0063] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0064] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0065] Among them, the memory 120 can be used to store software codes related to the data link analysis method in the Hive script, the processor 170 can execute the steps of the data link analysis method in the Hive script, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to implement corresponding functions.

[0066] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0067] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive processing results sent by the server 200.

[0068] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0069] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0070] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0071] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0072] Next describe Figure 1 The product form of the server 200;

[0073] Figure 3 A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0074] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0075] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0076] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a nonvolatile memory, such as a read only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0077] The memory 204 may be used to store software codes related to the data link analysis method in the Hive script, and the processor 202 may execute the steps of the data link analysis method in the Hive script, and may also schedule other units to implement corresponding functions.

[0078] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field programmable gate array (FPGA), general processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0079] When processing and presenting data in the big data field, it is often necessary to create a large number of scripts and tables for complex processing queries. This complex processing will lead to very complex data links, which is not conducive to data tracing and problem troubleshooting based on the data link.

[0080] One data link analysis method is full-link analysis based on metadata. However, the maintenance and updating of metadata requires certain costs and resources. If the metadata is incomplete or inaccurate, it will directly affect the results of the full-link analysis. Especially in a complex big data environment, metadata often cannot cover all data flow situations.

[0081] In order to solve the above problems, the present application embodiment provides a data link analysis method in a Hive script. The data link analysis method in a Hive script in the present application embodiment is described in detail below with reference to the accompanying drawings.

[0082] Reference Figure 4 , Figure 4 A flow chart of a data link analysis method in a Hive script provided in an embodiment of the present application is shown as follows: Figure 4 As shown, a data link analysis method in a Hive script provided in an embodiment of the present application may include steps 401 to 404, and these steps are described in detail below.

[0083] 401: Parse the target Hive script to be processed and obtain at least one processing statement;

[0084] Perform syntax analysis on the Hive script to be processed, identify various SQL statements in the Hive script to be processed, such as select, insert, update, etc., and then extract SQL statements involving data processing from the identified SQL statements, such as SQL statements involving data cleaning, data conversion, data aggregation and other operations.

[0085] Take the following Hive script 1 as an example:

[0086] insert into cTable 4

[0087]

[0088] The purpose of the above Hive script 1 is to perform specific calculations and aggregations on the data in mTable3, tTable1, and mTable2, and then insert the results into cTable4. The specific connection conditions, filtering conditions, and grouping conditions need to be determined according to the actual business logic.

[0089] That is to say, the above Hive script 1 is used to implement data insertion. By parsing the above Hive script, the SQL statement types obtained include: insert, select, join, group by, and the final extracted processing statements are SQL statements of the insert, select, and join types.

[0090] The where filter condition and group by sort condition can be ignored when combing links, and developers only need to pay attention to them when performing specific data analysis.

[0091] In addition, in some Hive scripts, the select * from table statement is set as an unavailable operation based on the readability and running accuracy of the script and does not need to be processed.

[0092] 402: extracting key information from at least one processing statement, where the key information includes a relationship between tables and fields, a relationship between tables, and a relationship between fields;

[0093] Identify keywords in at least one processing statement through lexical analysis, such as table name keywords, field keywords, input parameter identifiers, operators of basic mathematical operations, etc., and determine the relationship between tables and fields, the relationship between tables, and the relationship between fields based on the keywords.

[0094] Taking the above Hive script 1 as an example, according to the SQL statements of insert, select, and join types, we know that m table 3 is represented by a, t table 1 is represented by b, and m table 2 is represented by c. The fields of each table and the operators of basic mathematical operations are also identified. The relationship between tables and fields and the relationship between fields are extracted as follows:

[0095] mtable3.field1+ttable1.field1=mtable4.field1;

[0096] mtable3.field2+mtable2.field2=mtable4.field2;

[0097] mtable3.field3+ttable1.field3=mtable4.field3;

[0098] mtable3.field4+mtable2.field4=mtable4.field4;

[0099] sum(mtable3.field5)=mtable4.field5.

[0100] The relationship between the extracted tables is:

[0101] The fields in m table 3, t table 1, and m table 2 are added or summed and inserted into c table 4.

[0102] Left join m table 3 with t table 1;

[0103] M table 3 is left joined with m table 2.

[0104] Furthermore, in order to facilitate the storage of tables and fields, this embodiment creates a temporary table list for storing temporary table names and field names in the target Hive script, and a non-temporary table list for storing non-temporary table names and field names in the target Hive script. The list names in the temporary table list and the non-temporary table list are table names, and the list elements are field names.

[0105] 403: Determine the hierarchical and contextual relationships in key information;

[0106] In a possible implementation, determining the hierarchical relationship and contextual relationship in the key information includes the following steps 4031-4033:

[0107] 4031: Remove duplication of key information;

[0108] Since the key information extracted in the above step 402 includes the relationship between all tables and fields, the relationship between tables, and the relationship between fields in the target Hive script, there may be redundancy and duplication. For example, if the fields of a table are processed in different ways and inserted into the fields of the target table, then the relationship between the fields of this table and the fields of the target table and the relationship between this table and the target table in the extracted key information are repeated. Therefore, this embodiment first deduplicates the key information to avoid redundancy and duplication.

[0109] 4032: Determine the hierarchical relationship based on the relationship between tables in the key information;

[0110] The table hierarchy is usually designed in advance. Generally, a report will be formed after about five levels of processing. The data of the next level enters the previous level to form basic data.

[0111] Taking the above Hive script 1 as an example, according to the relationship between the extracted tables:

[0112] The fields in m table 3, t table 1, and m table 2 are added or summed and inserted into c table 4.

[0113] Left join m table 3 with t table 1;

[0114] M table 3 is left joined with m table 2.

[0115] Combined with the hierarchical relationship between pre-designed tables, it is determined that the table of the above Hive script 1 has three layers: t, m, and c.

[0116] 4033: Determine the contextual relationship based on the relationship between tables and fields and the relationship between fields in the key information.

[0117] The contextual relationship indicates a one-to-many, many-to-many, or one-to-one relationship between fields.

[0118] Taking the above Hive script 1 as an example, based on the relationship between tables and fields and the relationship between fields:

[0119] mtable3.field1+ttable1.field1=mtable4.field1;

[0120] mtable3.field2+mtable2.field2=mtable4.field2;

[0121] mtable3.field3+ttable1.field3=mtable4.field3;

[0122] mtable3.field4+mtable2.field4=mtable4.field4;

[0123] sum(mtable3.field5)=mtable4.field5.

[0124] The context relationship is determined as: m table 3. field 1 + t table 1. field 1 -> m table 4. field 1;

[0125] m table 3. field 2 + m table 2. field 2 -> m table 4. field 2;

[0126] m table 3. field 3 + t table 1. field 3 -> m table 4. field 3;

[0127] m table 3. field 4 + m table 2. field 4 -> m table 4. field 4;

[0128] m table 3. Field 5 -> m table 4. Field 5.

[0129] 404: Generate a target data link tree according to the hierarchical relationship and contextual relationship in the key information.

[0130] The target data link tree includes table nodes and field nodes. The edge between the table node and the field node represents the relationship between the table and the field, the edge between the table node and the table node represents the relationship between the tables, and the edge between the field node and the field node represents the relationship between the fields.

[0131] The data link tree can be displayed in three dimensions. Taking the above Hive script 1 as an example, the plane representation of the data link tree is as follows: Figure 5 shown.

[0132] The present embodiment discloses a method for analyzing data links in a Hive script, which parses a target Hive script to be processed to obtain at least one processing statement, and determines the hierarchical relationship and contextual relationship in the key information by extracting key information in the at least one processing statement, thereby automatically combing the data links in the Hive script, thereby generating a target data link tree according to the hierarchical relationship and contextual relationship in the key information, and displaying the combed data links in a visual form, which is convenient for data tracing and problem troubleshooting according to the data link tree.

[0133] Furthermore, after the target data link tree is generated, the target data link tree may be integrated into an existing data link forest. The data link forest includes data link trees corresponding to multiple Hive scripts. In a possible implementation, the following steps A1-A4 are included:

[0134] A1: Save the correspondence between the target Hive script and the target data link tree;

[0135] A2: Determine whether the data link tree corresponding to the target Hive script already exists in the data link forest;

[0136] A3: If the data link tree corresponding to the target Hive script already exists in the data link forest, update the data link forest corresponding to the target Hive script in the data link forest according to the target data link tree;

[0137] For example: if an existing target Hive script is modified, the data link forest already has a data link path corresponding to the target Hive script before the modification. After the modified target Hive script is analyzed, the target data link tree is obtained. The target Hive scripts before and after the modification correspond to the same Hive script identifier. In this case, the data link forest corresponding to the target Hive script in the data link forest needs to be updated according to the target data link tree.

[0138] After the Hive script is modified, the hierarchical relationship between tables generally does not change, but the data links between tables, that is, the relationship between fields, will be added or deleted as the script is modified.

[0139] A4: If the data link tree corresponding to the target Hive script does not exist in the data link forest, the target data link tree is spliced ​​into the data link forest.

[0140] For example: if a new target Hive script is added, the data link tree corresponding to the target Hive script does not exist in the data link forest. The target data link tree needs to be spliced ​​into the data link forest, and new nodes and / or connection relationships between new nodes are added according to the target data link tree.

[0141] It should also be noted that if the target Hive script is deleted, the data link tree corresponding to the target Hive script needs to be deleted in the data link forest to ensure the consistency between the data link forest and the Hive script.

[0142] Based on the data link forest, a variety of functions can be provided to improve the efficiency of data tracing and problem troubleshooting.

[0143] In a possible implementation, when an error occurs during the running of a target Hive script, the error information is obtained by reading the system log or system prompt information, and then the error data is determined by parsing the error information, such as data type mismatch, data format error, and other data corresponding to the error cause. Then, the node corresponding to the error data is marked and displayed in the data link forest, such as highlighting or changing color. The error cause can also be displayed next to the node corresponding to the error data, so that development and operation personnel can intuitively locate the error data and solve the problem in a timely manner.

[0144] In a possible implementation, the running parameters of each Hive script are obtained, where the running parameters at least include the data volume and the running time, and then the running parameters of each Hive script are displayed in the data link forest.

[0145] Exemplarily, in response to the operation of displaying the running parameters, the real-time running parameters of each Hive script are obtained, and the real-time running parameters of each Hive script are displayed in the data link forest.

[0146] Exemplarily, the average value of the running parameters of each Hive script in a period is periodically obtained, and the average value of the running parameters of each Hive script is displayed in the data link forest.

[0147] Furthermore, the operating parameters can be divided into multiple level ranges in advance. The data link trees of Hive scripts corresponding to different level ranges use different colors, and the data link trees of Hive scripts corresponding to the same level range use the same color. This makes it easier for development and operation personnel to intuitively understand the system pressure and memory usage of Hive scripts, and prompts development and operation personnel which scripts may have Cartesian product and other failures.

[0148] In a possible implementation, a data link forest is displayed in a visual interface, and in response to the connection operation between nodes in the data link forest, the hierarchical relationship and contextual relationship between nodes are determined, wherein the connection operation between nodes in the data link forest is used to add connections between new nodes, that is, to add relationships between new nodes, and the hierarchical relationship and contextual relationship between the new nodes are determined based on the relationships between the new nodes, and the nodes include table nodes and field nodes, and the Hive script corresponding to the connection operation is generated based on the hierarchical relationship and contextual relationship between the new nodes. It can be understood that this process is the reverse processing process of the data link analysis in the Hive script.

[0149] Since the hierarchical relationship between tables, the relationship between tables and fields, and the contextual relationship between fields can all be expressed using SQL statements, it is possible to generate Hive scripts corresponding to the connection operation based on the hierarchical relationship and contextual relationship between nodes. Developers can quickly develop Hive scripts by connecting nodes in a visual interface, effectively improving the development efficiency of Hive scripts.

[0150] The above introduces a data link analysis method in a Hive script provided in an embodiment of the present application. The following will introduce a device for executing the data link analysis method in the above Hive script.

[0151] See also Figure 6 , Figure 6 A schematic diagram of the structure of a data link analysis device in a Hive script provided in an embodiment of the present application. Figure 6 As shown, the data link analysis device in the Hive script includes:

[0152] The script parsing unit 601 is used to parse the target Hive script to be processed to obtain at least one processing statement; the information extraction unit 602 is used to extract key information from the at least one processing statement, wherein the key information includes the relationship between tables and fields, the relationship between tables and the relationship between fields;

[0153] A relationship combing unit 603, used to determine the hierarchical relationship and contextual relationship in the key information;

[0154] The link tree generation unit 604 is used to generate a target data link tree according to the hierarchical relationship and context relationship in the key information.

[0155] In a possible implementation, the relationship combing unit 603 is specifically used to deduplicate the key information; determine the hierarchical relationship based on the relationship between tables in the key information; and determine the contextual relationship based on the relationship between tables and fields and the relationship between fields in the key information.

[0156] In a possible implementation, the data link analysis device in the Hive script further includes:

[0157] A storage unit, used to store the corresponding relationship between the target Hive script and the target data link tree;

[0158] A judging unit, used to judge whether a data link tree corresponding to the target Hive script already exists in a data link forest, wherein the data link forest includes data link trees corresponding to multiple Hive scripts;

[0159] An updating unit, configured to update the data link forest according to the target data link tree if a data link tree corresponding to the target Hive script already exists in the data link forest;

[0160] A splicing unit is used to splice the target data link tree into the data link forest if the data link tree corresponding to the target Hive script does not exist in the data link forest.

[0161] In a possible implementation, the data link analysis device in the Hive script further includes:

[0162] An error information acquisition unit, used to acquire error information when an error is reported during the execution of the target Hive script;

[0163] An error locating unit, used to determine error data according to the error reporting information;

[0164] The first display unit is used to mark and display the node corresponding to the erroneous data in the data link forest.

[0165] In a possible implementation, the data link analysis device in the Hive script further includes:

[0166] An operation parameter acquisition unit, used to acquire the operation parameters of each Hive script, wherein the operation parameters at least include data volume and operation time;

[0167] The second display unit is used to display the running parameters of each Hive script in the data link forest.

[0168] In a possible implementation, the data link analysis device in the Hive script further includes:

[0169] A third display unit, used for displaying the data link forest on a visual interface;

[0170] A connection operation response unit, configured to determine the hierarchical relationship and contextual relationship between the nodes in the data link forest in response to a connection operation between the nodes, the nodes including table nodes and field nodes;

[0171] The script generation unit is used to generate a Hive script corresponding to the connection operation according to the hierarchical relationship and context relationship between the nodes.

[0172] The present embodiment discloses a data link analysis device in a Hive script, which parses a target Hive script to be processed to obtain at least one processing statement, and determines the hierarchical relationship and context relationship in the key information by extracting key information in the at least one processing statement, thereby realizing automatic combing of the data links in the Hive script, thereby generating a target data link tree according to the hierarchical relationship and context relationship in the key information, and displaying the combed data links in a visual form, which is convenient for data tracing and problem troubleshooting according to the data link tree.

[0173] The present application also provides an electronic device in an embodiment. Figure 7 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include but is not limited to fixed terminals such as mobile phones, notebook computers, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0174] like Figure 7As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0175] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a memory card, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0176] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the data link analysis method in any Hive script provided in the embodiment of the present application.

[0177] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the data link analysis method in any Hive script provided in the embodiment of the present application.

[0178] The data link analysis method and related devices in a Hive script provided by the present application can be used in the field of big data or finance. The above is only an example and does not limit the application field of the data link analysis method and related devices in a Hive script provided by the present invention.

[0179] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0180] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0181] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0182] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A data link analysis method in a Hive script, characterized in that: include: Parse the target Hive script to be processed to obtain at least one processing statement; Extracting key information from the at least one processing statement, the key information including a relationship between tables and fields, a relationship between tables, and a relationship between fields; Determine the hierarchical and contextual relationships among the key information; A target data link tree is generated according to the hierarchical relationship and contextual relationship in the key information.

2. The data link analysis method in Hive script according to claim 1, characterized in that: Determining the hierarchical relationship and contextual relationship in the key information includes: Deduplication of the key information; Determine the hierarchical relationship according to the relationship between the tables in the key information; The contextual relationship is determined based on the relationship between the table and the field and the relationship between the fields in the key information.

3. The data link analysis method in Hive script according to claim 1, characterized in that: The data link analysis method in the Hive script also includes: Saving the correspondence between the target Hive script and the target data link tree; Determine whether a data link tree corresponding to the target Hive script already exists in a data link forest, where the data link forest includes data link trees corresponding to multiple Hive scripts; If a data link tree corresponding to the target Hive script already exists in the data link forest, updating the data link forest according to the target data link tree; If the data link tree corresponding to the target Hive script does not exist in the data link forest, the target data link tree is spliced ​​into the data link forest.

4. The data link analysis method in Hive script according to claim 3, characterized in that: The data link analysis method in the Hive script also includes: When an error occurs during the running of the target Hive script, obtain error information; Determine error data according to the error information; The node corresponding to the erroneous data is marked and displayed in the data link forest.

5. The data link analysis method in Hive script according to claim 3, characterized in that: The data link analysis method in the Hive script also includes: Obtaining the running parameters of each Hive script, wherein the running parameters include at least the data volume and the running time; The running parameters of each Hive script are displayed in the data link forest.

6. The data link analysis method in Hive script according to claim 3, characterized in that: The data link analysis method in the Hive script also includes: Displaying the data link forest on a visual interface; In response to a connection operation between nodes in the data link forest, determining a hierarchical relationship and a contextual relationship between the nodes, the nodes including table nodes and field nodes; A Hive script corresponding to the connection operation is generated according to the hierarchical relationship and context relationship between the nodes.

7. A data link analysis device in a Hive script, characterized in that: include: A script parsing unit, used for parsing a target Hive script to be processed to obtain at least one processing statement; an information extraction unit, configured to extract key information from the at least one processing statement, wherein the key information includes a relationship between tables and fields, a relationship between tables, and a relationship between fields; A relationship combing unit, used to determine the hierarchical relationship and contextual relationship in the key information; The link tree generation unit is used to generate a target data link tree according to the hierarchical relationship and context relationship in the key information.

8. A computer program product, characterized in that It includes computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the data link analysis method in the Hive script as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the data link analysis method in the Hive script as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the data link analysis method in the Hive script as described in any one of claims 1 to 6.

Images