A code visualization method and related apparatus

Abstract

The application provides a code visualization method and a related device, and relates to the field of data processing.The method comprises the following steps: obtaining a data conversion script, the data in the data conversion script being in the form of a table, wherein the data conversion script comprises a plurality of variables, one of the plurality of variables belongs to one of a table, a row in the table and a column in the table, generating a blood relationship graph according to the relationship between the plurality of variables in the data conversion script, the blood relationship graph representing the blood relationship between the plurality of variables, and displaying the blood relationship graph on a view interface where the data conversion script is located. According to the embodiment of the application, the process relationship between the variables in the data conversion script is represented by the blood relationship graph, the blood relationship graph is displayed on the view interface where the data conversion script is located, the data analyst can analyze the data process realized by the data conversion script, and thus the data analysis is realized.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a code visualization method and related apparatus. Background Technology

[0002] In the digital age, decision-making is often data-driven, and data analysis has become a common business requirement across various industries to fully leverage its value. Data analysts typically use programming languages ​​(such as Python and R) to write data transformation scripts to accomplish data analysis tasks. During the data analysis process, data analysts must ensure that the data flow (i.e., data processing) implemented by their data transformation scripts is correct; otherwise, an incorrect data flow will lead to incorrect analytical conclusions, resulting in flawed decisions or even more serious consequences. To ensure the correctness of data transformation scripts, data analysts often need to perform data flow checks on them.

[0003] A data transformation script involves a series of data transformation operations on the original data table to obtain intermediate and final output tables. During this process, multiple data tables may be aggregated, and the same data table may derive multiple results, creating complex data flows and data chains. In practice, data analysts need to fully understand and examine the data transformation process within the data transformation script to ensure the correctness and effectiveness of the data flow implemented by the script.

[0004] For data analysts, examining data transformation processes by reviewing code is not a straightforward approach. While some code programmers use code highlighting to display different code components, this method doesn't reveal the transformation processes between data elements within the code. Therefore, it's not particularly useful or meaningful for data analysts analyzing data flow. Summary of the Invention

[0005] This application provides a code visualization method and related apparatus, which represents the flow relationship between various variables in a data transformation script through a lineage diagram, and displays the lineage diagram on the view interface where the data transformation script is located, so as to facilitate data analysts to analyze the data flow implemented by the data transformation script, thereby realizing data analysis.

[0006] Firstly, this application provides a code visualization method, comprising: obtaining a data transformation script, wherein the data in the data transformation script exists in the form of a table, the data transformation script includes multiple variables, one of which belongs to a table, a row in the table, and a column in the table; generating a lineage diagram based on the relationship between the multiple variables in the data transformation script, wherein the lineage diagram represents the lineage relationship between the multiple variables in the data transformation script; and displaying the lineage diagram on the view interface where the data transformation script is located, so that the user can analyze the data in the data transformation script based on the lineage diagram.

[0007] This application provides a code visualization method. The code implements the relationships between multiple variables, which are represented by a lineage diagram. This lineage diagram illustrates the relationships between the variables, that is, the data flow relationships (or data processing procedures) between them. By displaying the lineage diagram, code visualization is achieved. Implementing this application's embodiments, the relationships between multiple variables in a data transformation script are intuitively displayed using a lineage diagram, facilitating data analysts' analysis of the data flow implemented by the data transformation script, thereby enabling data analysis. Furthermore, compared to displaying the lineage diagram on a separate view interface, this application's solution displays the lineage diagram and the data transformation script on the same view interface, allowing data analysts to view the data transformation script and the lineage diagram side-by-side without switching between view interfaces.

[0008] Based on the first aspect, in a possible implementation, the kinship diagram is displayed on the view interface where the data conversion script is located and on the left side of the data conversion script.

[0009] Display the lineage diagram and the data transformation script on the same view interface, with the lineage diagram displayed to the left of the data transformation script. This makes it easier for data analysts to compare the data transformation script and the lineage diagram and analyze the data flow.

[0010] Based on the first aspect, in possible implementation methods, an initial lineage diagram is first generated according to the relationship between various variables in the data transformation script; then, multiple variables on the initial lineage diagram are colored to obtain a new lineage diagram, where different variables are colored differently and the same variables are colored the same.

[0011] The lineage diagram includes multiple variables from the data transformation script. Different variables in the lineage diagram are labeled with different colors, and the same variables are labeled with the same color. The colors are used to reveal the data flow relationship between multiple variables, making it easier to analyze the processing flow relationship between each variable more intuitively and clearly.

[0012] Based on the first aspect, in possible implementation methods, an initial lineage diagram is first generated according to the relationship between various variables in the data transformation script; then, multiple variables on the initial lineage diagram are colored to obtain a lineage diagram, where different variables on the lineage diagram are colored differently, and the same variables are colored the same; then, the horizontal position of at least one variable on the colored lineage diagram is adjusted to obtain a lineage diagram, where the horizontal space occupied by the lineage diagram is smaller than the horizontal space occupied by the colored lineage diagram.

[0013] The kinship diagram includes multiple variables from the data transformation script. Different variables in the kinship diagram are labeled with different colors, and the same variables are labeled with the same color, which makes it easier to analyze the processing flow relationship between the variables more intuitively and clearly. After coloring, one or more variables in the colored kinship diagram are adjusted in horizontal position to reduce the horizontal space occupied by the kinship diagram, so that the total space occupied by the kinship diagram is smaller.

[0014] Based on the first aspect, in possible implementations, coloring multiple variables on the initial lineage graph includes: establishing a tree diagram representing the relationships between multiple variables based on the initial lineage graph; for composite child nodes in the tree diagram: determining the coloring of the composite child node based on the coloring of its multiple parent nodes using a first preset algorithm; for non-composite child nodes in the tree diagram: when the number of non-composite child nodes sharing the same parent node does not exceed a preset number, determining the coloring of each non-composite child node under the parent node based on the hue space of the parent node using a second preset algorithm; when the number of non-composite child nodes sharing the same parent node exceeds a preset number, determining the coloring of each non-composite child node under the parent node using the hue drift loss function based on the hue space of the parent node; wherein, a composite child node refers to a child node having two or more parent nodes.

[0015] When coloring the initial kinship graph, a tree diagram is established based on the relationships between multiple variables. By determining the coloring of each variable / node in the tree diagram, the variables in the initial kinship graph are colored. Different algorithms are used for coloring composite child nodes and non-composite child nodes in the tree diagram. Different algorithms are also used for cases where the number of non-composite child nodes under the parent node exceeds a preset number, and cases where the number does not exceed the preset number.

[0016] Based on the first aspect, in a possible implementation, adjusting the horizontal position of at least one variable on the colored lineage graph to obtain the lineage graph includes: determining the weight of each input variable among multiple variables; arranging the branches of each input variable in the colored lineage graph sequentially according to their weights, such that the branches of input variables with larger weights are placed closer to the data conversion script, and the branches of input variables with smaller weights are placed further away from the data conversion script, while the vertical positions of each variable remain unchanged; when there are two or more variables in a horizontal position, determining the weight of each variable among the two or more variables in the horizontal position, placing the variable with larger weight closer to the data conversion script, and the variable with smaller weight further away from the data conversion script, while the vertical positions of each variable remain unchanged, to obtain a first-position lineage graph; adjusting the horizontal position of at least one variable in the first-position lineage graph to obtain the lineage graph, wherein the horizontal space occupied by the lineage graph is smaller than the horizontal space occupied by the first-position lineage graph, and the number of intersecting lines in the lineage graph is less than or equal to the number of intersecting lines in the first-position lineage graph; wherein the weight of a variable refers to the number of times the variable appears in the data conversion script.

[0017] When adjusting the horizontal position of variables, the main focus is on adjusting the horizontal position of the branch containing the input variable and the horizontal position of the intermediate and output variables obtained from the input variable. Adjustments include any one or more of the following combinations: 1) Shifting the entire branch containing the input variable horizontally; 2) Adjusting the horizontal position of the intermediate and output variables obtained from the input variable. By adjusting the horizontal position of one or more variables, the space occupied by the kinship diagram can be reduced, while simultaneously reducing or not increasing the number of intersections.

[0018] Based on the first aspect, in possible implementations, adjusting the horizontal position of at least one variable in the first position lineage graph to obtain the lineage graph includes: adjusting the horizontal position of one or more variables in the first position lineage graph, using a layout scoring function to score the changed position, adjusting the horizontal position of one or more variables again, using the layout scoring function to score the changed position again, and so on until all cases are traversed, and finally determining the lineage graph according to the layout scoring function, wherein the layout scoring function is related to the number of intersecting lines in the current layout, the position of the changed variable, and the weight of the changed variable.

[0019] Based on the first aspect, in possible implementations, the method further includes: identifying multiple variables in the data conversion script; coloring the multiple variables so that the coloring of each variable in the multiple variables is the same as the coloring of the corresponding variable in the kinship diagram; and displaying the colored multiple variables in the data conversion script.

[0020] It is understandable that coloring multiple variables in the data transformation script so that the coloring of each variable is the same as the coloring of the corresponding variable in the kinship diagram makes it easier to intuitively compare the data transformation script and the kinship diagram to analyze the data flow.

[0021] Based on the first aspect, in possible implementations, before generating the kinship diagram, the method further includes: receiving user operations to modify the data conversion script, wherein modifying the data conversion script includes any one or a combination of adding data conversion script content and deleting data conversion script content; and updating the kinship diagram according to the modified data conversion script.

[0022] During any stage of generating the kinship diagram, the user can modify the data conversion script, and the kinship diagram will be updated in real time based on the modified script. This embodiment of the application supports dynamic updates of the data conversion script; during these updates, the kinship diagram is also dynamically updated accordingly, adapting to scenarios where new code is inserted or modified.

[0023] Secondly, this application provides a code visualization device, comprising:

[0024] The acquisition module is used to acquire data transformation scripts. A data transformation script is a type of code used to manipulate data. These manipulations include one or more of the following: data cleaning, data transformation, data filtering, and data integration. The data in the data transformation script exists in the form of tables. The data transformation script includes multiple variables, one of which belongs to a table, a row in a table, or a column in a table. The data transformation script is used to manipulate the data by operating on these multiple variables.

[0025] The generation module is used to generate a lineage diagram based on the relationships between multiple variables in the data transformation script. The lineage diagram represents the lineage relationships between multiple variables in the data transformation script.

[0026] The display module is used to display the lineage diagram on the view interface where the data conversion script is located, so that users can analyze the data in the data conversion script based on the lineage diagram.

[0027] Based on the second aspect, in one possible implementation, the display module is used to display the lineage diagram to the left of the data conversion script on the view interface where the data conversion script is located.

[0028] Based on the second aspect, in one possible implementation, the generation module is used to generate an initial kinship diagram according to the data conversion script; the coloring module is used to color multiple variables on the initial kinship diagram to obtain a kinship diagram, wherein different variables on the kinship diagram are colored differently, and the same variables are colored the same.

[0029] Based on the second aspect, in one possible implementation, the generation module is used to generate an initial kinship diagram according to the data conversion script; the coloring module is used to color multiple variables on the initial kinship diagram to obtain a colored kinship diagram, wherein different variables on the colored kinship diagram are colored differently, and the same variables are colored the same; the generation module is used to adjust the horizontal position of at least one variable on the colored kinship diagram to obtain a kinship diagram, wherein the horizontal space occupied by the kinship diagram is smaller than the horizontal space occupied by the colored kinship diagram.

[0030] Based on the second aspect, in one possible implementation, the shading module is used for:

[0031] Based on the initial kinship diagram, construct a tree diagram representing the relationships between multiple variables;

[0032] For composite child nodes in a tree diagram: the coloring of the composite child node is determined based on the coloring of its multiple parent nodes and the first preset algorithm.

[0033] For non-composite child nodes in a tree diagram:

[0034] If the number of non-composite child nodes that share the same parent node with non-composite child nodes does not exceed a preset number, the coloring of each non-composite child node under the parent node is determined based on the hue space of the parent node and the second preset algorithm.

[0035] If the number of non-composite child nodes that share the same parent node with non-composite child nodes exceeds a preset number, the coloring of each non-composite child node under the parent node is determined by using the hue drift loss function based on the hue space of the parent node.

[0036] Among them, a composite child node refers to a child node that has two or more parent nodes.

[0037] Based on the second aspect, in one possible implementation, the generation module is used for:

[0038] Determine the weight of each input variable among multiple variables. Based on the weight of each input variable, arrange the branches of each input variable in the colored kinship diagram in order, so that the branches of input variables with larger weights are placed closer to the data transformation script and the branches of input variables with smaller weights are placed further away from the data transformation script. The vertical position of each variable remains unchanged.

[0039] When there are two or more variables in the horizontal position, determine the weight of each variable in the two or more variables in the horizontal position, place the variable with the larger weight closer to the data transformation script, and place the variable with the smaller weight further away from the data transformation script, while keeping the vertical position of each variable unchanged, to obtain the first position lineage diagram.

[0040] Adjust the horizontal position of at least one variable in the first position bloodline graph to obtain a bloodline graph, wherein the horizontal space occupied by the bloodline graph is smaller than the horizontal space occupied by the first position bloodline graph, and the number of intersecting lines in the bloodline graph is less than or equal to the number of intersecting lines in the first position bloodline graph; wherein, the weight of a variable refers to the number of times the variable appears in the data transformation script.

[0041] Based on the second aspect, in one possible implementation, the generation module is used to: adjust the horizontal position of one or more variables in the first position lineage graph, score the changed position using a layout scoring function, adjust the horizontal position of one or more variables again, score the changed position using a layout scoring function, and so on until all cases are traversed, and finally determine the lineage graph according to the layout scoring function, wherein the layout scoring function is related to the number of intersecting lines in the current layout, the position of the changed variable, and the weight of the changed variable.

[0042] Based on the second aspect, in one possible implementation, the coloring module is also used to determine multiple variables in the data conversion script, color the multiple variables so that the coloring of each variable in the multiple variables is the same as the coloring of the corresponding variable in the kinship diagram; the display module is also used to display the colored multiple variables in the data conversion script.

[0043] Based on the second aspect, in one possible implementation, the acquisition module is further configured to receive user operations that modify the data conversion script, wherein modifying the data conversion script includes any one or a combination of adding data conversion script content and deleting data conversion script content; the generation module is further configured to update the lineage diagram according to the modified data conversion script.

[0044] Thirdly, this application provides a computing device cluster including at least one computing device, the at least one computing device including a memory and a processor, the processor of the at least one computing device being configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the methods described in the first aspect and any possible implementation thereof.

[0045] Fourthly, this application provides a computer storage medium including program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to implement the method described in the first aspect and any possible implementation thereof.

[0046] Fifthly, this application provides a computer program product, including program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to implement the method described in the first aspect and any possible implementation thereof. Attached Figure Description

[0047] Figure 1 A flowchart illustrating a code visualization method provided in this application;

[0048] Figure 2 An example diagram provided in this application for generating an initial kinship diagram based on a data conversion script;

[0049] Figure 3 A flowchart illustrating the method for coloring multiple variables on an initial kinship diagram provided in this application;

[0050] Figure 4A An example of a tree diagram provided in this application;

[0051] Figure 4B An example diagram of tree diagram segmentation provided in this application;

[0052] Figure 4C An example diagram for coloring nodes in a tree diagram provided in this application;

[0053] Figure 4D An example diagram of coloring a bloodline diagram provided in this application;

[0054] Figure 5 An example diagram illustrating the division of a hue space provided in this application;

[0055] Figures 6A to 6C A schematic diagram of a bloodline relationship provided in this application;

[0056] Figure 7 A schematic diagram illustrating a code visualization scenario provided for this application;

[0057] Figure 8 A schematic diagram of the structure of a code visualization device provided in this application;

[0058] Figure 9 A schematic diagram of the structure of a computing device provided in this application;

[0059] Figure 10 This application provides a schematic diagram of the structure of a computing device cluster;

[0060] Figure 11 This is a schematic diagram of another computing device cluster provided in this application. Detailed Implementation

[0061] Before introducing the method embodiments, let's first introduce the technical terms involved in this application.

[0062] Data lineage refers to the natural relationship that forms between data throughout their entire lifecycle, from data generation, processing (including manipulation and fusion), transformation to their eventual demise.

[0063] In-situ code visualization is a technique that directly displays the relationships between variables in code on the view interface where the code resides, helping programmers understand the code. Code implements the relationships between various variables; code visualization refers to displaying these relationships in an intuitive and visual way. "In-situ" refers to the location of the code within the view interface. The code visualization method provided in this application implements an in-situ code visualization technique.

[0064] This application provides a code visualization method, see [link to documentation]. Figure 1 , Figure 1 This is a flowchart illustrating a code visualization method provided in this application, the method including but not limited to the following description.

[0065] S101, Obtain the data conversion script.

[0066] A data transformation script is code used to manipulate data. Data manipulation includes one or more of the following: data cleaning, data transformation, data filtering, and data integration. Data cleaning includes removing duplicate data, filling in missing data, deleting erroneous data, and data standardization. Data transformation includes data format conversion and data structure conversion; data format conversion refers to changing data from one format to another, and data structure conversion refers to changing data from one structure to another. Data filtering refers to filtering data based on specific conditions, such as selecting certain types of data or data that meet set values. Data integration includes combining multiple data sets, such as aggregating data from multiple data sources, and also includes performing calculations on multiple data sets, such as calculating a single value from multiple data sets. Data manipulation may also include other operations, which are not limited in this application.

[0067] A data transformation script is program code written by a user in a programming language (such as Python or R) to perform data analysis. For example, in the database field, operations between multiple tables are often involved. When there are many tables and a large amount of computation, it is difficult for programmers to visually determine whether the operation logic between multiple tables in the programming code is correct. In this case, a data transformation script can be written to help determine whether the operations between multiple tables in the programming code are correct.

[0068] Data transformation scripts typically include multiple variables, such as input variables, output variables, and intermediate variables, which are related to each other. For example, calculations on input variables yield intermediate variables, and calculations on intermediate variables yield output variables. It can be understood that there can be one or more input variables, one or more output variables, and one or more intermediate variables. One or more input variables can be used to obtain one or more intermediate variables, and one or more intermediate variables can be used to obtain one or more output variables.

[0069] In a data transformation script, data exists in the form of tables. One of the multiple variables included in the script can be the table itself, a row within the table, or a column within the table. The data transformation script manipulates the data by operating on these multiple variables.

[0070] S102. Generate an initial bloodline diagram based on the data conversion script.

[0071] An initial lineage graph is generated based on the data transformation script. Specifically, the data transformation script is parsed to identify its multiple variables; then, the script is parsed to determine the relationships between these variables; and finally, an initial lineage graph is generated based on these relationships, representing the lineage relationships between the variables in the data transformation script.

[0072] Optionally, multiple variables in the data conversion script can be determined using regular expression matching. It's understood that the data conversion script is written in a specific programming language, and any specific programming language has certain syntax rules. Based on the function library in the programming language's backend and the language's syntax rules, regular expression matching can be used to determine multiple variables in the data conversion script. The function library in the programming language's backend defines the meaning of different parameters for different functions. For example, in the programming language `sales = pd.read_csv()`, the syntax rules specify that `pd.read_csv` means creating a table. Therefore, this programming language represents creating the table `sale`. Regular expressions can be used to match `pd.read_csv` in the data conversion script to determine the input variable `sales`.

[0073] Variables in the data transformation script can also be determined in other ways. For example, the data transformation script is as follows:

[0074] sales = sales[…]

[0075] revenue = revenue [...]

[0076] s_sum=sales.groupby(…).sum()

[0077] r_sum=revenue.groupby(…).sum()

[0078] …

[0079] The variables sales, revenue, s_sum, and r_sum can be determined based on the "=" sign in the data conversion script described above. Each of these variables represents a table. Here, the left side of the "=" sign represents a variable; therefore, in this programming language, variables in the data conversion script can be determined using the "=" sign. This application does not limit the method used to determine variables in the data conversion script.

[0080] Optionally, the data conversion script can be parsed to determine the relationships between multiple variables. Specifically, parsing determines which variable is derived from which variable, thus establishing the input-output relationships between the variables. For example, parsing can determine that in the above data conversion script, variable s_sum is derived from variable sales, and variable r_sum is derived from variable revenue; that is, s_sum and sales are related, and r_sum and revenue are related. Other methods can also be used to determine the relationships between multiple variables, and this application does not limit this approach.

[0081] An initial kinship diagram is generated based on the relationships between multiple variables. This initial kinship diagram represents the kinship relationships between the variables. For example, such as... Figure 2 As shown, Figure 2 This is an example diagram illustrating the generation of an initial kinship diagram based on a data conversion script, as provided in this application. The data conversion script and the initial kinship diagram are shown in the right and left halves, respectively. Figure 2 In the text, a black rectangle represents a variable. The data transformation script has a total of 9 lines. The variable "sales" appears in the first, third, fifth, and seventh lines. Only one variable "sales" appears in the first, third, and fifth lines, so there is only one black rectangle in the first, third, and fifth lines to represent the variable "sales". Two variables, "sales" and "s_sum", appear in the seventh line. "s_sum" is obtained based on "sales", so there are two black rectangles in the seventh line. The black rectangle on the right represents the variable "sales", and the black rectangle on the left represents the variable "s_sum", connected by a solid line.

[0082] It's understandable that the same variable is located in the same column, such as the `sales` variable in rows 1, 3, and 5 being in the same column; different variables are located in different columns, such as `sales` and `s_sum` being in different columns here. The output variable for rows 1, 3, and 5 is `sales`. In row 7, the output variable changes; the output variable for row 7 is `s_sum`, which is calculated based on the `sales` value mentioned above. Therefore, the variables in rows 1, 3, and 5 are connected by solid lines, while the variables in rows 5 and 7 are connected by dashed lines.

[0083] The second, fourth, and sixth lines contain only one variable, `revenue`, therefore each line has a single black rectangle representing `revenue`. The eighth line contains two variables, `revenue` and `r_sum`. `r_sum` is calculated from `revenue`, so the eighth line has two black rectangles: the left one representing `revenue` and the right one representing `r_sum`, connected by a solid line. In other words, the `revenue` variables in the second, fourth, and sixth lines are in the same column, while `revenue` and `r_sum` are in different columns. The output variable in the second, fourth, and sixth lines is `revenue`. In the eighth line, the output variable changes to `r_sum`, which is calculated from `revenue`. Therefore, the variables in the second, fourth, and sixth lines are connected by a solid line, while the variables in the sixth and eighth lines are connected by a dashed line.

[0084] Line 9 contains three variables: s_sum, r_sum, and rev_sales. rev_sales is calculated from s_sum and r_sum. Therefore, line 9 has three black rectangles: the leftmost represents r_sum, the rightmost represents s_sum, and the middle represents rev_sales. These three variables are connected by solid lines. Line 7 outputs s_sum, line 8 outputs r_sum, and line 9 outputs rev_sales. Therefore, the variables s_sum and r_sum in line 7 and 9 are connected by dashed lines, as are the variables r_sum and r_sum in line 8 and 9.

[0085] In summary, based on Figure 2 The data transformation script in the right half generates the initial lineage graph in the left half. The initial lineage graph represents the lineage relationships between variables sales, revenue, s_sum, r_sum, and rev_sales, that is: variable s_sum is obtained from variable sales, variable r_sum is obtained from variable revenue, and variable rev_sales is obtained from variables s_sum and r_sum.

[0086] S103. Color the multiple variables on the initial bloodline diagram to obtain the colored bloodline diagram.

[0087] Coloring multiple variables on the initial kinship graph includes: establishing a tree diagram representing the relationships between multiple variables based on the initial kinship graph; dividing the tree diagram into multiple parts such that each part of the tree diagram does not contain composite nodes; determining the coloring of each variable in each part of the tree diagram; and coloring multiple variables on the initial kinship graph based on the coloring of each variable to obtain the colored kinship graph.

[0088] The following is combined with Figure 3 The following is a detailed description of this step in the method embodiment, which includes, but is not limited to, the description of the contents of S1031 to S1033.

[0089] S1031. Based on the initial bloodline diagram, establish a tree diagram representing the relationship between multiple variables.

[0090] Based on the relationships between multiple variables in the initial kinship diagram, a tree diagram representing the relationships between multiple variables can be constructed. For example, see... Figure 4A The example diagram shown is based on Figure 2 The initial lineage graph is used to construct a tree diagram. This tree diagram represents the relationship between the variables sales, revenue, s_sum, r_sum, and rev_sales. s_sum is obtained based on sales, so in this tree diagram, s_sum is a child node of sales (sales is the parent node of s_sum). r_sum is obtained based on revenue, so in this tree diagram, r_sum is a child node of revenue (revenue is the parent node of r_sum). rev_sales is obtained based on both s_sum and r_sum, so in this tree diagram, the rev_sales node is a child node of both s_sum and r_sum, and the rev_sales node has two parent nodes, s_sum and r_sum.

[0091] Here, the root node is assumed to be the parent node of all input variables.

[0092] S1032. Divide the tree diagram into multiple parts, such that each part of the tree diagram does not contain composite child nodes.

[0093] In this application, a composite child node refers to a child node that has two or more parent nodes. For example, Figure 4A The rev_sales node has two parent nodes, s_sum and r_sum, therefore, the rev_sales node is a composite child node.

[0094] If the tree diagram does not contain composite child nodes, no partitioning is necessary. If the tree diagram includes composite child nodes, partition the tree diagram into multiple parts such that each part of the tree diagram does not contain composite child nodes. For example, for Figure 4A The tree diagram shown is divided into two parts, as follows: Figure 4B As shown, the composite child node rev_sales is one part, and the tree diagram above the rev_sales node is another part.

[0095] It is understandable that step S1032 segments the tree diagram to separate composite child nodes, so that step S1033 can use different algorithms to color the composite and non-composite child nodes respectively. Optionally, segmentation can be omitted; it is sufficient to determine which child nodes in the tree diagram are composite and which are non-composite, and then color the composite and non-composite child nodes separately in step S1033.

[0096] S1033. Determine the hue space of each variable in each part of the tree diagram, and color the initial bloodline diagram according to the hue space of each variable to obtain the colored bloodline diagram.

[0097] Each node has its own hue space, and the hue space of child nodes is calculated based on the hue space of their parent nodes. The calculation method for the hue space differs depending on whether the child node is a composite node or a non-composite node. In this application, by default, a node is set to have a maximum of m child nodes, where m is a positive integer greater than or equal to 2. m can be specifically set by the user according to different application scenarios; for example, m can be 2, 3, or 4.

[0098] When a child node is a non-composite child node, it is determined whether the number of non-composite child nodes sharing the same parent node exceeds a preset number m. If it does not exceed the preset number m, the hue space of the child node and the hue spaces of the non-composite child nodes sharing the same parent node are determined based on the hue space of the parent node and a second preset algorithm. The second preset algorithm can be, for example, dividing the hue space of the parent node equally according to the preset number m. For instance, if m is 3, the hue space of the parent node is divided into 3 equal parts. When the parent node includes one non-composite child node, this child node uses one of these parts as its own hue space. If the parent node includes two non-composite child nodes, these two child nodes each inherit one of the three parts as their own hue space. If the parent node includes three non-composite child nodes, these three non-composite child nodes each inherit one of the three parts as their own hue space. For example, if m is 3 and the hue space of the parent node is 0-360, then the hue space of the parent node is divided into 3 parts according to the preset number m: 0-120, 120-240, and 240-360. When the parent node includes one non-composite child node, the hue space of that child node can be 0-120. When the parent node includes two non-composite child nodes, the hue spaces of these two non-composite child nodes can be 0-120 and 120-240, respectively. When the parent node includes three non-composite child nodes, the hue spaces of these three non-composite child nodes can be 0-120, 120-240, and 240-360, respectively. It should be noted that m being 3 and the hue space of the parent node being 0-360 are merely examples. In actual applications, m can be other values, and the hue space of the parent node can be specified by the user, calculated by an algorithm, or determined based on the hue space of the parent node's parent node.

[0099] It should be noted that the second preset algorithm here can also be other algorithms, and this application does not limit it.

[0100] When a child node is a non-composite child node, if it is determined that the number of non-composite child nodes with the same parent node as the child node exceeds a preset number m, then the hue space of the child node and the hue space of the non-composite child nodes with the same parent node are determined based on the hue space of the parent node and using the hue drift loss function.

[0101] It is understandable that when a child node is a non-composite child node, the default setting for the number of non-composite child nodes under its parent node is m, and the default setting is to use the second preset algorithm to determine the hue space of each non-composite child node under its parent node. When the number of non-composite child nodes under the parent node does not exceed the preset number m, the default second preset algorithm can be used to determine the hue space of each non-composite child node. However, when the number of non-composite child nodes under the parent node exceeds the preset number m, the default second preset algorithm can no longer be used to determine the hue space of each non-composite child node. Instead, a hue shift loss function needs to be used to dynamically reallocate and determine the hue space of each non-composite child node.

[0102] The hue shift loss function is as follows:

[0103] V = α·Cost(HS,HS′) + β·h

[0104] Where V represents the hue shift loss, α and β are coefficients, the values ​​of which can be set by the user, h is the hue space of the newly added variable or node, Cost(HS,HS′) is the hue shift loss of the original variable or node after the hue space is redistributed, HS is the hue space of the original variable or node before the hue space is redistributed, and HS′ is the hue space of the original variable or node after the hue space is redistributed.

[0105] Cost(HS,HS′)=∑(|h i -h ′ ′|) 1

[0106] h i h is the i-th variable in HS, that is, the hue space of the i-th original variable or original node before the hue space is redistributed. i ' is the i-th variable in HS', that is, the hue space of the i-th original variable or the original node after the hue space is redistributed.

[0107] The following example illustrates how to dynamically allocate the hue space of each variable using this hue shift loss function.

[0108] Assuming the preset quantity m is 3, when the number of non-composite child nodes under a parent node exceeds 3, the hue space of each non-composite child node under that parent node needs to be reallocated. Assuming the hue space of the parent node is 0-360, if the default second preset algorithm divides the parent node's hue space into 3 parts according to the preset quantity, such as... Figure 5The upper part of the diagram shows the corresponding hue space for each node. Node A has a hue space of 0-120, node B has a hue space of 120-240, and node C has a hue space of 240-360. This parent node now has four non-composite child nodes: A, B, C, and D, exceeding the preset number of three. Therefore, the hue spaces of the child nodes need to be redistributed. If the hue space of node A is redistributed to 30-130, node B to 130-230, node C to 230-330, and node D to 330-30, then Cost(HS,HS′) represents the sum of the hue shift losses of nodes A, B, and C after the hue space redistribution. h represents the hue space of node D, and the magnitude of the hue shift loss is calculated using the hue shift loss function. By traversing all possible cases, the magnitude of the hue shift loss V is calculated. When the hue shift loss V is minimized, the hue space of each variable is determined.

[0109] It should be noted that this explanation uses four non-composite child nodes as an example. When the parent node includes five non-composite child nodes, the hue space of the four non-composite child nodes is first determined using the above method, and then the above method is repeated to determine the hue space of the five non-composite child nodes. When the parent node includes six non-composite child nodes, the hue space of the four non-composite child nodes is first determined using the above method, and then the above method is repeated to determine the hue space of the five non-composite child nodes, and so on. That is, in the above hue drift loss function, h is the hue space of a newly added node or variable. When there are multiple newly added nodes or variables, it needs to be determined multiple times, determining the hue space of only one newly added node at a time. This example is merely an illustration and does not constitute any limitation of this application.

[0110] After determining the hue space of each non-composite child node, the hue space is a range. The median value of the hue space can be used as the color of the non-composite child node to color it.

[0111] For a composite child node, its color space is determined based on the color spaces of its multiple parent nodes using a first preset algorithm. This first preset algorithm could, for example, be the average of the colorings of the multiple parent nodes, used as the coloring of the composite child node. Figure 4C In this process, the coloring of the parent node s_sum and the coloring of the parent node r_sum are averaged to determine the coloring of the rev_sales variable. The first preset algorithm can also be other algorithms, which are not limited in this application.

[0112] Optionally, the coloring method for non-composite child nodes described in this step can be applied to... Figure 4BColor each node in the upper part of the tree diagram, and color the nodes according to the coloring method for composite child nodes. Figure 4B The nodes in the lower half (rev_sales variable) are colored.

[0113] For the root node in the tree diagram, its hue space can be determined by the user or system device. Similarly, for input variables among multiple variables, such as sales and revenue variables, their hue spaces can be determined by the user or system device. Then, based on the coloring methods for non-composite and composite child nodes provided above, the hue spaces of the other variables are determined.

[0114] After determining the hue space of each variable, or after determining the coloring of each variable, the variables on the initial kinship diagram are colored to obtain the colored kinship diagram, such as... Figure 4D As shown.

[0115] S104. Adjust the horizontal position of at least one variable on the colored kinship diagram to obtain the kinship diagram.

[0116] S105. Display the lineage diagram on the view interface where the data conversion script is located, so that users can analyze the data in the data conversion script based on the lineage diagram.

[0117] Adjust the horizontal position of at least one variable on the colored kinship diagram to obtain a new kinship diagram. The horizontal space occupied by the new kinship diagram is smaller than the horizontal space occupied by the colored kinship diagram.

[0118] Adjusting the horizontal position of at least one variable in the colored kinship graph includes: determining the weight of each input variable among multiple variables; and, according to the weight of each input variable, sequentially arranging the branches containing each input variable in the colored kinship graph so that branches containing input variables with larger weights are placed closer to the data transformation script, and branches containing input variables with smaller weights are placed further away from the data transformation script. The vertical position of each variable remains unchanged. The weight of each variable refers to the number of times each variable appears in the data transformation script. For example, for... Figure 4D For example, the input variables among multiple variables include two variables: sales and revenue. The sales variable appears more often in the data transformation script than the revenue variable, meaning the sales variable has a greater weight than the revenue variable. Therefore, the branch containing the sales variable is placed closer to the data transformation script, and the branch containing the revenue variable is placed further away from the data transformation script.

[0119] In this application, the lineage diagram is displayed on the view interface where the data conversion script is located. Specifically, the lineage diagram is displayed on the left side of the data conversion script view interface. Therefore, the branch containing the sales variable is placed on the right, and the branch containing the revenue variable is placed on the left.

[0120] Adjusting the horizontal position of at least one variable in the colored kinship diagram further includes: when there are two or more variables in a horizontal position, determining the weight of each variable among the two or more variables in the horizontal position, placing the variable with the larger weight closer to the data transformation script, and placing the variable with the smaller weight further away from the data transformation script, while keeping the vertical position of each variable unchanged, to obtain the first-position kinship diagram. For example, for Figure 4D In the seventh line, there are two variables, `sales` and `s_sum`. The data transformation script determines the frequency of occurrence of `sales` and `s_sum`, thus determining their respective weights. It is determined that the `sales` variable occurs more frequently than `s_sum`, meaning its weight is greater. Therefore, the `sales` variable is placed closer to the data transformation script, and the `s_sum` variable is placed further away, effectively placing it to the right of `s_sum`.

[0121] Similarly, in line 8, there are two variables, `revenue` and `r_sum`. Because `revenue` appears more frequently than `r_sum`, meaning `revenue` has a greater weight, we place `revenue` closer to the data transformation script and `r_sum` further away, i.e., to the right of `r_sum`. Figure 6A As shown.

[0122] The ninth line contains three variables: s_sum, r_sum, and rev_sales. The s_sum variable is a continuation of the branch containing s_sum from the seventh line, and the r_sum variable is a continuation of the branch containing s_sum from the eighth line. Therefore, the positions of these two variables are fixed. We only need to determine the relationship between the weights of rev_sales, s_sum, and r_sum. It has been determined that the weight of rev_sales is less than the weight of both s_sum and r_sum. Therefore, rev_sales is placed on the leftmost side. Figure 6A As shown.

[0123] Adjust the horizontal position of at least one variable in the obtained first-position lineage graph to obtain the final lineage graph. The final lineage graph occupies less horizontal space than the first-position lineage graph, and the number of intersecting lines in the final lineage graph is less than or equal to the number of intersecting lines in the first-position lineage graph. Specifically, adjust the horizontal position of the first variable in the first-position lineage graph, score the changed position using a layout scoring function, adjust the horizontal position of the first variable again, score the changed position using the layout scoring function, and so on, until all possible horizontal positions of all variables are traversed. Based on the layout scoring function, the final lineage graph is determined. The layout scoring function is related to the number of intersecting lines in the current layout, the position of the changed variable, and the weight of the changed variable.

[0124] The kinship diagram at the first position is adjusted to minimize the horizontal space occupied by the kinship diagram and the number of intersecting lines. To maintain the layout of the kinship diagram at the first position, the algorithm should be as lazy as possible regarding the existing layout, minimizing changes in variable positions. Therefore, this application's scheme introduces α·w into the original layout scoring function. i To minimize changes in variable positions, where α is the penalty coefficient for layout adjustments, and w i This represents the weight of variables whose positions change. Furthermore, to avoid layout redundancy and inadequate optimization due to prolonged inactivity, the algorithm should avoid remaining inactive and instead optimize the layout smoothly and gradually. To achieve this smoothness, the algorithm dynamically adjusts the penalty coefficient α. Whenever a variable's position changes, α is dynamically adjusted to decrease it, thereby increasing the probability of layout adjustments and preventing the layout from remaining inactive indefinitely.

[0125] The original layout scoring function is as follows:

[0126] B = cross - cross' + ∑(w i (p i -p i ′))

[0127] The improved layout scoring function is as follows:

[0128] B′=cross-cross′+∑(w i (p i -p′ i )-α·w i )

[0129] Where cross represents the number of intersecting lines in the kinship diagram at the first position before the variable's position changes, cross′ represents the number of intersecting lines in the kinship diagram at the first position after the variable's position changes, and p ip represents the coordinates of the variable before its position changes. i ′ represents the position coordinates of the variable after its position changes, α is the penalty coefficient for layout adjustment, and w i B represents the layout score, indicating the weight of variables whose positions change. Here, intersecting lines refer to lines connecting variables that intersect, for example, Figure 6B In the middle, the number of intersecting lines is 1. Figure 6C The number of intersecting lines is 0.

[0130] in,

[0131]

[0132] When adjusting the horizontal position of variables, the main focus is on adjusting the horizontal position of the branch containing the input variable and the horizontal position of the intermediate and output variables obtained from the input variables. Adjustments include any one or more of the following combinations: 1) Shifting the entire branch containing the `sales` variable horizontally; 2) Shifting the entire branch containing the `revenue` variable horizontally; 3) Adjusting the horizontal position of the intermediate and output variables obtained from the input variables, including the horizontal positions of `s_sum`, `r_sum`, and `rev_sales`, where `s_sum` and `r_sum` are intermediate variables, and `rev_sales` is the output variable. The number of cross lines can be reduced by inserting a variable into an empty position or swapping the positions of two variables.

[0133] This approach attempts to change the horizontal position of one or more variables. Using an improved layout scoring function, the changed layout is scored. If B is greater than 0, the change is executed, and the changed layout is retained. If B is less than or equal to 0, the change is not executed, and the layout is discarded. This process iterates through all possible adjustments to ultimately obtain a lineage diagram, such as... Figure 6C As shown.

[0134] The method in this application further includes: identifying multiple variables in the data transformation script, coloring the multiple variables so that the coloring of each variable is the same as the coloring of the corresponding variable in the lineage diagram. For example, coloring all instances of the word "sales" in the data transformation script so that "sales" is the same coloring as the "sales" variable in the lineage diagram; coloring all instances of the word "revenue" in the data transformation script so that "revenue" is the same coloring as the "revenue" variable in the lineage diagram; similarly, coloring "s_sum", "r_sum", and "rev_sales" in the data transformation script so that "s_sum", "r_sum", and "rev_sales" are the same coloring as the "s_sum", "r_sum", and "rev_sales" variables in the lineage diagram, respectively. The colored variables are then displayed in the data transformation script. Finally, the result is as follows: Figure 7 The view interface shown displays a lineage diagram on the left side of the data conversion script. The colors of the variables in the data conversion script are the same as the colors of the corresponding variables in the lineage diagram. The lineage diagram represents the lineage relationship between the variables in the data conversion script, or in other words, it represents the data flow relationship between the variables in the data conversion script.

[0135] It's worth noting that during the generation of the kinship diagram based on the data conversion script, the system can accept user modifications to the script. These modifications include adding or deleting content from the data conversion script, or a combination of both. In other words, users can modify the data conversion script at any step in the kinship diagram generation process. The kinship diagram is then updated based on the modified data conversion script.

[0136] Optionally, the column coordinates of each variable in the kinship diagram can be aligned with their column positions in the data transformation script, such as... Figure 7 As shown, if the first sales variable in the data conversion script appears in the first row, then the first sales variable in the lineage diagram is located in the first row. If the rev_sales variable in the data conversion script appears in the ninth row, then the rev_sales variable in the lineage diagram is located in the ninth row.

[0137] As can be seen, this application provides a code visualization method that displays the data flow relationships (or data processing procedures) between variables in a data transformation script through a lineage diagram, thus achieving code visualization. Implementing the embodiments of this application, the relationships between multiple variables in a data transformation script are intuitively displayed through a lineage diagram, facilitating data analysts to analyze the data flow implemented by the data transformation script and thereby achieving data analysis. Compared to displaying the lineage diagram on a separate view interface, this application's solution displays the lineage diagram and the data transformation script on the same view interface, allowing data analysts to compare the data transformation script and the lineage diagram without switching back and forth between view interfaces.

[0138] The above describes the method embodiments provided in this application. The following describes the apparatus embodiments corresponding to the method embodiments.

[0139] This application provides a code visualization device 500, such as... Figure 8 As shown, Figure 8 A schematic diagram of the structure of a code visualization device 500 provided in this application.

[0140] The device 500 includes an acquisition module 510 for acquiring a data transformation script. A data transformation script is code used to manipulate data, including one or more of data cleaning, data transformation, data filtering, and data integration. The data in the data transformation script exists in the form of a table. The data transformation script includes multiple variables, one of which belongs to a table, a row in a table, or a column in a table. The data transformation script is used to manipulate the data by operating on multiple variables. A generation module 530 is used to generate a lineage diagram based on the data transformation script. The lineage diagram represents the lineage relationships between the multiple variables. A display module 540 is used to display the lineage diagram on the view interface where the data transformation script is located, so that the user can analyze the data in the data transformation script based on the lineage diagram.

[0141] In one possible implementation, the display module 540 is used to display a lineage diagram to the left of the data conversion script on the view interface where the data conversion script is located.

[0142] In one possible implementation, the generation module 530 is used to generate an initial kinship diagram according to the data conversion script; the coloring module 520 is used to color multiple variables on the initial kinship diagram to obtain a kinship diagram, wherein different variables on the kinship diagram are colored differently, and the same variables are colored the same.

[0143] In one possible implementation, the generation module 530 is used to generate an initial lineage diagram according to the data conversion script; the coloring module 520 is used to color multiple variables on the initial lineage diagram to obtain a colored lineage diagram, wherein different variables on the colored lineage diagram are colored differently, and the same variables are colored the same; the generation module 530 is used to adjust the horizontal position of at least one variable on the colored lineage diagram to obtain a lineage diagram, wherein the horizontal space occupied by the lineage diagram is smaller than the horizontal space occupied by the colored lineage diagram.

[0144] In one possible implementation, the coloring module 520 is used for:

[0145] Based on the initial kinship diagram, construct a tree diagram representing the relationships between multiple variables;

[0146] For composite child nodes in a tree diagram: the coloring of the composite child node is determined based on the coloring of its multiple parent nodes and the first preset algorithm.

[0147] For non-composite child nodes in a tree diagram:

[0148] If the number of non-composite child nodes that share the same parent node with non-composite child nodes does not exceed a preset number, the coloring of each non-composite child node under the parent node is determined based on the hue space of the parent node and the second preset algorithm.

[0149] If the number of non-composite child nodes that share the same parent node with non-composite child nodes exceeds a preset number, the coloring of each non-composite child node under the parent node is determined by using the hue drift loss function based on the hue space of the parent node.

[0150] Among them, a composite child node refers to a child node that has two or more parent nodes.

[0151] In one possible implementation, the generation module 530 is used for:

[0152] Determine the weight of each input variable among multiple variables. Based on the weight of each input variable, arrange the branches of each input variable in the colored kinship diagram in order, so that the branches of input variables with larger weights are placed closer to the data transformation script and the branches of input variables with smaller weights are placed further away from the data transformation script. The vertical position of each variable remains unchanged.

[0153] When there are two or more variables in the horizontal position, determine the weight of each variable in the two or more variables in the horizontal position, place the variable with the larger weight closer to the data transformation script, and place the variable with the smaller weight further away from the data transformation script, while keeping the vertical position of each variable unchanged, to obtain the first position lineage diagram.

[0154] Adjust the horizontal position of at least one variable in the first position bloodline graph to obtain a bloodline graph, wherein the horizontal space occupied by the bloodline graph is smaller than the horizontal space occupied by the first position bloodline graph, and the number of intersecting lines in the bloodline graph is less than or equal to the number of intersecting lines in the first position bloodline graph; wherein, the weight of a variable refers to the number of times the variable appears in the data transformation script.

[0155] In one possible implementation, the generation module 530 is used to: adjust the horizontal position of one or more variables in the first position lineage graph, score the changed position using a layout scoring function, adjust the horizontal position of one or more variables again, score the changed position using a layout scoring function, and so on until all cases are traversed, and finally determine the lineage graph according to the layout scoring function, wherein the layout scoring function is related to the number of intersecting lines in the current layout, the position of the changed variable, and the weight of the changed variable.

[0156] In one possible implementation, the coloring module 520 is further configured to determine multiple variables in the data conversion script, color the multiple variables so that the coloring of each variable in the multiple variables is the same as the coloring of the corresponding variable in the kinship diagram; the display module 540 is further configured to display the colored multiple variables in the data conversion script.

[0157] In one possible implementation, the acquisition module 510 is further configured to receive user operations that modify the data conversion script, wherein modifying the data conversion script includes any one or a combination of adding data conversion script content and deleting data conversion script content; the generation module 530 is further configured to update the lineage diagram according to the modified data conversion script.

[0158] The acquisition module 510, coloring module 520, generation module 530, and display module 540 in device 500 can be implemented in software or in hardware. For example, the implementation of generation module 530 will be described below. Similarly, the implementation of acquisition module 510, coloring module 520, and display module 540 can refer to the implementation of generation module 530.

[0159] A module is an example of a software functional unit. The generation module 530 may include code running on compute instances, which may include, for example, compute devices, virtual machines, containers, etc. Further, a compute instance may be one or more. For example, the generation module 530 may include code running on multiple compute devices / virtual machines / containers. It should be noted that the multiple compute instances used to run the code can be distributed in the same region or in different regions. Further, the multiple compute instances used to run the code can be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple availability zones (AZs).

[0160] Similarly, multiple compute instances used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a single region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0161] A module is one example of a hardware functional unit. Generation module 530 may include at least one computing device, such as a server, virtual machine, or container. Alternatively, generation module 530 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0162] The multiple computing devices included in the generation module 530 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the generation module 530 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the generation module 530 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0163] It should be noted that, in other embodiments, the generation module 530 can be used to execute any step in a code visualization method, and the acquisition module 510, coloring module 520, and display module 540 can all be used to execute any step in a code visualization method. The steps implemented by the acquisition module 510, coloring module 520, generation module 530, and display module 540 can be specified as needed. By implementing different steps in a code visualization method through the acquisition module 510, coloring module 520, generation module 530, and display module 540, all functions of the code visualization device 500 can be realized.

[0164] This application provides a computing device 600, see [link to documentation] Figure 9 , Figure 9 This application provides a schematic diagram of the structure of a computing device 600, which can be configured as a code visualization device. The computing device 600 includes a bus 602, a processor 604, a memory 606, and a communication interface 608. The processor 604, the memory 606, and the communication interface 608 communicate via the bus 602. It should be understood that this application does not limit the number of processors and memories in the computing device 600.

[0165] Bus 602 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 9 The bus 602 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 602 may include a path for transmitting information between various components of the computing device 600 (e.g., memory 606, processor 604, communication interface 608).

[0166] Processor 604 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0167] Memory 606 may include volatile memory, such as random access memory (RAM). Processor 604 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0168] The memory 606 stores executable code, which the processor 604 executes to implement the functions of the aforementioned acquisition module 510, coloring module 520, generation module 530, and display module 540, thereby realizing a code visualization method. In other words, the memory 606 stores instructions for executing a code visualization method.

[0169] The communication interface 608 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 600 and other devices or communication networks.

[0170] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, virtual machine, or container, such as a central server, edge server, or sidecar container.

[0171] like Figure 10 As shown, Figure 10 This application provides a schematic diagram of the structure of a computing device cluster, which includes at least one computing device 600. The memory 606 of one or more computing devices 600 in the computing device cluster may store the same instructions for executing a code visualization method.

[0172] In some possible implementations, the memory 606 of one or more computing devices 600 in the computing device cluster may also store partial instructions for executing a code visualization method. In other words, a combination of one or more computing devices 600 can be used to jointly execute instructions for a code visualization method.

[0173] When at least one computing device in the computing device cluster is configured as computing device 600, the memory 606 in different computing devices 600 in the computing device cluster can store different instructions, which are used to execute some functions of computing device 600 respectively. That is, the instructions stored in the memory 606 of different computing devices 600 can implement the functions of one or more modules of the acquisition module 510, the coloring module 520, the generation module 530, and the display module 540.

[0174] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network, which can be a wide area network (WAN) or a local area network (LAN), etc. Figure 11 This illustrates the structure of yet another type of computing device cluster, such as... Figure 11 As shown, two computing devices 600A and 600B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 606 in computing device 600A stores instructions for the functions of the acquisition module 510 and the display module 540, while the memory 606 in computing device 600B stores instructions for executing the functions of the coloring module 520 and the generation module 530. Computing device 600A is used to acquire a data conversion script and send it to computing device 600B. Computing device 600A is also used to receive a lineage diagram sent by computing device 600B and display the lineage diagram on the view interface where the data conversion script is located, displaying it to the left of the data conversion script. Computing device 600B is used to generate an initial lineage diagram based on the received data conversion script, color the initial lineage diagram to obtain a colored lineage diagram, and then generate a final lineage diagram based on the colored lineage diagram.

[0175] It should be understood that Figure 8 The functions of computing device 600A shown can also be performed by multiple computing devices 600, or a cluster of computing devices can include multiple computing devices with the same functions as computing device 600A. Similarly, the functions of computing device 600B can also be performed by multiple computing devices 600, or a cluster of computing devices can include multiple computing devices with the same functions as computing device 600B.

[0176] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 10 and Figure 11The connection method of the computing device cluster differs in that the memory 606 of one or more computing devices 600 in the cluster may store different instructions for executing a code visualization method. In some possible implementations, the memory 606 of one or more computing devices 600 in the cluster may also each store a portion of the instructions for executing a code visualization method. In other words, a combination of one or more computing devices 600 can jointly execute the instructions for executing a code visualization method.

[0177] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute a code visualization method.

[0178] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device or cluster of computing devices to execute a code visualization method.

[0179] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A code visualization method, characterized in that, include: Obtain a data transformation script, which is a type of code used to manipulate data. The data manipulation includes one or more of the following: data cleaning, data transformation, data filtering, and data integration. The data in the data transformation script exists in the form of a table. The data transformation script includes multiple variables, one of which belongs to one of the following: a table, a row in a table, or a column in a table. The data transformation script is used to manipulate the data by operating on the multiple variables. A lineage diagram is generated based on the relationships between the multiple variables in the data conversion script, the lineage diagram representing the lineage relationships between the multiple variables in the data conversion script; The kinship diagram is displayed on the view interface where the data conversion script is located, so that users can analyze the data in the data conversion script based on the kinship diagram.

2. The method according to claim 1, characterized in that, Displaying the kinship diagram on the view interface where the data conversion script is located includes: The lineage diagram is displayed to the left of the data conversion script on the view interface where the data conversion script is located.

3. The method according to claim 1 or 2, characterized in that, The step of generating a bloodline diagram based on the data conversion script includes: Based on the data conversion script, an initial bloodline diagram is generated; Multiple variables on the initial kinship diagram are colored to obtain the kinship diagram. Different variables on the kinship diagram are colored differently, while the same variables are colored the same.

4. The method according to claim 1 or 2, characterized in that, The step of generating a bloodline diagram based on the data conversion script includes: Based on the data conversion script, an initial bloodline diagram is generated; Multiple variables on the initial bloodline diagram are colored to obtain a colored bloodline diagram. Different variables on the colored bloodline diagram are colored differently, while the same variables are colored the same. Adjust the horizontal position of at least one variable on the colored kinship diagram to obtain the kinship diagram, wherein the horizontal space occupied by the kinship diagram is smaller than the horizontal space occupied by the colored kinship diagram.

5. The method according to claim 3 or 4, characterized in that, The step of coloring multiple variables on the initial kinship diagram includes: Based on the initial bloodline diagram, a tree diagram representing the relationships between the multiple variables is established; For the composite child node in the tree diagram: based on the coloring of the multiple parent nodes of the composite child node, the coloring of the composite child node is determined according to the first preset algorithm; For non-composite child nodes in the tree diagram: If the number of non-composite child nodes that share the same parent node with the non-composite child node does not exceed a preset number, the coloring of each non-composite child node under the parent node is determined based on the hue space of the parent node and a second preset algorithm. If the number of non-composite child nodes that share the same parent node with the non-composite child node exceeds a preset number, the coloring of each non-composite child node under the parent node is determined by using the hue drift loss function based on the hue space of the parent node. The composite child node refers to a child node that has two or more parent nodes.

6. The method according to claim 4, characterized in that, The step of adjusting the horizontal position of at least one variable on the colored kinship diagram to obtain the kinship diagram includes: Determine the weight of each input variable among the multiple variables, and arrange the branches of each input variable in the colored kinship diagram in sequence according to the weight of each input variable, so that the branches of input variables with larger weights are placed closer to the data conversion script and the branches of input variables with smaller weights are placed further away from the data conversion script, wherein the vertical position of each variable remains unchanged. When there are two or more variables in a horizontal position, determine the weight of each variable in the two or more variables in the horizontal position, place the variable with the larger weight closer to the data conversion script, place the variable with the smaller weight further away from the data conversion script, and keep the vertical position of each variable unchanged to obtain the first position lineage diagram. The kinship diagram is obtained by adjusting the horizontal position of at least one variable in the first position kinship diagram, wherein the horizontal space occupied by the kinship diagram is smaller than the horizontal space occupied by the first position kinship diagram, and the number of intersecting lines in the kinship diagram is less than or equal to the number of intersecting lines in the first position kinship diagram. The weight of a variable refers to the number of times the variable appears in the data transformation script.

7. The method according to claim 6, characterized in that, The step of adjusting the horizontal position of at least one variable in the first position kinship diagram to obtain the kinship diagram includes: The horizontal positions of one or more variables in the first position lineage graph are adjusted, and the changed positions are scored using the layout scoring function. The horizontal positions of one or more variables are adjusted again, and the changed positions are scored again using the layout scoring function, and so on, until all cases are traversed. Based on the layout scoring function, the lineage graph is finally determined. The layout scoring function is related to the number of intersecting lines in the current layout, the position of the changed variable, and the weight of the changed variable.

8. The method according to claim 3 or 4, characterized in that, The method further includes: Determine the multiple variables in the data transformation script; The variables are colored such that the coloring of each variable is the same as the coloring of the corresponding variable in the bloodline diagram; The colored variables are displayed in the data conversion script.

9. The method according to any one of claims 1 to 8, characterized in that, Prior to generating the kinship diagram, the method further includes: The system receives user operations to modify the data conversion script, wherein modifying the data conversion script includes any one or a combination of adding data conversion script content and deleting data conversion script content; Update the bloodline diagram according to the modified data conversion script.

10. A code visualization device, characterized in that, include: The acquisition module is used to acquire a data transformation script, which is a type of code used to manipulate data. The data manipulation includes one or more of the following: data cleaning, data transformation, data filtering, and data integration. The data in the data transformation script exists in the form of a table. The data transformation script includes multiple variables, one of which belongs to one of the following: a table, a row in a table, or a column in a table. The data transformation script is used to manipulate the data by operating on the multiple variables. A generation module is used to generate a lineage diagram based on the relationships between the multiple variables in the data conversion script, wherein the lineage diagram represents the lineage relationships between the multiple variables in the data conversion script; The display module is used to display the kinship diagram on the view interface where the data conversion script is located, so that the user can analyze the data in the data conversion script based on the kinship diagram.

11. The apparatus according to claim 10, characterized in that, The display module is used to display the bloodline diagram to the left of the data conversion script on the view interface where the data conversion script is located.

12. The apparatus according to claim 10 or 11, characterized in that, The generation module is used to generate an initial bloodline diagram based on the data conversion script; A coloring module is used to color multiple variables on the initial kinship diagram to obtain the kinship diagram. Different variables on the kinship diagram are colored differently, while the same variables are colored the same.

13. The apparatus according to claim 10 or 11, characterized in that, The generation module is used to generate an initial bloodline diagram based on the data conversion script; The coloring module is used to color multiple variables on the initial bloodline diagram to obtain a colored bloodline diagram. Different variables on the colored bloodline diagram are colored differently, while the same variables are colored the same. The generation module is used to adjust the horizontal position of at least one variable on the colored bloodline diagram to obtain the bloodline diagram, wherein the horizontal space occupied by the bloodline diagram is smaller than the horizontal space occupied by the colored bloodline diagram.

14. The apparatus according to claim 12 or 13, characterized in that, The coloring module is used for: Based on the initial bloodline diagram, a tree diagram representing the relationships between the multiple variables is established; For the composite child node in the tree diagram: based on the coloring of the multiple parent nodes of the composite child node, the coloring of the composite child node is determined according to the first preset algorithm; For non-composite child nodes in the tree diagram: If the number of non-composite child nodes that share the same parent node with the non-composite child node does not exceed a preset number, the coloring of each non-composite child node under the parent node is determined based on the hue space of the parent node and a second preset algorithm. If the number of non-composite child nodes that share the same parent node with the non-composite child node exceeds a preset number, the coloring of each non-composite child node under the parent node is determined by using the hue drift loss function based on the hue space of the parent node. The composite child node refers to a child node that has two or more parent nodes.

15. The apparatus according to claim 13, characterized in that, The generation module is used for: Determine the weight of each input variable among the multiple variables, and arrange the branches of each input variable in the colored kinship diagram in sequence according to the weight of each input variable, so that the branches of input variables with larger weights are placed closer to the data conversion script and the branches of input variables with smaller weights are placed further away from the data conversion script, wherein the vertical position of each variable remains unchanged. When there are two or more variables in a horizontal position, determine the weight of each variable in the two or more variables in the horizontal position, place the variable with the larger weight closer to the data conversion script, place the variable with the smaller weight further away from the data conversion script, and keep the vertical position of each variable unchanged to obtain the first position lineage diagram. The kinship diagram is obtained by adjusting the horizontal position of at least one variable in the first position kinship diagram, wherein the horizontal space occupied by the kinship diagram is smaller than the horizontal space occupied by the first position kinship diagram, and the number of intersecting lines in the kinship diagram is less than or equal to the number of intersecting lines in the first position kinship diagram. The weight of a variable refers to the number of times the variable appears in the data transformation script.

16. The apparatus according to claim 15, characterized in that, The generation module is used for: The horizontal positions of one or more variables in the first position lineage graph are adjusted, and the changed positions are scored using the layout scoring function. The horizontal positions of one or more variables are adjusted again, and the changed positions are scored again using the layout scoring function, and so on, until all cases are traversed. Based on the layout scoring function, the lineage graph is finally determined. The layout scoring function is related to the number of intersecting lines in the current layout, the position of the changed variable, and the weight of the changed variable.

17. The apparatus according to claim 12 or 13, characterized in that, The coloring module is further configured to determine the plurality of variables in the data conversion script, and color the plurality of variables such that the coloring of each variable among the plurality of variables is the same as the coloring of the corresponding variable in the bloodline diagram; The display module is also used to display the colored variables in the data conversion script.

18. The apparatus according to any one of claims 10 to 17, characterized in that, The acquisition module is also used to receive user operations that modify the data conversion script, wherein modifying the data conversion script includes any one or a combination of adding data conversion script content and deleting data conversion script content; The generation module is also used to update the bloodline diagram according to the modified data conversion script.

19. A computing device cluster, characterized in that, The system includes at least one computing device, the at least one computing device comprising a memory and a processor, the processor of the at least one computing device being configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 9.

20. A computer storage medium, characterized in that, Includes program instructions that, when executed by a computing device cluster, cause the computing device cluster to implement the method as described in any one of claims 1 to 9.

Images