Method, apparatus and device for semiconductor hierarchical data merging based on merge map structure

Abstract

The application provides a semiconductor hierarchical data merging method, device and equipment based on a MergeMap structure, belongs to the field of semiconductor intelligent manufacturing, and specifically comprises the following steps: acquiring first row data of a first table from a source end database; storing a first node by processing the first row data through a configured expression; filtering and judging the first row data according to a configured filtering condition until all row data of the first table is stored in a temporary node set as the first node; judging whether a corresponding second table exists in the source end database; finding a second associated row in the second table according to a first key field, and storing the second associated row into the temporary node set as a second node; merging the first node and the second node in the temporary node set to construct a MergeMap structure; and generating a target mapping table based on the MergeMap structure. Through the processing scheme, the space utilization rate and the data merging efficiency are improved.

Description

Technical Field

[0001] This application relates to the field of semiconductor intelligent manufacturing, and in particular to a method, apparatus and computer equipment for semiconductor hierarchical data merging based on the MergeMap structure. Background Technology

[0002] Semiconductor manufacturing plants contain numerous tables, which are sometimes interconnected, and data from these tables needs to be aggregated for analysis. For example, the source database might have a user table (Table 1.1 below, which stores user information) and an order table (Table 1.2 below, which stores order messages).

[0003] User Table 1.1

[0004]

[0005] Order Form 1.2

[0006]

[0007] When a server needs to calculate the order volume for each user for data analysis, a common approach is to join two tables, first calculating the Cartesian product (Table 1.3), then filtering out the corresponding rows of data based on the user ID, ultimately returning three rows of data (Table 1.4), and then aggregating these three rows to calculate the order volume for each user. As can be seen from Table 1.3, the result of the Cartesian product of the two tables is the product of the two tables themselves. When there are many joined tables in the database, the required computational memory increases exponentially, and it is also not conducive to the rapid merging of tables.

[0008] Table 1.3 Cartesian product of the user table and the order table

[0009]

[0010] Table 1.4 shows that the Cartesian product of the user table and the order table, plus the user ID, are equal.

[0011]

[0012] Alternatively, the server can search the order table using the user ID from the user table, find the related data, and then merge the two sets of data into a single row using a HashMap. This approach avoids the problem of excessive data volume caused by performing a Cartesian product of the two tables. However, as shown in Table 1.4, the server needs to create an additional table to record the data from the two tables for merging. Therefore, when two orders are found based on user ID 2, the server will generate a new table with two rows of data, requiring the copying of all fields and making it difficult to trace the data source. However, semiconductor manufacturing plants handle extremely large orders, and this would consume a significant amount of memory, hindering subsequent data analysis. Summary of the Invention

[0013] Therefore, in order to overcome the shortcomings of the prior art, the present invention provides a method, apparatus and computer device for semiconductor hierarchical data merging based on the MergeMap structure. This method does not create a new table for merging data, reduces data memory space and thus improves space utilization.

[0014] To achieve the above objectives, this invention provides a method for merging semiconductor-level data based on a MergeMap structure, comprising: S1, obtaining the first row of data from a first table in a source database; S2, processing the first row of data using a configured expression, and storing the processed first row of data as the first node of a MergeMap structure; S3, filtering and judging the first row of data according to configured filtering conditions to determine whether the first row of data needs to be split into columns and rows; when it is determined that it does not need to be split, obtaining the next row of data and repeating step S2 until all rows of data in the first table have undergone the filtering and judgment, and storing all rows of data in a temporary node set as... S4. Based on the association condition referencing the primary key value in the first table, determine whether there is a corresponding second table in the source database; S5. According to the first key field carried by the association condition, find the second related row in the second table, and store each second related row as a second node associated with the first node in the temporary node set according to a preset method; S6. When it is determined that there is no second table to be merged in the source database, merge the first node and the second node in the temporary node set to construct a MergeMap structure; S7. Generate a target mapping table corresponding to the semiconductor level data based on the MergeMap structure.

[0015] In one embodiment, between S5 and S6, the method further includes: S8, determining the corresponding third table in the source database based on the second foreign key referencing the primary key value in the second table; S9, searching for the third related row in the third table according to the second key field carried by the second foreign key, and storing each third related row as a third node associated with the second node in a temporary node set according to a preset method; S10, repeating steps S8 to S9 until the primary key values ​​of each table no longer have a reference relationship, and proceeding to step S6.

[0016] In one embodiment, step S3 further includes: when it is determined that a specific column of the first row of data stores multiple parallel values, determining that the first row of data needs to be split into columns; creating a new data row for each split parallel value, and storing the data row in a temporary node set corresponding to the first node.

[0017] In one embodiment, storing each second associated row as a second node associated with the first node in a temporary node set according to a preset method includes: setting the first node as a parent node and the second node as a child node; merging each second associated row with the second node associated with the first node according to a preset method, and storing the merged row data and the child node in a temporary node set.

[0018] In one embodiment, storing each second associated row as a second node associated with the first node in a temporary node set according to a preset method includes: obtaining the column categories corresponding to the row data of the second associated row and the row data of the first node; when the second associated row has the same column category as the first node, overwriting the row data mapping of the first node with the row data of the second associated row corresponding to that column category; when the second associated row does not have the same column category as the first node, merging the row data of the second associated row with that column category into the row data mapping of the first node.

[0019] In one embodiment, S7 further includes: aggregating and grouping multiple rows of data in the temporary node set to obtain grouped row data; sorting the aggregated grouped row data using a custom function, and generating a target mapping table corresponding to the semiconductor level data based on the MergeMap structure.

[0020] A method for searching semiconductor-level data, characterized in that the semiconductor-level data is merged using the above-described method, comprising: obtaining a preset method in a target mapping table, determining a merging method for a first node and a second node; determining the order of data queries based on the merging method; sequentially searching for the data to be searched from the first node and the second node according to the order, and outputting the storage location corresponding to the data to be searched.

[0021] A semiconductor-level data merging apparatus based on a MergeMap structure, the apparatus comprising: a data acquisition module for acquiring the first row of data from a source database; an expression execution module for processing the first row of data using a configured expression and storing the processed first row of data as a first node of a MergeMap structure; a filtering module for filtering the first row of data according to configured filtering conditions to determine whether the first row of data needs to be split into columns and rows; when it is determined that it does not need to be split, acquiring the next row of data and repeating step S2 until all rows of data in the first table have undergone the filtering judgment, and storing all rows of data as first nodes in a temporary node set; and a connection module for... The system uses the primary key value association condition in the first table to determine whether a corresponding second table exists in the source database; the mapping module is used to find the second associated row in the second table according to the first key field carried by the association condition when it is determined that the source database has a second table that needs to be merged, and to store each second associated row as a second node associated with the first node in a temporary node set according to a preset method; the merging module is used to merge the first node and the second node in the temporary node set to construct a MergeMap structure when it is determined that the source database does not have a second table that needs to be merged; and the target mapping table generation module is used to generate a target mapping table corresponding to the semiconductor level data based on the MergeMap structure.

[0022] A computer device includes a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program to implement the steps of the above-described method.

[0023] A computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the above-described method.

[0024] Compared with the prior art, the advantages of the present invention are as follows: by analyzing the data of the tables, the mapping of related data between different tables is determined. Without creating an additional set to merge the data retrieved from the two tables, the one-way access to and retrieval of table data is achieved layer by layer through the mapping relationship between the tables. The mapping table only stores the mapping between the data of different tables and does not store the table data itself, thus saving unnecessary space, which not only improves space utilization but also improves the efficiency of data merging. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating the semiconductor-level data merging method based on the MergeMap structure in an embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram of the MergeMap structure in an embodiment of the present invention;

[0028] Figure 3 This is a structural block diagram of a semiconductor hierarchical data merging device based on the MergeMap structure in one embodiment;

[0029] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0030] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0031] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] It should be noted that the following description covers various aspects of embodiments within the scope of protection of this invention. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this application, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0033] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0034] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0035] This application provides a method for semiconductor hierarchical data merging based on the MergeMap structure, which can be applied to servers or terminals. Terminals can be, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable smart devices. Servers can be implemented using independent servers or server clusters composed of multiple servers.

[0036] In one embodiment, such as Figure 1 As shown, a method for semiconductor-level data merging based on the MergeMap structure is provided. Taking the application of this method to a server as an example, the method includes the following steps:

[0037] S1 retrieves the first row of data from the first table in the source database.

[0038] The first table can be the main table or header table of the source database, or it can be a separate main table containing independent content. The main table of the source database stores core data and is linked to other tables via foreign keys. It typically contains a primary key that uniquely identifies each row and can be linked to other tables via foreign keys. The data in the main table of the source database is usually independent, complete, and forms the basis for the data in other tables. For example, the main table of the source database could be a factory number table, and its sub-tables could be order system tables, payment system tables, after-sales system tables, etc. A separate main table containing independent content stores only the core data specific to that table. For example, the independent content in a main table could be order system content. In an order system, the Customer table can be the main table, and the Order table can be a sub-table. The Order table references the primary key of the Customer table via the CustomerID foreign key.

[0039] Semiconductor-level data can be stored data of semiconductor products or production data from the manufacturing process. For example, when the semiconductor-level data is stored data of semiconductor products, this stored data can be organized into a standardized hierarchical structure (specifically, which components exist on the semiconductor product, which equipment the semiconductor product can be installed on, and which production line the equipment can be on). This facilitates supply chain analysis and revenue aggregation. Each hierarchical node has a clear parent-child relationship, forming a tree structure, thereby supporting subsequent attribution calculations, industry linkage analysis, etc. When the semiconductor-level data is production data from the manufacturing process, this semiconductor-level data can be a data chain composed of processing data from each process step the semiconductor product undergoes. Based on this processing data, it is convenient to identify the product's production path and determine the product's quality.

[0040] The MergeMap structure is used to represent a composite structure of mapping and merging, indicating that multiple different rows of data are merged and mapped to the same row of data.

[0041] The server can start processing from the main table of the source database; the server can also process multiple independent main tables in the main table at the same time, and then summarize the multiple independent main tables through the main table of the source database.

[0042] The server retrieves the first row of data from the first table in the source database. The first row can be any row in the first table and can be used as the base row.

[0043] S2 processes the first row of data using the configured expression and stores the processed first row of data as the first node of the MergeMap structure.

[0044] The server processes the first row of data (BaseRow) using a configured expression and stores the processed data as the first node of the MergeMap structure. The first node can be the initial node (root node) of the MergeMap structure. The server can also set the first node to have no associated parent node; for example, the server can set the parent of the first node to null.

[0045] The server uses a configured expression to perform calculations or transformations on the first row of data. The server stores the expression result in the first node of the MergeMap (this may overwrite existing fields in the row data or add new fields). When the source data cannot meet the preset requirements, the server uses the configured expression to perform calculations on the first row of data. For example, the database stores time data in the format "2020-01-01 12:12:12", but this time corresponds to the UTC time zone. When converting to the GMT time zone, 8 hours need to be added to the source data. The server needs to perform a +8-hour calculation on the value of this field. Specifically, the expression could be toTimeStamp(${time})+8*60*60*1000. The server first replaces ${time} with a specific value, where time is the field name (i.e., the value of the time column), then converts it to a timestamp using the toTimeStamp method, and then adds the milliseconds corresponding to 8 hours (8*60*60*1000 milliseconds).

[0046] When the same field name exists in different tables, the server can set an alias for that field name. For example, if both the user table and the order table have a field named "id", the server will set an alias for the "id" in both tables. Then, the server will convert "id" to this alias, and subsequently, the server can retrieve the value of this field from the user table or the order table using the alias. S3: Based on the configured filtering conditions, the first row of data is filtered to determine whether it needs to be split into columns. If it is determined that it does not need to be split, the next row of data is retrieved, and step S2 is repeated until all rows of data in the first table have been filtered. All rows of data are then stored as the first node in a temporary node set.

[0047] Filtering conditions are conditions set according to business needs to filter row data. For example, if the server needs to filter data where user id = 1, but the table stores 100 users with unique ids, the server would configure the filter condition to id = 1, excluding data where user id = 1. During a query, the server will then filter out data where id is not 1.

[0048] The server filters the first row of data according to the configured filtering conditions to determine whether the first row of data needs to be split into columns and rows. If it is determined that it does not need to be split, the next row of data is obtained and the S2 step is repeated until all rows of data in the first table have been filtered and judged, and all rows of data are stored as the first node in a temporary node set.

[0049] The server receives the MergeMap node information output from step S2 and filters the first row of data. If the filtering fails, it jumps to the next row of data in the first table (starting from re-acquiring a new BaseRow) until all rows of data in the first table have been filtered and judged, and all rows of data are stored as the first node in a temporary node set.

[0050] S4. Based on the association conditions referencing the primary key values ​​in the first table, determine whether a corresponding second table exists in the source database.

[0051] The server determines the corresponding second table in the source database based on the association conditions referencing the primary key values ​​in the first table. Association conditions can be foreign keys, table fields, or references. The server can search for rows in the next table based on the association conditions (such as foreign keys). This module determines whether there is another table to process. If a next table exists, it proceeds to step S5 to find the associated rows (potentially multiple rows) in that table using the association fields, creates a new MergeMap for each associated row (setting the parent to the first row's data), and then recursively performs subsequent operations (such as executing expressions, filtering, etc.). If no next table exists, it proceeds directly to step S6. The second table represents all tables in the source database that need to be merged with the first table.

[0052] S5, when it is determined that there is a second table in the source database that needs to be merged, the second related row in the second table is found according to the first key field carried by the association condition, and each second related row is stored in the temporary node set as a second node mapping associated with the first node according to the preset method.

[0053] When the source database is determined to have a second table that needs to be merged, the server searches for the second related row in the second table based on the first key field carried by the association condition. Then, according to a preset method, each second related row is mapped as a second node associated with the first node and stored in a temporary node set. For example, if the first key field of the association condition is "User A", the server associates "User A" with 3 order rows (the second related row), sets these 3 order rows as the second node, and sets the parent node of these 3 nodes to the first node containing "User A". If the server is configured to associate a third table, it uses these three nodes to find the data in the third table using the association relationship, just like the operation for the first node. When the source database stores multiple tables, the server performs an iterative process until it finally obtains a leaf node. This leaf node can find all the data along this path through the parent node information stored in the node.

[0054] After obtaining the leaf node, the server will write the leaf node into the temporary node set. After processing all the data in the first table and obtaining all the relevant leaf nodes, a set of all leaf nodes (temporary node set) will be obtained. These leaf nodes will undergo some processing and finally generate the target table.

[0055] The second related row is the row data in the second table that is related to the row data in the first table. There may be multiple second related rows in the second table. The server generates a new MergeMap node for each second related row, executes the configured expression on each second related row, and stores the result of the processed second related row as the second node of the MergeMap structure. The server can set the parent of the second node to the first row data. S6, when it is determined that there is no second table that needs to be merged in the source database, the first node and the second node in the temporary node set are merged to construct the MergeMap structure.

[0056] When it is determined that the source database does not contain a second table that needs to be merged, the server uses the MergeMap feature to merge the data, combining the first and second nodes in the temporary node set to construct the MergeMap structure. When the Join module finds the related row, it merges the related row with the current row (i.e., forming a hierarchical mapping through the parent pointer). This step is implicit and guaranteed by the MergeMap structure. The merged data is the complete row after mapping. For example, merging the data from User Table 1.1 and Order Table 1.2 results in a mergeMap storing the Cartesian product of the two tables. The arrows in this mapping table are pointers to the parent node (pointers are one way to display mapping relationships; mapping relationships can also be displayed in other ways, such as text or strings). Therefore, Table 1.5 only establishes a reference relationship between the Order Table and the User Table, without creating an additional set to merge the data retrieved from the two tables. Even when performing a data query, the server returns the data from Order Table 1.2, and then accesses the data in User Table 1.1 through the pointer from Order Table 1.2 to User Table 1.1. The entire table requires no additional space beyond the references, and there is no need to merge two rows of data.

[0057] Table 1.5 shows the mergeMap, which stores the Cartesian product of two tables.

[0058]

[0059] S7 generates a target mapping table corresponding to the semiconductor level data based on the MergeMap structure.

[0060] The server generates a target mapping table corresponding to the semiconductor-level data based on the MergeMap structure.

[0061] The speed of this method (hereinafter referred to as MergeMap) is compared with that of the regular HashMap operation merging method (hereinafter referred to as HashMap), and the following performance data is obtained:

[0062] = Test Scenario 1: Basic Merging Performance =

[0063] >>Test size: 1000 elements<<

[0064] HashMap average merge time: 92,840 ns (92.84 μs)

[0065] MergeMap average merge time: 90 ns (0.09 μs)

[0066] Performance difference: MergeMap is 1031 times faster than HashMap.

[0067] >>Test size: 10000 elements<<

[0068] HashMap average merge time: 337,030 ns (337.03 μs)

[0069] MergeMap average merge time: 240 ns (0.24 μs)

[0070] Performance difference: MergeMap is 1404 times faster than HashMap.

[0071] >>Test size: 100,000 elements<<

[0072] HashMap average merge time: 2,620,210 ns (2620.21 μs)

[0073] MergeMap average merge time: 770 ns (0.77 μs)

[0074] Performance difference: MergeMap is 3402 times faster than HashMap.

[0075] = Test Scenario 2: Merging Multi-Level Inheritance Structures =

[0076] >>Inheritance chain depth test: 5-level inheritance, 1000 elements per level<<

[0077] Average merge time for HashMap: 175,750 ns

[0078] MergeMap average merge time: 1,230 ns

[0079] Performance difference: MergeMap is 142.9 times faster than HashMap.

[0080] >>Inheritance chain depth test: 5 levels of inheritance, 10,000 elements per level<<

[0081] Average merge time for HashMap: 3,724,050 ns

[0082] MergeMap average merge time: 1,360 ns

[0083] Performance difference: MergeMap is 2738.3 times faster than HashMap.

[0084] >>Inheritance chain depth test: 5-level inheritance, 100,000 elements per level<<

[0085] Average merge time for HashMap: 40,098,290 ns

[0086] MergeMap average merge time: 1,380 ns

[0087] Performance difference: MergeMap is 29056.7 times faster than HashMap.

[0088] MergeMap offers two data retrieval methods: one is to retrieve the value of the current node first (pre-order), and if not found, search the parent node; the other is to search the parent node first, and if the parent node doesn't have the data, search the current node (post-order). If two identical keys are written, but with different values, and the value of the later key needs to overwrite the value of the earlier key, a pre-order lookup should be used. This involves first searching the leaf nodes, as the data in the leaf nodes is the new data. Once the new node is found, the corresponding value is returned.

[0089] Preorder lookup and HashMap output the same data; both return a reference to the storage address of the value.

[0090] If the keys in the collection are unique, then the post-order and pre-order outputs of mergeMap are the same as those of hashMap. If the keys are duplicated, hashMap will keep one, mergeMap will keep one at the same level, and multiple at different levels.

[0091] The above method analyzes the data in the tables to determine the mapping of related data between different tables. Without creating additional sets to merge the data retrieved from the two tables, it achieves one-way access and retrieval of table data layer by layer through the mapping relationship between the tables. The mapping table only stores the mapping between the data of different tables and does not store the table data itself, thus saving unnecessary space. This not only improves space utilization but also improves the efficiency of data merging.

[0092] In one embodiment, between S5 and S6, the method further includes:

[0093] S8. Based on the second foreign key that references the primary key value in the second table, determine the corresponding third table in the source database.

[0094] The server determines the corresponding third table in the source database based on the second foreign key that references the primary key value in the second table. At this point, the third table is a child table of the second table, therefore the parent setting for rows in the third table can be the second related row.

[0095] S9. Based on the second key field carried by the second foreign key, find the third related row in the third table, and store each third related row as a third node associated with the second node in the temporary node set according to the preset method.

[0096] The server uses the second key field carried by the second foreign key to find the third related row in the third table, and stores each third related row as a third node associated with the second node in the temporary node set according to a preset method.

[0097] The third related row is the row data in the third table that is related to the row data in the second table. There may be multiple third related rows in the third table. The server generates a new MergeMap node for each third related row and executes the configured expression for each third related row. The result of processing the second related row is stored as the third node of the MergeMap structure. The server can set the parent of the second node to the second related row.

[0098] S10, repeat steps S8~S9 until the primary key values ​​of each table no longer have reference relationships, then proceed to step S6.

[0099] The server repeats steps S8-S9 until the primary key values ​​of each table no longer have reference relationships, then proceeds to step S6. The server repeats steps S8-S9 to check all main tables and sub-tables with reference relationships until the final table has no corresponding sub-table. Therefore, the resulting MergeMap structure may have multiple levels of child nodes under certain nodes.

[0100] The above method examines various main tables and sub-tables with reference relationships, so that the resulting target mapping table can map all tables in the source database.

[0101] In one embodiment, step S3 further includes the following steps: when it is determined that multiple parallel values ​​are stored in a specific column of the first row of data, it is determined that the first row of data needs to be split into columns; a new data row is created for each split parallel value, and the data row is stored in a temporary node set corresponding to the first node.

[0102] When the server determines that a specific column in the first row of data contains multiple parallel values, it decides that the first row needs to be split into columns. For example, assuming the current row is stored in a MergeMap (denoted as mapParent), and the value of a certain column (e.g., column A) is a string concatenated by delimiters ("a,b,c"), the server will split the value "a,b,c" in column A into three values. The server creates a new MergeMap for each split value (setting the parent to the original row). After splitting the value of that column into multiple values ​​(["a", "b", "c"]), the server creates a new MergeMap for each value (denoted as mapChild1, mapChild2, mapChild3), and sets the parent of these child nodes to the original row mapParent. The server sets the values ​​of the split columns: in the child nodes, it sets the corresponding split value of column A (i.e., the value of column A in mapChild1 is set to "a", in mapChild2 to "b", and in mapChild3 to "c"), to overwrite the value of that column in the parent node. The values ​​of other columns do not need to be stored repeatedly; they can be accessed through the parent chain. Each sub-MergeMap represents a new row of data.

[0103] The above method can facilitate the subsequent construction of a complete mapping link for certain data to achieve accurate retrieval and avoid omissions.

[0104] In one embodiment, storing each second associated row as a second node associated with the first node in a temporary node set according to a preset method includes: setting the first node as the parent node and the second node as the child node; merging each second associated row with the second node associated with the first node according to a preset method, and storing the merged row data and child node in a temporary node set.

[0105] The server sets the first node as the parent node and the second node as the child node. The server stores the current row (currentRow) in a MergeMap (denoted as mapA), and the row from the other table found through the related field (foundRow) is stored in another MergeMap (denoted as mapB). The server merges each second related row with the second node associated with the first node according to a preset method: the server calls mapA.merge(mapB), making mapB the parent node of mapA (mapA's parent points to mapB). At this point, mapA possesses all the data from mapB, but existing keys in mapA will overwrite identical keys in the parent node.

[0106] Then, when the server accesses data, when accessing a key in mapA, it first searches in mapA; if the key is not found there, it searches in the parent node mapB, and so on recursively, forming a structure like this. Figure 2 The MergeMap structure diagram is shown below.

[0107] The above method processes data in the MergeMap structure from left to right, and the data processing method is to first recursively process the parent node and then process the data of the current node, so that the target mapping table is generated in one go.

[0108] In one embodiment, the MergeMap structure data can start from the leaf nodes, i.e., the rightmost ones. The data in the nodes is processed first, and then the data of the parent nodes is processed recursively. In this way, the data processing order is from right to left, which can ensure that no omissions occur during the generation of the target mapping table.

[0109] In one embodiment, storing each second associated row as a second node associated with the first node in a temporary node set according to a preset method includes: obtaining the column categories corresponding to the row data of the second associated row and the row data of the first node; when the second associated row has the same column category as the first node, overwriting the row data mapping of the first node with the row data of the second associated row corresponding to that column category; when the second associated row does not have the same column category as the first node, merging the row data of the second associated row with that column category into the row data mapping of the first node.

[0110] The above method only stores the incremental changes and differential fields of the second table relative to the first table, and shares the parent data of the first table, saving unnecessary memory space and thus improving space utilization.

[0111] In one embodiment, S7 further includes: aggregating and grouping multiple rows of data in the temporary node set to obtain grouped row data; sorting the aggregated grouped row data using a custom function, and generating a target mapping table corresponding to the semiconductor level data based on the MergeMap structure.

[0112] The server aggregates and groups multiple rows of data in the ephemeral node set, resulting in grouped row data. The server can group multiple rows in the ephemeral node set by ID (Group By). This module aggregates rows with the same ID into a group, preparing for the execution of aggregation functions or subsequent processing.

[0113] The server uses a custom function to process the aggregated grouped row data and generates a target mapping table corresponding to the semiconductor-level data based on the MergeMap structure. The server can execute the custom function before writing to the target mapping table (functionBefore) and / or after writing to the mapping table (functionAfter). functionBefore performs aggregation processing on the grouped data before writing (such as SUM, MAX, etc.), and functionAfter performs subsequent cleanup or notifications. After the server processes the aggregated grouped row data using the custom function, it generates a target mapping table corresponding to the semiconductor-level data based on the MergeMap structure.

[0114] In one embodiment, a method for searching semiconductor-level data is also provided. The semiconductor-level data is merged using the method described above, and includes the following steps: obtaining a preset method in the target mapping table, determining the merging method of the first node and the second node; determining the order of data query according to the merging method; searching for the data to be searched from the first node and the second node in sequence according to the order, and outputting the storage location corresponding to the data to be searched.

[0115] The server retrieves the preset method from the target mapping table and determines the merging method for the first and second nodes. The server determines whether the merging method for the first and second nodes is processed in the MergeMap structure from left to right or from right to left.

[0116] The server then determines the parent and child nodes in the first or second node through a merging method, and then determines the order of data queries based on the merging method. For example, the order could be to first search from the child node (second node), and if it cannot be found, then search from the parent node (first node).

[0117] The server searches for the data to be found from the first node and the second node in sequence, and outputs the storage location corresponding to the data to be found.

[0118] In one embodiment, such as Figure 3 As shown, a semiconductor-level data merging device based on the MergeMap structure is also provided. The device includes a data acquisition module 301, an expression execution module 302, a filtering module 303, a connection module 304, a mapping module 305, a merging module 306, and a target mapping table generation module 307.

[0119] The data acquisition module 301 is used to obtain the first row of data from the first table in the source database.

[0120] The expression execution module 302 is used to process the first row of data through the configured expression and store the processed first row of data as the first node of the MergeMap structure.

[0121] The filtering module 303 is used to filter and judge the first row of data according to the configured filtering conditions to determine whether the first row of data needs to be split into columns and rows. When it is determined that it does not need to be split, the next row of data is obtained and the S2 step is repeated until all rows of data in the first table have been filtered and judged, and all rows of data are stored as the first node in the temporary node set.

[0122] The Join module 304 is used to determine the corresponding second table in the source database based on the association conditions that reference the primary key value in the first table.

[0123] The mapping module 305 is used to find the second related row in the second table according to the first key field carried by the association condition, and store each second related row as a second node associated with the first node in the temporary node set according to a preset method.

[0124] The Merge module 306 is used to merge the first and second nodes in the temporary node set to construct a MergeMap structure when it is determined that the source database does not have a second table that needs to be merged.

[0125] The target mapping table generation module 307 is used to generate a target mapping table corresponding to the semiconductor level data based on the MergeMap structure.

[0126] In one embodiment, the apparatus further includes:

[0127] The determination module is used to determine the corresponding third table in the source database based on the second foreign key that references the primary key value in the second table.

[0128] The lookup mapping module is used to find the third related row in the third table based on the second key field carried by the second foreign key, and store each third related row as a third node mapping associated with the second node in a temporary node set according to a preset method.

[0129] The loop module is used to repeat steps S8 to S9 until the primary key values ​​of each table no longer have reference relationships, and then proceed to step S6.

[0130] In one embodiment, the apparatus further includes a column-to-row splitting module, which includes:

[0131] The determination unit is used to determine whether the first row of data needs to be split into columns when it is determined that a specific column of the first row of data stores multiple parallel values.

[0132] Create a cell to create a new data row for each extracted and parallel value, and store the data row in a temporary node set corresponding to the first node.

[0133] In one embodiment, the mapping module includes:

[0134] The node setting unit is used to set the first node as the parent node and the second node as the child node.

[0135] The mapping storage unit is used to merge each second associated row with the second node associated with the first node according to a preset method, and to map and store the merged row data and child nodes into a temporary node set.

[0136] In one embodiment, the mapping module includes:

[0137] The column category acquisition unit is used to acquire the column categories corresponding to the row data of the second associated row and the row data of the first node.

[0138] The overlay unit is used to overwrite the row data mapping storage of the first node with the row data of the second associated row corresponding to the column category when the second associated row has the same column category as the first node.

[0139] The mapping storage unit is used to map and store the row data of the second associated row and the row data of the first node when the second associated row does not have the same column category as the first node.

[0140] In one embodiment, the apparatus further includes:

[0141] The grouping module is used to aggregate and group multiple rows of data in a temporary node set to obtain grouped row data.

[0142] The function execution module is used to sort the aggregated grouped row data through custom functions and generate a target mapping table corresponding to the semiconductor level data based on the MergeMap structure.

[0143] Specific limitations regarding the device for semiconductor hierarchical data merging based on the MergeMap structure can be found in the limitations of the method for semiconductor hierarchical data merging based on the MergeMap structure above, and will not be repeated here. Each module in the aforementioned device for semiconductor hierarchical data merging based on the MergeMap structure can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0144] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and the database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data such as temporary node sets and target mapping tables. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a semiconductor-level data merging method based on a MergeMap structure.

[0145] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0146] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: S1, retrieving the first row of data from a source database; S2, processing the first row of data using a configured expression, and storing the processed first row of data as the first node of a MergeMap structure; S3, filtering the first row of data according to configured filtering conditions to determine whether the first row of data needs to be split into columns and rows; when it is determined that it does not need to be split, retrieving the next row of data and repeating step S2 until all rows of data in the first table have been filtered and the first row of data has been split into columns and rows. Row data is stored as the first node in the temporary node set; S4, based on the association condition referencing the primary key value in the first table, it is determined whether a corresponding second table exists in the source database; S5, based on the first key field carried by the association condition, the second related row in the second table is found, and each second related row is stored as a second node associated with the first node in the temporary node set according to a preset method; S6, when it is determined that there is no second table to be merged in the source database, the first node and the second node in the temporary node set are merged to construct a MergeMap structure; S7, based on the MergeMap structure, a target mapping table corresponding to the semiconductor level data is generated.

[0147] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: S1, obtaining the first row of data from a source database; S2, processing the first row of data using a configured expression, and storing the processed first row of data as the first node of a MergeMap structure; S3, filtering the first row of data according to configured filtering conditions to determine whether the first row of data needs to be split into columns and rows; when it is determined that it does not need to be split, obtaining the next row of data and repeating step S2, until all rows of data in the first table have been filtered and the first row of data is split into columns and rows. S4. Based on the association condition referencing the primary key value in the first table, determine whether a corresponding second table exists in the source database; S5. Based on the first key field carried by the association condition, find the second related row in the second table, and store each second related row as a second node associated with the first node in the temporary node set according to a preset method; S6. When it is determined that there is no second table to be merged in the source database, merge the first node and the second node in the temporary node set to construct a MergeMap structure; S7. Based on the MergeMap structure, generate a target mapping table corresponding to the semiconductor level data.

[0148] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for merging semiconductor hierarchical data based on a MergeMap structure, characterized in that, include: S1, retrieve the first row of data from the first table in the source database; S2, process the first row of data using the configured expression, and store the processed first row of data as the first node of the MergeMap structure; S3, filter the first row of data according to the configured filtering conditions to determine whether the first row of data needs to be split into columns and rows; when it is determined that it is not necessary, obtain the next row of data and repeat step S2 until all rows of data in the first table have been filtered and judged, and store all rows of data as the first node in the temporary node set; S4. Based on the association conditions referencing the first table, determine whether a corresponding second table exists in the source database; S5, when it is determined that there is a second table in the source database that needs to be merged, the second related row in the second table is found according to the first key field carried by the association condition, and each second related row is stored in the temporary node set as a second node mapping associated with the first node according to a preset method. S6, when it is determined that there is no second table to be merged in the source database, the first node and the second node in the temporary node set are merged to construct the MergeMap structure; S7. Generate a target mapping table corresponding to the semiconductor level data based on the MergeMap structure.

2. The method according to claim 1, characterized in that, Between S5 and S6, the method also includes: S8, based on the second foreign key referencing the primary key value in the second table, determine the corresponding third table in the source database; S9, based on the second key field carried by the second foreign key, find the third related row in the third table, and store each third related row as a third node associated with the second node in a temporary node set according to a preset method; S10, repeat steps S8~S9 until the primary key values ​​of each table no longer have reference relationships, then proceed to step S6.

3. The method according to claim 1, characterized in that, Step S3 also includes: When it is determined that a specific column of the first row of data stores multiple parallel values, it is determined that the first row of data needs to be split into columns and rows. Create a new data row for each extracted and parallel value, and store the data row in a temporary node set corresponding to the first node.

4. The method according to claim 1, characterized in that, The step of storing each second associated row as a second node associated with the first node in a temporary node set according to a preset method includes: Set the first node as the parent node and the second node as the child node; According to a preset method, each second associated row is merged with the second node associated with the first node, and the merged row data is mapped and stored in a temporary node set.

5. The method according to claim 1, characterized in that, The step of storing each second associated row as a second node associated with the first node in a temporary node set according to a preset method includes: Obtain the column categories corresponding to the row data of the second associated row and the row data of the first node; When the second associated row has the same column category as the first node, the row data of the second associated row corresponding to that column category will overwrite the row data mapping of the first node. When the second associated row does not have the same column category as the first node, the row data of the second associated row and the row data of the first node are merged and stored together.

6. The method according to claim 1, characterized in that, The S7 also includes: Aggregate and group multiple rows of data in the temporary node set to obtain grouped row data; The aggregated grouped row data is sorted out using a custom function, and a target mapping table corresponding to the semiconductor level data is generated based on the MergeMap structure.

7. A method for searching semiconductor-level data, characterized in that, The semiconductor-level data is obtained by combining the data using the method described in any one of claims 1 to 6, including: Obtain the preset method from the target mapping table and determine the merging method of the first node and the second node; The order of data queries is determined according to the merging method described above; According to the order, the data to be searched is searched from the first node and the second node in sequence, and the storage location corresponding to the data to be searched is output.

8. A device for semiconductor hierarchical data merging based on a MergeMap structure, characterized in that, The device includes: The data acquisition module is used to retrieve the first row of data from the first table in the source database; The expression execution module is used to process the first row of data using a configured expression and store the processed first row of data as the first node of a MergeMap structure. The filtering module is used to filter the first row of data according to the configured filtering conditions to determine whether the first row of data needs to be split into columns and rows. When it is determined that it does not need to be split, the next row of data is obtained and the S2 step is repeated until all rows of data in the first table have been filtered and the first row of data is stored in a temporary node set as the first node. The connection module is used to determine whether a corresponding second table exists in the source database based on the association condition that references the primary key value in the first table; The mapping module is used to find the second related row in the second table according to the first key field carried by the association condition when it is determined that there is a second table in the source database that needs to be merged, and to store each second related row as a second node associated with the first node in a temporary node set according to a preset method. The merging module is used to merge the first node and the second node in the temporary node set to construct a MergeMap structure when it is determined that there is no second table to be merged in the source database. The target mapping table generation module is used to generate a target mapping table corresponding to the semiconductor level data based on the MergeMap structure.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images