Semiconductor data ETL method, device, computer equipment and storage medium

Abstract

The application provides a semiconductor data ETL method, device, computer equipment and storage medium, belongs to the field of semiconductor manufacturing, specifically includes adding a target table; displaying the table names of multiple data source tables obtained from a semiconductor database on a canvas configuration interface of the target table; displaying a field list of the table structure of the data source table, realizing data association by connecting the fields to construct a table relationship link of the data source table pointing to the target table; obtaining an expression rule corresponding to the fields to configure source table fields in each table structure to generate target fields, and displaying the target fields and the source table fields in association in real time on the canvas configuration interface; obtaining the target fields as primary keys of the target table; based on the table relationship link and the expression rule, performing an ETL task, and storing all configured target fields and primary keys in a target table created in advance by the system. Through the processing scheme of the application, complex conversion rules of database tables can be intuitively and quickly constructed, and data consistency is ensured.

Description

Technical Field

[0001] This application relates to the field of semiconductor intelligent manufacturing, and in particular to a semiconductor data ETL method, apparatus, computer equipment, and storage medium. Background Technology

[0002] ETL (Extract-Transform-Load) is the process of extracting, cleaning, and transforming data from business systems and then loading it into a data warehouse. In the manufacturing process, production data from the semiconductor factory's MES system needs to be cleaned, transformed, and loaded again according to business requirements, and then transported to the Real-Time Dispatch (RTD) system and the Factory Scheduling (FSO) system, ultimately providing analytical support for the wafer fab's decision-making.

[0003] However, as business complexity increases, ETL process configuration also becomes more complex, including control conditions for branches, loops, and waits within the ETL process. This further increases the difficulty of ETL process development efficiency and task management. Moreover, most existing ETL tools on the market use multi-page configuration information, displaying a large and complex amount of data content, and requiring numerous parameter inputs, leading to high configuration difficulty and a high learning curve for users. Furthermore, because ETL design often adopts a task-based approach, using drag-and-drop nodes with different functions for data processing and development, the task design is flexible. Each task should handle a specific data transformation method, and complex tasks are broken down into several independent subtasks. This not only increases subsequent management difficulty but also prevents further optimization of the current configuration for problems arising in complex scenarios.

[0004] In addition, ETL development requires consideration of field mapping between the source and target ends of each process configuration. These products offer limited effectiveness in areas such as rapid business analysis, data development efficiency, and task management. Even with the use of templated configuration parameters to reduce configuration complexity, the effectiveness of templates is still limited, especially given the lack of precedent for using template configurations in complex semiconductor scenarios. In practical business integration, a combination of SQL and application code is currently used to handle data transmission and conversion across different wafer processing batches. The initial planning process is cumbersome and involves a huge workload. Summary of the Invention

[0005] Therefore, in order to overcome the shortcomings of the prior art, the present invention provides a semiconductor data ETL method, apparatus, computer equipment and storage medium that can quickly and intuitively configure the data processing path of fields in a database table, with a convenient configuration process and a user-friendly experience. The table relationship link mode in the configuration process avoids the occurrence of complex subqueries with multiple nested layers in SQL, thereby reducing the complexity.

[0006] To achieve the above objectives, this invention provides a semiconductor data ETL method, comprising: S1, adding a target table to the system configuration interface, wherein the target table has no table structure configuration; S2, displaying the table names of multiple data source tables obtained from a semiconductor database through a drop-down menu on the canvas configuration interface of the target table; S3, parsing the table structure of the selected data source table according to business requirements, and displaying the field list corresponding to the table structure, realizing data association between the selected data source table and other data source tables through field connections, and constructing a table relationship link from each data source table to the target table; S4, obtaining the corresponding expression rules based on the field list in each table structure displayed on the canvas configuration interface, configuring the source table fields in each table structure to generate target fields, and displaying the target fields and source table fields in association in real time on the canvas configuration interface; S5, obtaining at least one specified target field as the primary key of the target table; S6, performing an ETL task based on the table relationship link and the expression rules, storing all configured target fields and defined target table primary keys in the target table pre-created by the system.

[0007] In one embodiment, displaying the field list corresponding to the table structure, and establishing data association between the selected data source table and other data source tables through field connections to construct a table relationship link from each data source table to the target table, includes: displaying the field list of each table structure selected by the user in the canvas configuration interface; setting the selected source table field as the associated field, and establishing a field connection when it is determined that the selected associated field is consistent between two sequentially added data source tables; and constructing a table relationship link from each data source table to the target table based on the associated field and the field connection.

[0008] In one embodiment, the step of displaying the field list corresponding to the table structure, and establishing data association between the selected data source table and other data source tables through field connections to construct a table relationship link from each data source table to the target table, further includes: filtering the data content in the data source table based on the filter field selected by the user from the field list; and adding highlight blocks to the associated fields and / or filter fields in the table structure of the data source table for display.

[0009] In one embodiment, obtaining corresponding expression rules based on the field list in each table structure displayed on the canvas configuration interface to configure the source table fields in each table structure to generate target fields includes: the canvas configuration interface also associates the associated fields and / or filter fields with the component configuration page; the component configuration page displays multiple pre-set expression syntaxes and an expression input field; and based on the expression syntaxes, configure expression rules to convert the values ​​corresponding to the associated fields and / or filter fields into the values ​​of multiple target fields in the target table.

[0010] In one embodiment, the method further includes storing the configured expression rules and the corresponding associated fields and / or filter fields as newly set expression syntax in the component configuration page.

[0011] In one embodiment, prior to step S5, the method further includes: extracting the values ​​of related fields from the data source table based on the table relationship chain and the target field; performing data cleaning on the extracted values ​​of related fields; transforming and deriving the values ​​of the configured target fields from the data source table based on the expression rules of the target field to generate the values ​​of the target fields; displaying the target field and a predetermined number of the values ​​of the target fields in the canvas configuration interface, and verifying the values ​​of the target fields.

[0012] A semiconductor data ETL device includes: a target table module for adding a target table in a system configuration interface, wherein the target table has no table structure configuration; a canvas module for displaying the table names of multiple data source tables obtained from a semiconductor database via a drop-down menu in the canvas configuration interface of the target table; a table relationship link module for parsing the table structure of the selected data source table according to business requirements, displaying a list of fields corresponding to the table structure, and establishing data association between the selected data source table and other data source tables through field connections, thereby constructing a table relationship link from each data source table to the target table; a field configuration module for obtaining corresponding expression rules based on the field list in each table structure displayed in the canvas configuration interface, configuring source table fields in each table structure to generate target fields, and displaying the target fields in association with the source table fields in real time in the canvas configuration interface; a local repository for obtaining at least one specified target field as the primary key of the target table; and performing an ETL task based on the table relationship link and the expression rules, storing all configured target fields and defined primary keys of the target table in the target table pre-created by the system.

[0013] In one embodiment, the apparatus further includes a data verification module, comprising: an extraction unit for extracting values ​​of related fields from the data source table based on the table relationship chain and the target field; a cleaning unit for cleaning the extracted values ​​of related fields; a transformation unit for transforming and deriving values ​​of the configured target field from the data source table based on the expression rules of the target field to generate values ​​of the target field; and a verification unit for displaying the target field and a predetermined number of values ​​of the target field in the target table and verifying the values ​​of the target field.

[0014] A computer device includes a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program to implement the steps of the above-described method.

[0015] A computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the above-described method.

[0016] Compared with existing technologies, the advantages of this invention are as follows: The data extraction, transformation, and loading (ETL) logic of the MES system is set through a graphically visualized dynamic process in the canvas configuration interface. The canvas configuration interface allows for intuitive viewing of data source tables, data filtering, data processing development, and data output, thereby addressing issues in subsequent business process management and optimization. Furthermore, by connecting fields to link various data source tables, the canvas configuration interface is simple, easy to configure, and provides a user-friendly experience. In addition, this simple configuration method of field connections intuitively demonstrates the complexity of configuring various data source tables. The entire process can complete data development and processing using only one data path, avoiding the need for temporary tables, intermediate tables, and other subqueries. Even in complex scenarios, it reduces programming complexity, making it easier to quickly conduct business analysis, improve data development efficiency, and ensure the accuracy of task management. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of a semiconductor data ETL method in one embodiment of the present invention;

[0019] Figure 2This is a schematic diagram of a semiconductor data ETL method in one embodiment of the present invention;

[0020] Figure 3 This is a structural block diagram of a semiconductor data ETL device in one embodiment of the present invention;

[0021] Figure 4 This is an internal structural diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0022] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0023] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] It should be noted that the following description covers various aspects of embodiments within the scope of protection of this invention. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this application, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0025] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0026] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0027] This application provides a semiconductor data ETL method that can be applied to a server or a terminal. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable smart devices. The server can be a standalone server or a server cluster composed of multiple servers.

[0028] In one embodiment, such as Figure 1 As shown, the semiconductor data ETL method is applied to a server as an example, and includes the following steps:

[0029] S1. Add a target table in the system configuration interface. The table structure of this target table has not been configured.

[0030] The server receives the user's command to add a target table from the system configuration interface, loads the canvas configuration interface in the data processing module, and completes the connection with the semiconductor database in the system configuration. The semiconductor database can be a database storing production data from a semiconductor factory's MES system, or a database storing data from a semiconductor factory's monitoring system. The semiconductor database stores various types of tables. Each table's structure can include table fields (columns), field data types, field constraints (such as primary keys, foreign keys, uniqueness constraints, etc.), and relationships between tables.

[0031] The input to the data source is integrated with the semiconductor data ETL device through data synchronization functionality, for example, it can assist in the synchronization of the semiconductor MES database. Therefore, the server can obtain all data source tables in the semiconductor database in a timely manner.

[0032] Adding a target table in the system configuration interface beforehand is to define the location for subsequent data export. This is an important step before configuring the data path, requiring explicit specification of the final target table for data storage. Users do not need to define or set the table structure initially; they only need to fill in the table name. When the server performs the ETL task in S6, it will input the target fields and their corresponding values ​​into the target table according to the configuration requirements. Figure 2 As shown, the target_table in the box on the right is the target table that was set. No table structure was configured in the early stage. All the target fields in the figure are obtained after ETL processing based on the data source table on the left.

[0033] S2, in the target table's canvas configuration interface, displays the table names of multiple data source tables obtained from the semiconductor database via a drop-down menu.

[0034] The canvas configuration interface allows for source table import, data development, and data validation. It also provides functions such as dragging the selected data source table, full-screen mode, and screen clearing. In one embodiment, the canvas configuration interface can also configure and view canvas functions such as publish mode and draft mode. The canvas configuration interface is presented to the user as a canvas-shaped user interface. Within this interface, the server can add data source tables, configure table relationship chains, display source table fields, configure target fields, configure filter fields, configure drag-and-drop configuration of related fields and / or filter fields, preview data paths, perform data analysis, and manage versions.

[0035] In the target table's canvas configuration interface, the server displays the names of multiple data source tables retrieved from the semiconductor database via a drop-down menu. The server can also display multiple data source tables retrieved from the semiconductor database via a drop-down menu in the canvas configuration interface, parse the table structure of the selected data source table, and display the corresponding field list. For example... Figure 2 As shown, table_1 is the selected data source table. The server displays the table structure of this data source table in the canvas configuration interface. This table structure contains three modules: EQUIP_ID (machine ID), status, and running_mode. EQUIP_ID (machine ID) contains EQUIP_TYPE (machine type) and EQUIP_STATUS (machine status). status contains DELETE_YN (machine deletion flag) and RECIPE_TYPE (recipe type). running_mode (running mode) contains LAST_EVENT_TIME (last processing time).

[0036] S3, based on business requirements, parses the table structure of the selected data source table and displays the field list corresponding to the table structure. It realizes the data association between the selected data source table and other data source tables through field connections, and builds the table relationship link between each data source table and the target table.

[0037] The server parses the table structure of the selected data source tables based on business requirements and displays the corresponding field list. The server can display all selected data source tables in the canvas configuration interface, and then add data relationships between all data source tables through field connections, constructing table relationship links from each data source table to the target table based on these connections. Alternatively, the server can display the selected data source tables one by one in the canvas configuration interface, and then dynamically add all configured data source tables in the development order through field connections, constructing table relationship links from each data source table to the target table based on the field connections and addition order. For example, Figure 2The table relationship chain is table_1→table_2→table_3, and the final processed data is input into target_table. The server can also set a user-selected source table field as a related field and determine whether the selected related field is consistent across different data source tables: if the related field uses the same expression across different data source tables, the server can establish a field connection; if the related field uses different expressions across different data source tables or the related field does not have a correlation in the data source tables, the server can issue a warning about the related field, allowing the user to reselect a source table field as the related field.

[0038] In one embodiment, JOIN relationships between tables can be formed by adding connections between fields in the canvas configuration interface, thereby creating a table relationship chain between different data source tables. In another embodiment, JOIN relationships between tables can also be formed by adding "+" between fields, thereby creating a complete table relationship chain between different data source tables. The server can dynamically construct data processing paths based on the connection order, avoiding the generation of temporary tables. Moreover, the server can also easily configure field associations and data filtering between different data source tables in sequence (at this time, the server can associate fields without involving additional expressions).

[0039] S4: Based on the list of fields in each table structure displayed in the canvas configuration interface, obtain the corresponding expression rules to generate target fields from the source table fields in each table structure, and display the target fields and source table fields in real time in the canvas configuration interface.

[0040] The server retrieves the corresponding expression rules based on the list of fields in each table structure displayed in the canvas configuration interface, so as to generate the target fields from the source table fields in each table structure, and displays the target fields and source table fields in real time on the canvas configuration interface.

[0041] The server can use expressions to allow users to process source table fields before associating them, and also to transform and filter target fields using expressions. The server can also configure variable relationships between filter and target fields using expressions. The server can directly retrieve all associated expression rules based on a field, display them in the canvas configuration interface, and then process the source table fields according to the expression rule selected by the user. Alternatively, the server can display fields and receive user-inputted expression rules to process the source table fields.

[0042] Specifically, the server receives user input by clicking on a selected source table field and editing an expression, directly displaying the source table page on the canvas configuration interface. This source table page also provides features such as field aggregation and field aliases. A single source table field can be configured with multiple target fields. The entire configuration is completed through click selection. The confirmed and generated target fields are highlighted with different colors to distinguish between the source and target tables, eliminating the need for additional definition of the mapping relationship between the data source and target tables. The server associates the target fields with the source table fields according to the configured expression rules.

[0043] S5, retrieve at least one specified target field as the primary key of the target table.

[0044] The server displays all target fields in the canvas configuration interface, and then receives at least one target field selected by the user as the primary key of the target table. Users can select multiple target fields as the primary key of the target table.

[0045] S6 performs ETL tasks based on table relationship links and expression rules, storing all configured target fields and defined primary keys of target tables in target tables pre-created by the system.

[0046] The server performs ETL tasks based on table relationship chains and expression rules. After extracting, cleaning, and transforming the data from the source table according to the expression rules, the server loads it into the target table according to the table relationship chains. The server stores the target fields and their corresponding primary keys in the target table. The complete target table can then be stored as a new data source table in the semiconductor database.

[0047] The above method uses a graphically visualized, dynamic workflow to configure the data extraction, transformation, and loading (ETL) logic of the MES system within the canvas configuration interface. This interface provides a clear view of the data source tables, data filtering, data processing development, and data output, thus addressing subsequent business process management and optimization issues. Furthermore, the method uses a dynamic, sequential connection to link the various data source tables. The canvas configuration interface is simple, easy to configure, and offers a user-friendly experience. Additionally, the simple configuration method of adding the order visually demonstrates the complexity of configuring the various data source tables. The entire process uses only one data path to complete data development and processing, avoiding the need for temporary tables, intermediate tables, and other subqueries. Even in complex scenarios, this reduces programming complexity, facilitating rapid business analysis, improving data development efficiency, and ensuring accurate task management.

[0048] In one embodiment, a list of fields corresponding to the table structure is displayed, and data association between the selected data source table and other data source tables is achieved through field connections, constructing a table relationship link from each data source table to the target table. This includes: displaying a list of fields in each table structure selected by the user in the canvas configuration interface; setting the selected source table fields as associated fields, and establishing field connections when it is determined that the selected associated fields are consistent between two sequentially added data source tables; and constructing a table relationship link from each data source table to the target table based on the associated fields and field connections.

[0049] Within the canvas configuration interface, the server displays a list of fields from each table structure selected by the user. The server then sets the selected source table fields as related fields, and establishes a connection between the fields if it determines that the selected related fields are consistent between the two sequentially added data source tables. For example... Figure 2 In this process, the server can receive the user-selected source table field EQUIP_ID as a related field. The server then determines that EQUIP_ID exists in tables 1, 2, and 3, and establishes a join between these fields to connect the data. Based on the related fields and their joins, the server can construct table relationship links from each data source table to the target table.

[0050] In one embodiment, the table structure is displayed with a list of fields corresponding to the table structure. Data association between the selected data source table and other data source tables is achieved through field connections, and a table relationship link is constructed from each data source table to the target table. The method also includes: filtering the data content in the data source table based on the filter fields selected by the user from the field list; and adding highlight blocks to the associated fields and / or filter fields in the table structure of the data source table for display.

[0051] The above method not only selects filter fields to filter the data content of the data source table, but also optimizes the display function, supporting one-click display of related fields and / or filter fields. Users can intuitively see the configured related fields and / or filter fields, making it easier for different users to understand the configuration content of the target table.

[0052] In one embodiment, the corresponding expression rules are obtained based on the list of fields in each table structure displayed on the canvas configuration interface, so as to configure the source table fields in each table structure to generate target fields. This includes: the canvas configuration interface also associates the associated fields and / or filter fields with the component configuration page; in the component configuration page, multiple pre-set expression syntaxes and an expression input field are displayed; based on the expression syntax, expression rules are configured to convert the values ​​corresponding to the associated fields and / or filter fields into the values ​​of multiple target fields in the target table.

[0053] The server can provide user experience optimization features such as field configuration feedback. The target table's fields, field sources, data transformations via expression editing, related fields, filter fields, and target table primary key (PK) are all displayed intuitively on the canvas configuration interface. The server receives expression rules input by the user. Based on the related fields and / or filter fields, the server can transform the corresponding source table data into values, and then, according to the expression rules, convert the values ​​of the related fields and / or filter fields into the corresponding target fields in the target table.

[0054] In one embodiment, the method further includes storing the configured expression rules and the corresponding associated fields and / or filter fields as newly set expression syntax in the component configuration page.

[0055] The above method allows for the syntactic storage of newly added rules and their corresponding associated and / or filtering fields, enabling business personnel to configure complex conversion rules without needing to be proficient in coding, reducing learning costs by 70% and improving operational efficiency by 200%.

[0056] In one embodiment, prior to S5, the method further includes: extracting the values ​​of related fields from the data source table based on the table relationship chain and the target field; performing data cleaning on the extracted values ​​of related fields; transforming and deriving the values ​​of the configured target fields from the data source table based on the expression rules of the target fields to generate the values ​​of the target fields; displaying the target fields and a predetermined number of the values ​​of the target fields in the canvas configuration interface, and verifying the values ​​of the target fields.

[0057] The server extracts the values ​​of related fields from the data source table based on table relationship links and target fields. For example, if the related field is user accounts, the server can extract all user accounts. The server can then perform data cleaning on the extracted related field values ​​to eliminate errors, duplicates, and incomplete data in the source table. Based on expression rules for the target field, the server transforms and derives the values ​​of the configured target fields from the data source table to generate the target field values. The values ​​of the configured target fields from the data source table refer to the values ​​obtained by processing the source table data according to the related fields and / or filter fields. Transformation refers to direct conversion based on expression rules. Data derivation refers to generating new data fields from existing source table fields through calculation, transformation, and combination based on expression rules.

[0058] The server can also perform expression validation by inputting parameters in the canvas configuration interface according to the expression content, and check the accuracy of the expression configuration.

[0059] In one embodiment, such as Figure 3As shown, a semiconductor data ETL apparatus is also provided, which communicates with a semiconductor database. The apparatus includes a target table module, a canvas module, a table relationship link module, a field configuration module, and a local repository.

[0060] The target table module is used to add a target table in the system configuration interface. This target table does not have a pre-defined table structure. The target table module defines the location where data is exported and is an important step before configuring the data path. In this module, you only need to explicitly specify the final target table for the data; you do not need to define the table structure. You can simply fill in the table name. After the data is calculated, the target fields will be entered into the target table according to the configuration requirements.

[0061] The Canvas module is used to display the names of multiple data source tables obtained from the semiconductor database via drop-down menus in the target table's canvas configuration interface. The Canvas module displays these tables through drop-down menus in the canvas configuration interface, parses the table structure of the selected data source table, and displays the corresponding field list. As the core module, the Canvas module is responsible for importing source tables, data development, and data validation. It also supports common canvas functions such as table dragging, full-screen mode, clear screen mode, and viewing configurations for both publish and draft modes. This module presents users with a canvas-like user interface, supporting functions such as adding source tables, configuring table relationship chains, displaying source table fields, configuring target fields, configuring filtered fields, dragging and dropping related fields, previewing data paths, data analysis, and version management.

[0062] The Table Relationship Link module is used to parse the table structure of the selected data source table according to business requirements and display the corresponding field list. It establishes data associations between the selected data source table and other data source tables through field joins, building a table relationship link between each data source table and the target table. This module can also dynamically add all configured data source tables in the canvas configuration interface according to business needs, building the table relationship link between each data source table and the target table based on the addition order. The path for configuring table associations in the Table Relationship Link module is a link composed of JOIN relationships between tables, supporting various JOIN types. It can import source table data into the canvas one sheet at a time according to business requirements, preparing for data development and transformation.

[0063] The field configuration module retrieves corresponding expression rules from the field list of each table structure displayed on the canvas configuration interface. This allows it to configure source table fields to generate target fields and displays the associated target fields with the source table fields in real-time on the canvas configuration interface. The module selects at least one associated field from the field list displayed on the canvas configuration interface and configures expression rules to transform the associated field into multiple target fields in the target table. This allows the source table data to be converted into values ​​corresponding to the target fields based on these expression rules. The field configuration module can be applied to three field types (target fields, associated fields, and filter fields). It supports users processing source table fields before field association and filtering target fields after transformation. Furthermore, the built-in expression function of the field configuration module also supports configuring variable relationships in filter fields. In one embodiment, to reduce user configuration, the field configuration module also provides expression syntax for common semiconductor manufacturing needs, allowing users to directly apply it through querying and simple analysis, thus reducing the difficulty of expression configuration. In the target field configuration, you can display and define field aliases, define target table PK, add aggregate function selection, etc. By clicking on the source table field, the data is transformed based on the source table field data to generate the target field, which is displayed in the table structure of the current source table with a highlight block for distinction. At the same time, the display function has been optimized to support the one-click display of target fields, so that the configured target fields can be seen intuitively.

[0064] A local repository is used to retrieve at least one specified target field as the primary key of the target table; based on table relationship links and expression rules, an ETL task is performed to store all configured target fields and defined primary keys of the target table in a pre-created target table.

[0065] In one embodiment, the device further includes a data verification module. The data verification module includes an extraction unit, a cleaning unit, a transformation unit, and a verification unit. The data verification module not only supports data preview and analysis but also expression validation. Users can input parameters according to the expression content to check the accuracy of the expression configuration.

[0066] The extraction unit is used to extract the values ​​of related fields from the data source table based on the table relationship chain and the primary key of the target field.

[0067] The data cleaning unit is used to clean the values ​​of extracted related fields. It can determine cleaning rules for related fields based on common filtering operators used in filtering processes. The cleaning unit handles errors, missing values, duplicate records, and other issues in the data, ensuring data quality and consistency.

[0068] The transformation unit is used to transform and derive the value of the configuration target field from the data source table based on the expression rules of the target field, and generate the value of the target field.

[0069] The validation unit is used to display the target field and a predetermined number of target field values ​​in the canvas configuration interface, and to validate the values ​​of the target fields.

[0070] The aforementioned semiconductor data ETL device provides a canvas-based data development approach to complete the data ETL configuration process. Through data path configuration, this device not only eliminates intermediate and temporary tables but also intuitively constructs data flows (add source table → display data source → directly configure target fields → target table output), replacing traditional code development and existing complex ETL configuration tools. Its configuration operations allow business personnel to configure complex transformation rules without coding expertise, reducing learning costs by 70% and improving operational efficiency by 200%. Furthermore, data preview, data analysis, and expression validation functions support effect verification, reducing rework rates by 50% and ensuring data consistency. Exporting the target table configuration also enables standardized reuse of process templates, improving team collaboration efficiency by 80% and supporting version management. Through the target table module, canvas module, table relationship chain module, field configuration module, and local repository, it addresses the pain points of existing ETL tools, such as hidden logic, difficult debugging, high development threshold, and delayed verification.

[0071] Specific limitations regarding the semiconductor data ETL device can be found in the limitations of the semiconductor data ETL method described above, and will not be repeated here. Each module in the aforementioned semiconductor data ETL device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0072] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores semiconductor data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a semiconductor data ETL method.

[0073] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: S1, adding a target table in the system configuration interface, the target table having no table structure configuration; S2, displaying the table names of multiple data source tables obtained from a semiconductor database through a drop-down menu in the canvas configuration interface of the target table; S3, parsing the table structure of the selected data source table according to business requirements, displaying the field list corresponding to the table structure, and realizing data association between the selected data source table and other data source tables through field connections, constructing a table relationship link from each data source table to the target table; S4, obtaining the corresponding expression rules based on the field list in each table structure displayed in the canvas configuration interface, configuring the source table fields in each table structure to generate target fields, and displaying the target fields and source table fields in association in real time in the canvas configuration interface; S5, obtaining at least one specified target field as the primary key of the target table; S6, performing an ETL task based on the table relationship link and expression rules, storing all configured target fields and the defined primary key of the target table in a pre-created target table in the system.

[0074] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: S1, adding a target table in the system configuration interface, the target table having no table structure configuration; S2, displaying the table names of multiple data source tables obtained from a semiconductor database through a drop-down menu in the canvas configuration interface of the target table; S3, parsing the table structure of the selected data source table according to business requirements, displaying the field list corresponding to the table structure, and realizing data association between the selected data source table and other data source tables through field connections, constructing a table relationship link from each data source table to the target table; S4, obtaining the corresponding expression rules based on the field list in each table structure displayed in the canvas configuration interface, configuring the source table fields in each table structure to generate target fields, and displaying the target fields and source table fields in association in real time in the canvas configuration interface; S5, obtaining at least one specified target field as the primary key of the target table; S6, performing an ETL task based on the table relationship link and expression rules, storing all configured target fields and the defined primary key of the target table in a pre-created target table in the system.

[0075] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A semiconductor data ETL method, characterized in that, include: S1, Add a target table in the system configuration interface. The table structure of this target table has not been configured. S2, in the canvas configuration interface of the target table, the table names of multiple data source tables obtained from the semiconductor database are displayed through a drop-down menu; S3. Based on business requirements, parse the table structure of the selected data source table and display the field list corresponding to the table structure. Realize the data association between the selected data source table and other data source tables through field connection, and build the table relationship link between each data source table and the target table. S4. Obtain the corresponding expression rules based on the field list in each table structure displayed on the canvas configuration interface, so as to configure the source table fields in each table structure to generate target fields, and display the target fields and the source table fields in real time on the canvas configuration interface. S5, retrieve at least one specified target field as the primary key of the target table; S6. Based on the table relationship chain and the expression rules, perform an ETL task, storing all configured target fields and defined primary keys of the target table in the target table pre-created by the system. The process of displaying the field list corresponding to the table structure, and establishing data associations between the selected data source table and other data source tables through field connections, constructs a table relationship link from each data source table to the target table, including: The canvas configuration interface displays a list of fields in each table structure selected by the user. Set the selected source table field as the related field, and establish the field connection when it is determined that the selected related field is consistent between the two sequentially added data source tables; Based on the associated fields and the connections between the fields, a table relationship link is constructed from each data source table to the target table.

2. The semiconductor data ETL method according to claim 1, characterized in that, The step of displaying the field list corresponding to the table structure, establishing data associations between the selected data source table and other data source tables through field connections, and constructing table relationship links from each data source table to the target table, also includes: The data content in the data source table is filtered based on the filter field selected by the user from the field list; In the table structure of the data source table, highlight blocks are added to display the associated fields and / or filter fields.

3. The semiconductor data ETL method according to claim 2, characterized in that, The process of obtaining corresponding expression rules from the field list of each table structure displayed on the canvas configuration interface to configure the source table fields in each table structure to generate target fields includes: The canvas configuration interface also associates the associated fields and / or filter fields with the component configuration page; The component configuration page displays several pre-set expression syntaxes and an expression input field; Based on the expression syntax, configure expression rules to convert the values ​​corresponding to the associated fields and / or filter fields into the values ​​of multiple target fields in the target table.

4. The semiconductor data ETL method according to claim 3, characterized in that, The method also includes: The configured expression rules and the corresponding associated fields and / or filter fields are stored as newly set expression syntax in the component configuration page.

5. The semiconductor data ETL method according to claim 1, characterized in that, Before S5, it also includes: Based on the table relationship chain and the target field, extract the value of the related field from the data source table; Perform data cleaning on the extracted values ​​of the related fields; Based on the expression rules of the target field, the value of the target field is generated by transforming and deriving the value of the configuration target field from the data source table; The canvas configuration interface displays the target field and a predetermined number of values ​​for the target field, and verifies the values ​​of the target field.

6. A semiconductor data ETL device, characterized in that, The device includes: The target table module is used to add a target table in the system configuration interface. The target table has not been configured with a table structure. The canvas module is used to display the names of multiple data source tables obtained from the semiconductor database via a drop-down menu in the canvas configuration interface of the target table. The table relationship link module is used to parse the table structure of the selected data source table according to business needs, and display the field list corresponding to the table structure. It realizes the data association between the selected data source table and other data source tables through the connection of fields, and builds the table relationship link between each data source table and the target table. The field configuration module is used to obtain the corresponding expression rules based on the field list in each table structure displayed on the canvas configuration interface, so as to configure the source table fields in each table structure to generate target fields, and to associate and display the target fields with the source table fields in the canvas configuration interface in real time. A local repository is used to retrieve at least one specified target field as the primary key of the target table; based on the table relationship chain and the expression rules, an ETL task is performed to store all configured target fields and defined primary keys of the target table in the target table pre-created by the system. The table relationship link module includes: The canvas configuration interface displays a list of fields in each table structure selected by the user. Set the selected source table field as the related field, and establish the field connection when it is determined that the selected related field is consistent between the two sequentially added data source tables; Based on the associated fields and the connections between the fields, a table relationship link is constructed from each data source table to the target table.

7. The semiconductor data ETL device according to claim 6, characterized in that, The device further includes a data verification module, which includes: The extraction unit is used to extract the value of the associated field from the data source table based on the table relationship chain and the target field; The cleaning unit is used to clean the extracted values ​​of related fields. A transformation unit is used to transform and derive the value of the configuration target field from the data source table based on the expression rules of the target field, and generate the value of the target field. The verification unit is used to display the target field and a predetermined number of the values ​​of the target field in the canvas configuration interface, and to verify the values ​​of the target field.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images