Visualizing semiconductor manufacturing etl systems

Abstract

The application provides a visual semiconductor manufacturing ETL system, comprising: a front-end module, a scheduling center module, an executor cluster module and a distributed cache module; the front-end module provides a visual operation interface; the scheduling center module is used for centralized management and scheduling of all tasks; the executor cluster module comprises a plurality of executor nodes deployed on a plurality of servers, each of the executor nodes is independently operated and uniformly scheduled by the scheduling center module; the plurality of servers constitute a distributed execution environment; and the distributed cache module is used for providing a distributed lock mechanism and data cache service to guarantee the uniqueness of task execution in the distributed execution environment. In this way, the visual operation interface is provided by the front-end module, so that the execution state of the task is visualized; the system resource scheduling is effectively optimized, the frequency of manual operation is reduced, and the automation level and execution efficiency of ETL management are comprehensively improved.

Description

Technical Field

[0001] This invention relates to the field of semiconductor manufacturing technology, and in particular to a visual semiconductor manufacturing ETL system. Background Technology

[0002] In traditional manufacturing, task execution commonly relies on the Windows system's built-in scheduled task mechanism. This approach has several limitations: First, the real-time performance of task execution is poor, and the lack of a unified centralized management platform leads to low overall system response efficiency. Second, the feedback mechanism for task execution status is inadequate, requiring operators to manually check the execution results of each task individually, making it impossible to achieve visualized monitoring of execution status and severely impacting operational efficiency. Furthermore, existing solutions lack disaster recovery mechanisms and dynamic load balancing capabilities, making it difficult to guarantee task continuity and stability when servers malfunction or load fluctuates. This increases the complexity of data analysis and system maintenance, significantly increasing the workload of manual intervention and the probability of errors. Summary of the Invention

[0003] The purpose of this invention is to provide a visualized semiconductor manufacturing ETL system to solve the problems of invisible task execution status, decentralized management, and delayed response to anomalies in the existing manufacturing industry.

[0004] To address the aforementioned technical problems, this invention provides a visual semiconductor manufacturing ETL system, comprising:

[0005] The module consists of a front-end module, a scheduling center module, an executor cluster module, and a distributed cache module.

[0006] The front-end module provides a visual user interface;

[0007] The scheduling center module is used for centralized management and scheduling of all tasks;

[0008] The executor cluster module includes multiple executor nodes deployed on multiple servers. Each executor node operates independently and is uniformly scheduled by the scheduling center module; the multiple servers constitute a distributed execution environment.

[0009] The distributed caching module provides a distributed locking mechanism and data caching services to ensure the uniqueness of task execution in the distributed execution environment.

[0010] Optionally, the scheduling center module is implemented using the XXL-JOB distributed task scheduling framework, and a unique request ID is generated for each task to cooperate with the distributed cache module to implement a distributed lock mechanism.

[0011] Optionally, before each task is executed, a distributed lock is acquired through the distributed caching module based on the unique request ID; regardless of whether the task is executed successfully or not, the distributed lock is released after a preset time.

[0012] Optionally, the scheduling center module includes a timed task unit, a centralized log unit, and a dynamic executor selection unit;

[0013] The scheduled task unit manages all tasks on a timed basis and supports scheduled triggering of tasks.

[0014] The centralized logging unit records all tasks and supports manual or event-triggered task activation.

[0015] The dynamic actuator selection unit automatically selects the corresponding actuator node in the actuator cluster module according to the selected strategy.

[0016] Optionally, when any of the executor nodes of the servers fails, the scheduling center module migrates unfinished tasks to healthy servers through the distributed lock mechanism and data caching service provided by the distributed cache module, thereby achieving automatic disaster recovery switching.

[0017] Optionally, the visualized semiconductor manufacturing ETL system includes an exception handling module. During automatic disaster recovery switching, the exception handling module queries unexecuted tasks and executes them automatically. If an execution error occurs, the error information is recorded.

[0018] Optionally, the visualized semiconductor manufacturing ETL system includes an exception handling module. If a task fails to execute within a preset time period and the last execution of the task also fails, the exception handling module resubmits the task to add it back to the scheduling queue of the scheduling center module for repeated execution.

[0019] Optionally, the distributed cache module is deployed using a three-sentinel architecture with one master and two slaves, wherein:

[0020] The primary sentinel node is responsible for write operations;

[0021] Two sentinel nodes are responsible for data backup and read operations;

[0022] The three sentinel nodes monitor each other, and automatically perform failover when the main sentinel node fails.

[0023] Optionally, the distributed cache module sets up a main service password authentication and specifies the authentication password in the sentinel configuration.

[0024] Optionally, the executor cluster module has dynamic load balancing capabilities, allowing the number of executor nodes to be dynamically expanded as the workload increases.

[0025] In summary, the visualized semiconductor manufacturing ETL system provided by this invention includes: a front-end module, a scheduling center module, an actuator cluster module, and a distributed cache module; the front-end module provides a visual operation interface; the scheduling center module is used for centralized management and scheduling of all tasks; the actuator cluster module includes multiple actuator nodes deployed on multiple servers, each actuator node operating independently and subject to unified scheduling by the scheduling center module; the multiple servers constitute a distributed execution environment; the distributed cache module is used to provide a distributed locking mechanism and data caching services to ensure the uniqueness of task execution in the distributed execution environment.

[0026] This configuration provides a visual interface through the front-end module, making the task execution status visible. The scheduling center module centrally manages and schedules all tasks, and the executor cluster module, deployed in a multi-server architecture, executes ETL tasks, thus building a unified task management and scheduling system. Combined with the distributed locking mechanism provided by the distributed caching module, it ensures that the same task is executed by only a single node in the cluster, eliminating concurrency conflicts at the architectural level. This effectively optimizes system resource scheduling, reduces the frequency of manual operations, significantly reduces the operational burden, and improves system stability, real-time performance, and data processing accuracy, thereby comprehensively improving the automation level and execution efficiency of ETL management. Attached Figure Description

[0027] Those skilled in the art will understand that the accompanying drawings are provided to better understand the invention and do not constitute any limitation on the scope of the invention.

[0028] Figure 1 This is a schematic diagram of the modules of a visualized semiconductor manufacturing ETL system according to an embodiment of the present invention.

[0029] Figure 2 This is a schematic diagram of the architecture of the distributed caching module in an embodiment of the present invention.

[0030] In the attached diagram: 1-Front-end module; 2-Schedule center module; 3-Executor cluster module; 4-Distributed cache module; 41-Master sentinel node; 42-Slave sentinel node; 5-Exception handling module. Detailed Implementation

[0031] To make the objectives, advantages, and features of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the drawings are all in a very simplified form and are not drawn to scale, and are only used to facilitate and clarify the explanation of the embodiments of this invention. Furthermore, the structures shown in the drawings are often part of the actual structures. In particular, different figures may emphasize different aspects and may sometimes use different scales.

[0032] As used in this invention, the singular forms “a,” “an,” “one,” and “the” include plural objects; the term “or” is generally used to mean “and / or”; the term “a number” is generally used to mean “at least one”; and the term “at least two” is generally used to mean “two or more”. Furthermore, the terms “first,” “second,” and “third” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first,” “second,” or “third” may explicitly or implicitly include one or at least two of that feature; “one end” and “the other end,” and “proximal end” and “distal end” generally refer to two corresponding parts, which include not only endpoints. Furthermore, the terms "installed," "connected," and "attached," as used in this invention, and the term "set" on one element from another, should be interpreted broadly. They generally only indicate a connection, coupling, cooperation, or transmission relationship between the two elements, which can be direct or indirect through an intermediate element. They should not be construed as indicating or implying a spatial relationship between the two elements, meaning one element can be located inside, outside, above, below, or to one side of another element, unless otherwise explicitly stated. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances. Additionally, directional terms such as above, below, up, down, upward, downward, left, and right are used relative to exemplary embodiments as shown in the figures, with upward or upper directions pointing towards the top of the corresponding figure, and downward or lower directions pointing towards the bottom of the corresponding figure.

[0033] The purpose of this invention is to provide a visualized semiconductor manufacturing ETL system to solve the problems of unvisualized task execution status, decentralized management, and delayed response to anomalies in existing manufacturing systems. The following description refers to the accompanying drawings.

[0034] Please refer to Figure 1 This invention provides a visualized semiconductor manufacturing ETL (Extract, Transform, Load) system, comprising: a front-end module 1, a scheduling center module 2, an executor cluster module 3, and a distributed cache module 4; the front-end module 1 provides a visual operation interface; the scheduling center module 2 is used for centralized management and scheduling of all tasks; the executor cluster module 3 includes multiple executor nodes deployed on multiple servers, each executor node operating independently and subject to unified scheduling by the scheduling center module 2; the multiple servers constitute a distributed execution environment; the distributed cache module 4 is used to provide a distributed locking mechanism and data caching services to ensure the uniqueness of task execution in the distributed execution environment.

[0035] The visualized semiconductor manufacturing ETL system provided in this embodiment preferably adopts a B / S (Browser / Server) architecture, supporting browser access and eliminating the need for client installation, thereby reducing deployment and maintenance costs. In one example, the front-end module 1 is built based on technologies such as the front-end UI framework (Bootstrap), the JavaScript library (jQuery), and the visualization chart library (ECharts) to realize a responsive visualization interface and user interaction functions.

[0036] In existing technologies, manufacturing ETL tasks for various production lines and equipment are dispersed and configured in Windows scheduled tasks on different servers. Configuration modifications require operation on a machine-by-machine basis, resulting in high management costs. In contrast, in the visualized semiconductor manufacturing ETL system of this embodiment, the scheduling center module 2 centrally manages and schedules all manufacturing ETL tasks (i.e., manufacturing data processing tasks), improving response efficiency and overall system operating efficiency.

[0037] The executor cluster module 3 is used to execute tasks assigned and scheduled by the scheduling center module 2. It consists of multiple executor nodes deployed on multiple servers. The executors deployed on multiple servers can be identical. The scheduling center module 2 can assign a task to a specific executor node for execution. Combined with the scheduling center module 2, distributed task execution and load balancing can be achieved.

[0038] However, the inventors discovered that executors deployed across multiple servers face certain conflict risks in task scheduling under high concurrency scenarios. For example, simultaneous triggering of the same stored procedure can easily lead to database table locking, duplicate data writes, and other anomalies. To address this, the visualized semiconductor manufacturing ETL system in this embodiment incorporates a distributed cache module 4, providing a distributed lock mechanism and data caching services. Each task must acquire a distributed lock before execution, ensuring that the same task is executed by only a single executor node within the executor cluster module 3. This architectural approach eliminates concurrency conflicts, guarantees the uniqueness of task execution in a distributed execution environment, and improves the reliability of high-concurrency task scheduling.

[0039] One function of the scheduling center module 2 is to balance the load of executors deployed on various servers. Optionally, the scheduling center module 2 includes a scheduled task unit, a centralized log unit, and a dynamic executor selection unit; the scheduled task unit performs timed management of all tasks (e.g., Oracle, MySQL, Redis), and supports scheduled task triggering; the centralized log unit logs all tasks and supports manual or event-triggered task triggering; optionally, the centralized log unit supports page queries, such as by executor / task name / status / scheduling time, etc. The dynamic executor selection unit automatically selects the corresponding executor node in the executor cluster module 3 according to the selected strategy (e.g., round-robin, random, least frequently used, or busy transfer strategies). Furthermore, the executor cluster module 3 has dynamic load balancing capabilities, allowing the number of executor nodes to be dynamically expanded as the task volume increases.

[0040] In one embodiment, the distributed cache module 4 can be deployed based on a Redis (RemoteDictionaryServer) cluster. It contains several distributed Redis nodes to ensure a highly available distributed locking mechanism and data caching service.

[0041] Optionally, the scheduling center module 2 is implemented using the XXL-JOB distributed task scheduling framework, and a unique request ID is generated for each task to cooperate with the distributed cache module 4 to implement a distributed lock mechanism.

[0042] In one embodiment, before each task is executed, a distributed lock is acquired through the distributed cache module 4 based on the unique request ID; the distributed lock is released after the task is successfully executed; if the task fails to execute, the distributed lock is released after a preset time to avoid subsequent tasks being unable to execute due to the failure to release the distributed lock.

[0043] The following is a illustrative example of the normal execution process of a task:

[0044] First, the page log is recorded to indicate that the task has started execution: XxlJobHelper.log("start");

[0045] Generate a unique request ID (requestId), and add a distributed lock (redisLock.tryLock(key, requestId, 30)) using the distributed caching module 4.

[0046] The system parses the parameters passed from the page, executes different logic based on different parameters, concatenates the parsed parameters into a complete stored procedure name, and executes it dynamically. For example, it can use the MyBatis @Insert annotation to call an Oracle stored procedure.

[0047] After the task is executed, the distributed lock is released using redisLock.releaseLock(key, requestId).

[0048] Optionally, when any of the executor nodes of the servers fails, the scheduling center module 2 migrates unfinished tasks to a healthy server through the distributed lock mechanism and data caching service provided by the distributed cache module, achieving automatic disaster recovery switching. Another function of the scheduling center module 2 is automatic disaster recovery switching. Specifically, when a server of the executor cluster module 3 goes down, the scheduling center module 2 can migrate unfinished tasks to a healthy server of the executor cluster module 3 based on a distributed lock, ensuring business continuity, eliminating the inefficient mode of traditional manual intervention, and improving task completion rate and system stability.

[0049] Optionally, the visualized semiconductor manufacturing ETL system includes an exception handling module 5. During automatic disaster recovery switching, this module queries unexecuted tasks and executes them automatically. If an execution error occurs, error information is recorded. One function of the exception handling module 5 is that in the event of a server crash, some tasks may not be scheduled or executed. In this case, the exception handling module 5 can query unexecuted tasks and execute them automatically during automatic disaster recovery switching. The automatic execution process can refer to the normal task execution process described above, including steps such as generating a unique request ID and acquiring a distributed lock, which will not be detailed here. In some embodiments, the queried unexecuted tasks may not be executed automatically but can be manually executed by the user. After execution, the task execution status is changed to successful execution, and the distributed lock is released. Specifically, if an execution error occurs, such as a program error, the task execution status is changed to failed execution, the distributed lock is released, and error information (such as error code, error stack, error time, etc.) is recorded for subsequent tracking and analysis. Preferably, the system further automatically pushes alarm information containing error information to the user's Outlook email address for rapid response and fault location.

[0050] Optionally, for tasks that failed to execute within a preset time period, and whose last execution also failed, the exception handling module resubmits the task, adding it back to the scheduling queue of the scheduling center module 2 for repeated execution. Another function of the exception handling module 5 is the rescheduling of failed tasks. Due to anomalies such as empty data, network failures, or system crashes, some tasks may fail. The exception handling module 5 can query tasks that failed to execute within a preset time period (e.g., one hour), and if the last execution of the same task within that preset time period also failed, it obtains the current user's login cookie and adds the task back to the scheduling queue of the scheduling center module 2 for repeated execution.

[0051] Please refer to Figure 2 Optionally, the distributed cache module 4 is deployed in a three-sentinel architecture with one master and two slaves, wherein: the master sentinel node 41 is responsible for write operations; the two slave sentinel nodes 42 are responsible for data backup and read operations; the three sentinel nodes monitor each other, and automatically perform failover when the master sentinel node 41 fails.

[0052] The three-sentinel architecture with one master and two slaves ensures the continuity of the distributed lock service. If the master sentinel node 41 fails, automatic master-slave switching can be achieved. After the switch, the data of the slave sentinel node 42 is consistent with the original master sentinel node 41, avoiding the loss of the distributed lock state. This ensures that the upper-layer disaster recovery logic remains unchanged, without needing to be aware of whether the distributed cache module 4 is faulty.

[0053] Optionally, the distributed cache module sets a master service password authentication and specifies the authentication password in the sentinel configuration. In the distributed cache module 4, the master sentinel node 41 is configured with an access password, and the configuration file of the slave sentinel node 42 contains authentication parameters that match the access password. This enables the slave sentinel node 42 to operate on the master sentinel node 41 through authentication during monitoring and failover, ensuring the continuity and security of the distributed lock service.

[0054] To ensure the security and high availability of the distributed caching module, a dual authentication approach is implemented in a three-sentinel architecture (one master, two slaves): First, an access password (requirepass) is configured on the master sentinel node 41, forcing all clients to authenticate. Second, the authentication parameters for the master sentinel node 41 are explicitly specified in the configuration of the slave sentinel node 42. This design ensures that the slave sentinel node 42 has legitimate operating permissions throughout continuous monitoring and failover, preventing monitoring interruptions or failover due to authentication failures. When the master sentinel node 41 fails, the slave sentinel node 42 authenticates itself as the new master sentinel node using the pre-configured password, enabling seamless recovery of the distributed lock service and providing dual assurance of reliability and security for automatic disaster recovery switching.

[0055] In summary, the visualized semiconductor manufacturing ETL system provided by this invention includes: a front-end module, a scheduling center module, an executor cluster module, and a distributed cache module. The front-end module provides a visual operation interface; the scheduling center module is used for centralized management and scheduling of all tasks; the executor cluster module includes multiple executor nodes deployed on multiple servers, each executor node operating independently and subject to unified scheduling by the scheduling center module; the multiple servers constitute a distributed execution environment; the distributed cache module provides a distributed locking mechanism and data caching service to ensure the uniqueness of task execution in the distributed execution environment. This configuration, with the front-end module providing a visual operation interface to visualize the task execution status; the scheduling center module centrally managing and scheduling all tasks; and the executor cluster module executing ETL tasks in conjunction with the multi-server deployment architecture, constructs a unified task management and scheduling system. Combined with the distributed locking mechanism provided by the distributed cache module, it ensures that the same task is executed by only a single node in the cluster, eliminating concurrent conflicts at the architectural level. This effectively optimizes system resource scheduling, reduces the frequency of manual operations, significantly reduces the operational burden, and improves system stability, real-time performance, and data processing accuracy, thereby comprehensively improving the automation level and execution efficiency of ETL management.

[0056] It should be noted that the above embodiments can be combined with each other. The above description is only a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the present invention.

Claims

1. A visual semiconductor manufacturing ETL system, characterized in that, include: The module consists of a front-end module, a scheduling center module, an executor cluster module, and a distributed cache module. The front-end module provides a visual user interface; The scheduling center module is used for centralized management and scheduling of all tasks; The executor cluster module includes multiple executor nodes deployed on multiple servers. Each executor node operates independently and is uniformly scheduled by the scheduling center module; the multiple servers constitute a distributed execution environment. The distributed caching module provides a distributed locking mechanism and data caching services to ensure the uniqueness of task execution in the distributed execution environment.

2. The visual semiconductor manufacturing ETL system according to claim 1, characterized in that, The scheduling center module is implemented using the XXL-JOB distributed task scheduling framework, and generates a unique request ID for each task to cooperate with the distributed cache module to implement a distributed lock mechanism.

3. The visual semiconductor manufacturing ETL system according to claim 2, characterized in that, Before each task is executed, a distributed lock is acquired through the distributed caching module based on the unique request ID; regardless of whether the task is executed successfully or not, the distributed lock is released after a preset time.

4. The visual semiconductor manufacturing ETL system according to claim 1, characterized in that, The scheduling center module includes a timed task unit, a centralized log unit, and a dynamic executor selection unit. The scheduled task unit manages all tasks on a timed basis and supports scheduled triggering of tasks. The centralized logging unit records all tasks and supports manual or event-triggered task activation. The dynamic actuator selection unit automatically selects the corresponding actuator node in the actuator cluster module according to the selected strategy.

5. The visual semiconductor manufacturing ETL system according to claim 1, characterized in that, When any of the executor nodes of the servers fails, the scheduling center module migrates unfinished tasks to healthy servers through the distributed locking mechanism and data caching service provided by the distributed cache module, thereby achieving automatic disaster recovery switching.

6. The visualized semiconductor manufacturing ETL system according to claim 5, characterized in that, The visualized semiconductor manufacturing ETL system includes an exception handling module. During automatic disaster recovery switching, the exception handling module queries unexecuted tasks and executes them automatically. If an execution error occurs, the error information is recorded.

7. The visual semiconductor manufacturing ETL system according to claim 5, characterized in that, The visualized semiconductor manufacturing ETL system includes an exception handling module. If a task fails to execute within a preset time period and the last execution of the task also fails, the exception handling module resubmits the task so that it is added back to the scheduling queue of the scheduling center module for repeated execution.

8. The visual semiconductor manufacturing ETL system according to claim 1, characterized in that, The distributed caching module is deployed using a three-sentinel architecture with one master and two slaves, wherein: The primary sentinel node is responsible for write operations; Two sentinel nodes are responsible for data backup and read operations; The three sentinel nodes monitor each other, and automatically perform failover when the main sentinel node fails.

9. The visual semiconductor manufacturing ETL system according to claim 8, characterized in that, The distributed cache module sets up main service password authentication and specifies the authentication password in the sentinel configuration.

10. The visual semiconductor manufacturing ETL system according to claim 1, characterized in that, The actuator cluster module has dynamic load balancing capabilities, allowing the number of actuator nodes to be dynamically expanded as the workload increases.

Images