Intelligent big data analysis public cloud and method for semiconductor industry

Abstract

The invention discloses an intelligent big data analysis public cloud and method in the semiconductor industry. The system comprises a task scheduling platform used for receiving scheduling tasks issued by different tenants, and performing task scheduling according to task features corresponding to the scheduling tasks and a task configuration table; the resource computing platform is used for calling a pre-configured integration algorithm according to the task scheduling to carry out task processing to obtain a corresponding data processing result; and the distributed storage platform is used for encrypting the data processing result and storing the encrypted data processing result in the object storage space. In the embodiment of the invention, through the mutual cooperation of the task scheduling platform, the resource computing platform and the distributed storage platform, the requirements of real-time processing and analysis implementation and the complex data analysis requirements of each semiconductor enterprise can be met, and the data security and privacy of each tenant are ensured.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing and big data technology, and in particular to a public cloud and method for intelligent big data analysis in the semiconductor industry. Background Art

[0002] In the semiconductor industry, the entire manufacturing process involves multiple companies and factories, and a large amount of data is generated in each link. These data come from a wide range of sources, including simulation software, chip testing equipment, wafer manufacturing plants, packaging and testing plants, etc. Traditional big data analysis systems mainly rely on local data centers and internal enterprise data warehouses, and the technologies used include traditional relational databases (such as MySQL, PostgreSQL) and non-relational databases (such as MongoDB, Cassandra), as well as computing frameworks such as Hadoop and Spark. However, these systems have many limitations in dealing with the complexity, diversity, and massiveness of data unique to the semiconductor industry, and traditional systems usually share physical resources with other systems within the enterprise, making them difficult to operate and manage, and prone to data leakage. Summary of the invention

[0003] In view of this, the present invention provides a public cloud and method for intelligent big data analysis in the semiconductor industry, which can meet the needs of real-time processing, implementation analysis and the complex data analysis needs of various semiconductor companies, and ensure the data security and privacy of each tenant.

[0004] According to one aspect of the present invention, an embodiment of the present invention provides a public cloud for intelligent big data analysis in the semiconductor industry, the public cloud for intelligent big data analysis in the semiconductor industry comprising: the public cloud for intelligent big data analysis comprises: a pre-built task scheduling platform, a resource computing platform, and a distributed storage platform with encryption features;

[0005] The scheduling platform pre-configures a task configuration table; the task configuration table includes at least: task ID, task type, task triggering method, task priority, task resource configuration, dependency, attribute information of task input data and output data, task retry strategy after execution failure, and task log configuration;

[0006] The task scheduling platform is used to receive the scheduling tasks issued by different tenants and perform task scheduling according to the task characteristics corresponding to the scheduling tasks and the task configuration table;

[0007] The resource computing platform is used to call a pre-configured integrated algorithm according to the task scheduling to perform task processing and obtain corresponding data processing results;

[0008] The distributed storage platform is used to encrypt the data processing results and store the encrypted data processing results in the object storage space.

[0009] According to another aspect of the present invention, an embodiment of the present invention further provides an intelligent big data analysis method for the semiconductor industry, which is applied to the intelligent big data analysis of the semiconductor industry. The intelligent big data analysis public cloud includes: a pre-built task scheduling platform, a resource computing platform, and a distributed storage platform with encryption features, which serve as the basic platform of the intelligent big data analysis system;

[0010] The scheduling platform pre-configures a task configuration table; the task configuration table includes at least: task ID, task type, task triggering method, task priority, task resource configuration, dependency, attribute information of task input data and output data, task retry strategy after execution failure, and task log configuration;

[0011] The intelligent big data analysis method comprises:

[0012] Receiving scheduling tasks issued by different tenants through the task scheduling platform, and performing task scheduling according to task features corresponding to the scheduling tasks and the task configuration table;

[0013] The resource computing platform calls a pre-configured integrated algorithm according to the task scheduling to perform task processing to obtain corresponding data processing results;

[0014] The data processing result is encrypted through the distributed storage platform, and the encrypted data processing result is stored in the object storage space.

[0015] The technical solution of the embodiment of the present invention is to schedule tasks according to the task characteristics and task configuration table corresponding to the scheduled tasks through the task scheduling platform, and the resource computing platform calls the pre-configured integrated algorithm according to the task scheduling to perform task processing to obtain the corresponding data processing results. The distributed storage platform encrypts the data processing results and stores the encrypted data processing results in the object storage space. Therefore, through the mutual cooperation between the task scheduling platform, the resource computing platform and the distributed storage platform, it can meet the needs of real-time processing, implementation analysis and the complex data analysis needs of various semiconductor companies, and ensure the data security and privacy of each tenant.

[0016] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A schematic diagram of the structure of a public cloud for intelligent big data analysis in the semiconductor industry provided by one embodiment of the present invention;

[0019] Figure 2 A schematic diagram of the structure of another intelligent big data analysis public cloud for the semiconductor industry provided by an embodiment of the present invention;

[0020] Figure 3 A flowchart of an intelligent big data analysis method for the semiconductor industry provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] In one embodiment, Figure 1 A schematic diagram of the structure of a public cloud for intelligent big data analysis in the semiconductor industry provided in accordance with an embodiment of the present invention. This embodiment is applicable to situations when data analysis is performed on data in the semiconductor industry.

[0024] like Figure 1As shown, in this embodiment, the intelligent big data analysis public cloud for the semiconductor industry includes the following specific steps:

[0025] A pre-built task scheduling platform 110, a resource computing platform 120, and a distributed storage platform 130 with encryption features;

[0026] Among them, the task scheduling platform 110 pre-configures a task configuration table; the task configuration table at least includes: task ID, task type, task triggering method, task priority, task resource configuration, dependency, attribute information of task input data and output data, task retry strategy after execution failure, and task log configuration;

[0027] The task scheduling platform 110 is used to receive the scheduling tasks issued by different tenants and perform task scheduling according to the task characteristics and task configuration table corresponding to the scheduling tasks;

[0028] The resource computing platform 120 is used to call the pre-configured integrated algorithm to perform task processing according to task scheduling to obtain corresponding data processing results;

[0029] The distributed storage platform 130 is used to encrypt the data processing results and store the encrypted data processing results in the object storage space.

[0030] Among them, the task characteristics corresponding to the scheduling task can be understood as the task characteristics of the scheduling task. Each scheduling task corresponds to a corresponding task ID. The task characteristics corresponding to different task IDs can be the same or different. The task characteristics can include but are not limited to the task startup method is event-driven, clock-driven, manually triggered, etc. The integrated algorithm pre-configured by the resource computing platform is at least two algorithms integrated in the resource computing platform. The integrated algorithm at least includes: the jackknife algorithm, the Kriging algorithm, and the chip volatility heartbeat check. Among them, the jackknife algorithm can also be called the jackknife distance algorithm. Jackknife is a resampling method for calculating the Jackknife distance using the estimated values ​​of the mean, standard deviation and correlation matrix, that is, to estimate the accuracy of sample statistical data by excluding one observation value from the sample set at a time through the distance, and to evaluate the variability of statistical data, especially when the number of samples is small, it is more accurate. The Kriging algorithm is a regression algorithm used for spatial modeling and prediction. It interpolates the random process according to the covariance function and is used to calculate the deviation between data. The chip volatility heartbeat check is used for heartbeat detection of chips in semiconductors. In this embodiment, different integrated algorithms have different functions, and a corresponding algorithm can be selected according to the scheduling task to execute the scheduling task.

[0031] In this embodiment, the task scheduling platform 110 is pre-configured with a task configuration table, which includes at least: task ID, task type, task triggering method, task priority, task resource configuration, dependency, attribute information of task input data and output data, retry strategy after task execution failure, and task log configuration; wherein, the task type may include but is not limited to Spark, Flink, Python, Shell and other types; the task triggering method may include but is not limited to event-driven, clock-driven, manual triggering, etc. For event-driven tasks, the type and source of the triggering event can be specified; for clock-driven tasks, a specific execution time or time interval can be set; the task resource configuration defines the task The required computing and storage resources, such as CPU, memory, etc., are mainly configured with upper limits, and actual use can be achieved through Kubernetes resource requests and restrictions; dependencies describe the dependencies between tasks, ensuring that dependent tasks can be executed only after the predecessor tasks are completed; the attribute information of the task input data and output data can be understood as the input data and output data parameters of the task, which may include but is not limited to the input and output data sources, data formats and other information; the retry strategy of the task after execution failure is the retry strategy of the task after execution failure, including but not limited to the number of retries, retry interval, failure alarms, etc.; the task log configuration specifies the monitoring indicators and log recording methods of the task, and whether centralized management of the log monitoring and analysis platform is required.

[0032] In this embodiment, the intelligent big data analysis public cloud for the semiconductor industry includes a task scheduling platform 110, a resource computing platform 120, and a distributed storage platform 130 with encryption features; the constructed task scheduling platform, resource computing platform, and distributed storage platform with encryption features provide elastic cloud-native data computing and storage solutions. In one embodiment, the construction of the distributed storage platform 130 includes: using the container engine Docker to run the MinIO instance, using the StatefulSet controller of Kubernetes to deploy the MinIO cluster, so that each MinIO instance has a stable network identity and persistent storage, and performing MinIO cluster orchestration; configuring the Kubernetes storage class, which can be expressed as StorageClass, and the Kubernetes storage class provides dynamic storage supply for the MinIO cluster to ensure that storage resources can be automatically expanded and reduced according to demand; configuring the JuiceFS file system, configuring the MinIO cluster as the backend storage, configuring the encryption policy for MinIO, configuring the SSL certificate in the MinIO instance, and enabling the HTTPS protocol; integrating the key management system in the MinIO cluster, using the preset external key to encrypt / decrypt the data processing results stored in the object storage space, and ensuring the high security of the stored data; configuring the NetworkPolicy of JuiceFS to ensure that only authorized tenants can access and operate the JuiceFS file system. It can be understood that the MinIO cluster is deployed using the containerization technology Docker, and the cluster is orchestrated and managed using Kubernetes. Configure the access rights of MinIO and JuiceFS so that only authorized users and services can access the storage system. Configure the JuiceFS file system, use MinIO as the backend storage, enable file encryption, and use the AES-256 encryption standard.

[0033] In one embodiment, the construction of the resource computing platform 120 includes: configuring a distributed computing engine; wherein the distributed computing engine includes at least Spark on K8s and Flink on K8s; using the Pod elastic scaler function of Kubernetes to automatically expand and reduce computing resources according to the load of the scheduling task; it can be understood that Spark on K8s and Flink on K8s are installed and configured on the Kubernetes cluster, and support the use of the Pod elastic scaler function of Kubernetes according to the actual shipment requirements of the tenants, and automatically expand and reduce computing resources according to the load of the computing task. Among them, MinIO is a subcluster on k8s.

[0034] In one embodiment, the construction of the task scheduling platform 110 includes: configuring a Helm Chart or Kubernetes YAML file to deploy and manage the scheduling platform, and the configuration content includes at least: name, description, type, version, keywords, resources, open source tools, and custom annotations. It can be understood that the Helm Chart or Kubernetes YAML file is used to deploy and manage the scheduling platform, and the main configuration includes name, description, type, version, appVersion, keywords, sources, dependencies, and annotations.

[0035] In this embodiment, the task scheduling platform 110 is used to receive scheduling tasks issued by different tenants, and perform task scheduling according to the task characteristics and task configuration table corresponding to the scheduling tasks; specifically, the task configuration information corresponding to the task ID of the scheduling task and the task characteristics corresponding to the scheduling task can be matched in the task configuration table, and the task scheduling can be performed according to the task configuration information; in this embodiment, the matching method is not limited, and it can be through traversal search or through keyword search identified by ID, and this embodiment does not make any restrictions here.

[0036] In this embodiment, the resource computing platform 120 is used to call the pre-configured integrated algorithm according to the task scheduling to perform task processing and obtain the corresponding data processing results; it can be understood that the resource scheduling platform integrates multiple algorithms, such as the jackknife distance algorithm, the Kriging algorithm, the chip volatility heartbeat check, etc., and uses cloud computing resources for on-demand calling to achieve efficient calculation and analysis. For example, when executing specifically, the scheduling tasks may include but are not limited to yield anomaly comparison, wafer special pattern judgment, continuous abnormal die failure judgment, equipment site yield and distribution anomalies, univariate distribution of tests, binary and multivariate correlation anomalies, and different tasks will call different integrated algorithms for execution. It should be noted that the algorithm integrated by the resource computing platform supports user updates and custom configurations, that is, through the user-friendly interface developed, it supports the configuration and import of user-defined algorithms, and performs compatible processing through the rule engine to achieve flexible rule management.

[0037] In this embodiment, the distributed storage platform 130 is used to encrypt the data processing results and store the encrypted data processing results in the object storage space. In this embodiment, the method of encrypting the data processing results may include symmetric encryption and asymmetric encryption. When using symmetric encryption, the key used to encrypt and decrypt the data processing results is the same key. When using asymmetric encryption, two keys are required, a public key and a private key. The public key and the private key are a pair. If the data is encrypted with a public key, it can only be decrypted with the corresponding private key. If the data is encrypted with a private key, it can only be decrypted with the corresponding public key. In this embodiment, asymmetric encryption may include: digital signature algorithm, elliptic curve encryption; symmetric encryption may include: AES256 encryption standard.

[0038] The technical solution of the embodiment of the present invention is to schedule tasks according to the task characteristics and task configuration table corresponding to the scheduled tasks through the task scheduling platform, and the resource computing platform calls the pre-configured integrated algorithm according to the task scheduling to perform task processing to obtain the corresponding data processing results. The distributed storage platform encrypts the data processing results and stores the encrypted data processing results in the object storage space. Therefore, through the mutual cooperation between the task scheduling platform, the resource computing platform and the distributed storage platform, it can meet the needs of real-time processing, implementation analysis and the complex data analysis needs of various semiconductor companies, and ensure the data security and privacy of each tenant.

[0039] In one embodiment, the distributed storage platform 130 includes a data warehouse, which is divided into a data collection layer, a data storage layer, a data processing layer, an intelligent analysis layer, and an application service layer; data transmission and interaction are performed between the layers through standardized interfaces; the data warehouse is configured with role-based access control permissions;

[0040] The data storage layer includes an object storage space; in the object storage space, an independent data storage space is configured for each tenant, and the data processing results corresponding to each tenant are stored in the configured data storage space to isolate the data between the tenants;

[0041] The data storage space corresponding to each tenant corresponds to a corresponding tenant ID, and access control authority is pre-configured in the data storage space so that each tenant can access the corresponding data storage space through the tenant ID.

[0042] In this embodiment, the distributed storage platform 130 includes a data warehouse, which is divided into a data collection layer, a data storage layer, a data processing layer, an intelligent analysis layer and an application service layer; data transmission and interaction are performed between each layer through a standardized interface; wherein the data storage layer includes an object storage space; in the object storage space, an independent data storage space is configured for each tenant, and the data processing results corresponding to each tenant are stored in the configured data storage space to isolate the data between each tenant. In this embodiment, the data storage space corresponding to each tenant corresponds to the corresponding tenant ID identification, and access control permissions are pre-configured in the data storage space so that each tenant can access the corresponding data storage space through the tenant ID identification, so as to protect the data security and privacy of each tenant through data isolation, access control, data transmission, storage encryption, and log auditing and monitoring in a multi-tenant environment.

[0043] In this embodiment, role-based access control permissions are configured in the data warehouse. The role-based access control permissions can be understood as a method of managing access permissions, which manages user access permissions to systems, networks, or resources based on the user's role in a team or a larger department. It can be understood that one or more roles are assigned to each tenant, permissions are assigned to each tenant role, and the tenant becomes a valid member of a role, obtains access permissions, and then accesses various layers in the data warehouse through the access permissions, thereby realizing a multi-level permission management mechanism.

[0044] In this embodiment, the data storage space corresponding to each tenant corresponds to a corresponding tenant ID identifier, and access control permissions are pre-configured in the data storage space so that each tenant can access the corresponding data storage space through the tenant ID identifier, that is, each tenant can access a fixed data storage space through the corresponding tenant ID, thereby ensuring data isolation between tenants.

[0045] In one embodiment, the distributed storage platform 130 further includes: a deployed log monitoring platform;

[0046] The log monitoring platform is used to collect the audit logs collected by the preset audit log collector tool and the platform log data reported by the task scheduling platform, resource computing platform, and distributed storage platform, so as to centrally manage the platform log data.

[0047] The preset audit log collector tool may include tools such as auditbeat, etc. Auditbeat is a lightweight data collector that can collect and centrally audit events and monitor file integrity.

[0048] In this embodiment, the log monitoring platform deployed in the distributed storage platform is used to collect the audit logs collected by the preset audit log collector tool and the platform log data reported by the task scheduling platform, resource computing platform, and distributed storage platform, so as to centrally manage the platform log data. It can be understood that a log monitoring platform such as ELK is deployed on the Kubernetes cluster to collect the audit logs collected by tools such as auditbeat and the log data reported by each functional component (such as the scheduling platform, computing system, storage system, microservice application, etc.), so as to realize the centralized management, monitoring and auditing functions of the log. In this embodiment, the audit log can be understood as the centralized collection of security events, user access records, operation logs, operation status and other types of information in the public cloud of big data analysis.

[0049] In one embodiment, the distributed storage platform 130 further includes: a data modeling module;

[0050] Among them, the data modeling module is used to model the processed data processing results based on the preset tenant requirements, forming at least two microservices with data functional characteristics, so that the tenants use the microservices to perform demand processing according to the data functional characteristics corresponding to the at least two microservices to obtain corresponding processing results, and use the preset container orchestration tool to enable the microservices to be automatically expanded and deployed; wherein each microservice corresponds to a data functional characteristic; wherein the data functional characteristics include at least: yield anomaly comparison, wafer special pattern judgment, continuous abnormal failure judgment, equipment site yield and distribution anomalies, univariate distribution anomaly detection of tests, binary distribution anomaly detection and multivariate distribution anomaly detection.

[0051] Among them, the preset container orchestration tool is a container orchestration tool such as Kubernetes, which can realize automatic expansion and deployment of microservices.

[0052] In this embodiment, each microservice corresponds to a data function feature; wherein the data function features at least include: yield anomaly comparison, wafer special pattern judgment, continuous abnormal failure judgment, equipment site yield and distribution anomaly, and test distribution anomaly detection. Exemplarily, microservice 1 is used for yield anomaly comparison, microservice 2 is used for wafer special pattern judgment, and microservice 3 is used for continuous abnormal failure judgment.

[0053] Among them, distribution anomaly detection can include: univariate distribution anomaly detection, bivariate distribution anomaly detection and multivariate distribution anomaly detection. Among them, univariate distribution anomaly detection refers to identifying outliers in a univariate data set, such as Z-score. When two or more variables are involved, because the interaction between variables needs to be considered, bivariate and multivariate correlation anomaly detection is required, such as principal component analysis, local anomaly factor method, etc.

[0054] In this embodiment, different tenants may have different corresponding requirements. Therefore, the processed data processing results are modeled according to the tenant requirements of different tenants to form at least two microservices with data function characteristics, so that the tenants use the microservices to process the requirements according to the data function characteristics corresponding to the at least two microservices to obtain the corresponding processing results, and the preset container orchestration tool is used to enable the microservices to be automatically expanded and deployed. It can be understood that the data modeling function is split into multiple microservices, each of which is responsible for the modeling and conversion of a specific data type or data source. For example, the yield management microservice defines the calculation method of the initial test and retest according to the requirements of the yield calculation, and the statistical method of the final test pass and fail is modeled. It is responsible for reading the data of the data storage layer, saving the intermediate reusable results to the data processing layer, and storing the final calculation results to the intelligent analysis layer. It can also access the data quality microservice to realize the monitoring and management of the yield.

[0055] In one embodiment, the distributed storage platform further includes: a unified data interface module;

[0056] Among them, the unified data interface module is used for at least two microservices, each microservice uses the API gateway to uniformly manage the data access interface, and automatically deploys and updates each microservice and the API gateway to uniformly manage the data access interface through a preset continuous integration and continuous deployment (CI / CD) pipeline.

[0057] Among them, the preset CI / CD pipeline can be understood as the process of driving software development by building, testing and deploying code. Its goal is to minimize human errors and maintain the consistency of the software release process through automation. It aims to improve the efficiency, quality and delivery speed of software development through automated processes and tools.

[0058] In this embodiment, for at least two microservices, each microservice uses an API gateway to uniformly manage data access interfaces, uses an API gateway to uniformly manage data access interfaces, provides a unified API entry, and improves the security and scalability of data access. Provides a CI / CD continuous integration and continuous deployment (CI / CD) pipeline for machine learning, completes automatic processing of training data, automatically adjusts and scores algorithms and hyperparameters, realizes automation of microservice and service interface deployment updates, and enables all data access modules to be continuously optimized and efficiently operated and maintained.

[0059] In one embodiment, the distributed storage platform further includes: a monitoring module;

[0060] Among them, the monitoring module is used to configure a pre-trained data quality monitoring model for at least two microservices according to the type of microservice, so as to monitor the data anomaly of each microservice through the data quality monitoring model to obtain the monitoring result, and through the pre-integrated monitoring tool, whether the data quality indicator of the monitoring result exceeds the preset monitoring indicator, and automatically trigger an alarm when the data quality indicator exceeds the preset monitoring indicator.

[0061] The pre-integrated monitoring tools may include but are not limited to monitoring tools such as Prometheus and Grafana. The preset monitoring indicators are pre-configured normal monitoring indicators, which can be customized based on experience or needs, and this embodiment does not limit this.

[0062] In this embodiment, the type of microservice may include one or at least two, and the data quality monitoring models corresponding to different types are different. It can be understood that for the same type of microservice, the same data quality monitoring model can be used for monitoring. In this embodiment, a pre-trained data quality monitoring model can be configured for at least two microservices according to the type of microservice, so as to monitor the data anomaly of each microservice through the data quality monitoring model to obtain the monitoring result, and through the pre-integrated monitoring tool, whether the data quality index of the monitoring result exceeds the preset monitoring index, and automatically trigger an alarm when the data quality index exceeds the preset monitoring index. It can be understood that each of the above microservices corresponds to a data type or function. For each of the above microservices, the microservices are first divided according to the type. For the same type of microservices, the same data quality monitoring will correspond to each microservice, and for each microservice of different types, different data quality monitoring will correspond to each microservice, so as to use data quality monitoring to monitor the corresponding microservice.

[0063] In one embodiment, an adapter is deployed in the task scheduling platform. Accordingly, the task scheduling platform 110 is also used for:

[0064] Use pre-configured adapters to pre-process scheduling tasks from different data sources; wherein the pre-processing includes: processing scheduling tasks from different data sources into a unified format by configuring a preset language program; different data sources include: data types, data file paths, and file addresses provided by different tenants.

[0065] The preset language program may include but is not limited to Python, SQL and other programs.

[0066] In this embodiment, a preconfigured adapter is used to preprocess the scheduling tasks of different data sources; wherein, the preprocessing includes: processing the scheduling tasks of different data sources into a unified format by configuring a preset language program; different data sources include: data types, data file paths, and file addresses provided by different tenants. It can be understood that a customized adapter is created in the scheduling platform, and the adapter configuration includes access parameters for different data sources. At the same time, programs such as Python and SQL can be configured to process the data into a unified format. The access and processing of different data sources can be conveniently managed through the adapter, and different microservices can configure adapters as needed.

[0067] In one embodiment, to better understand the intelligent big data analysis public cloud in the semiconductor industry, Figure 2 A schematic diagram of the structure of another intelligent big data analysis public cloud for the semiconductor industry provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the intelligent big data analysis public cloud for the semiconductor industry includes: a pre-built task scheduling platform 210, a resource computing platform 220 and a distributed storage platform 230 with encryption features; wherein the task scheduling platform 210 is pre-configured with a task configuration table; the resource computing platform 220 is pre-configured with at least two integrated algorithms, exemplarily, the jackknife distance algorithm, the Kriging algorithm, and the chip volatility heartbeat check algorithm; the distributed storage platform 230 with encryption features includes: a data warehouse 2301, which is divided into a data collection layer, a data storage layer, a data processing layer, an intelligent analysis layer and an application service layer; a deployed log monitoring platform 2302, a data modeling module 2303, a unified data interface module 2304, a monitoring module 2305, and a deployed adapter 2306.

[0068] In this embodiment, the task scheduling platform is used to receive scheduling tasks issued by different tenants, and perform task scheduling according to the task characteristics and task configuration table corresponding to the scheduling tasks. The resource computing platform is used to call the pre-configured integrated algorithm according to the task scheduling to perform task processing to obtain the corresponding data processing results. The distributed storage platform is used to encrypt the data processing results and store the encrypted data processing results in the object storage space; wherein, the object storage space in each data warehouse in the distributed storage platform is used to store data of different tenants, and an independent data storage space is configured for each tenant. The data storage space corresponding to each tenant corresponds to the corresponding tenant ID identification, and access control permissions are pre-configured in the data storage space, and users can access it through role-based access control. In the process of storing the data processing results, the configured encryption algorithm is used to encrypt the data; a log monitoring platform is deployed to centrally manage the audit logs collected by tools such as auditbeat and the log data reported by each functional component; for the processed data processing results, the processed data processing results are modeled based on tenant needs to form multiple microservices with certain characteristics, each microservice corresponds to a data feature, and container orchestration tools such as Kubernetes are used to enable microservices to be automatically expanded and deployed; adapters are deployed in the scheduling platform, and access parameters from different sources are processed through adapters; for the multiple microservices formed, each microservice uses the API gateway to uniformly manage the data access interface, and each microservice and the API gateway uniformly manage the data access interface through the CI / CD pipeline for automatic deployment and update; for each microservice with data functional characteristics, a data quality monitoring model for each microservice is configured to monitor data anomalies for each microservice, and the data quality indicators are monitored in real time through the integrated monitoring tools, and alarms are automatically triggered when anomalies occur.

[0069] In one embodiment, Figure 3 A flowchart of an intelligent big data analysis method for the semiconductor industry provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the method is applied to the intelligent big data analysis public cloud in the semiconductor industry, and the method specifically includes the following steps:

[0070] S310: Receive scheduling tasks issued by different tenants through the task scheduling platform, and perform task scheduling according to the task characteristics and task configuration table corresponding to the scheduling tasks.

[0071] In this embodiment, task scheduling is performed according to the task characteristics corresponding to the scheduled task and the task configuration table, including: matching the task configuration information corresponding to the task ID of the scheduled task and the task characteristics corresponding to the scheduled task in the task configuration table, and scheduling the task according to the task configuration information.

[0072] S320, calling a pre-configured integrated algorithm according to task scheduling through the resource computing platform to perform task processing to obtain corresponding data processing results.

[0073] S330. Encrypt the data processing result through a distributed storage platform, and store the encrypted data processing result in an object storage space.

[0074] In one embodiment, the performing task scheduling according to the task characteristics corresponding to the scheduled task and the task configuration table includes:

[0075] Matching the corresponding task configuration information in the task configuration table according to the task ID of the scheduled task and the task feature corresponding to the scheduled task;

[0076] Task scheduling is performed according to the task configuration information.

[0077] In one embodiment, the distributed storage platform includes a data warehouse, which is divided into a data collection layer, a data storage layer, a data processing layer, an intelligent analysis layer, and an application service layer; data transmission and interaction are performed between the layers through standardized interfaces; the data warehouse is configured with role-based access control permissions;

[0078] The data storage layer includes an object storage space; in the object storage space, an independent data storage space is configured for each tenant, and the data processing results corresponding to each tenant are stored in the configured data storage space, so as to isolate the data between each tenant;

[0079] The data storage space corresponding to each tenant corresponds to a corresponding tenant ID, and access control authority is pre-configured in the data storage space so that each tenant can access the corresponding data storage space through the tenant ID.

[0080] In one embodiment, the distributed storage platform further includes: a deployed log monitoring platform; the method further includes:

[0081] The audit logs collected by the preset audit log collector tool and the platform log data reported by the task scheduling platform, resource computing platform, and distributed storage platform are collected through the log monitoring platform to centrally manage the platform log data.

[0082] In one embodiment, the method further includes:

[0083] The processed data processing results are modeled based on preset tenant requirements by the data modeling module in the distributed storage platform to form at least two microservices with data functional characteristics, so that the tenants use the microservices to perform demand processing according to the data functional characteristics respectively corresponding to the at least two microservices to obtain corresponding processing results, and the microservices can be automatically expanded and deployed by using a preset container orchestration tool;

[0084] Among them, each microservice corresponds to a data function feature; wherein, the data function features include at least: yield anomaly comparison, wafer special pattern judgment, continuous abnormal failure judgment, equipment site yield and distribution anomaly, test univariate distribution anomaly detection, binary distribution anomaly detection and multivariate distribution anomaly detection.

[0085] In one embodiment, the method further includes:

[0086] Through the unified data interface module in the distributed storage platform, for the at least two microservices, each of the microservices uses the API gateway to uniformly manage the data access interface, and through the preset CI / CD pipeline, each microservice and the API gateway uniformly manage the data access interface to automatically deploy and update.

[0087] In one embodiment, the method further includes:

[0088] A pre-trained data quality monitoring model is configured for the at least two microservices according to the type of the microservice through the monitoring module in the distributed storage platform, so as to perform data anomaly monitoring on each microservice through the data quality monitoring model to obtain a monitoring result, and through a pre-integrated monitoring tool, whether the data quality indicator of the monitoring result exceeds the preset monitoring indicator, and automatically trigger an alarm when the data quality indicator exceeds the preset monitoring indicator.

[0089] In one embodiment, the integrated algorithms pre-configured in the resource computing platform include at least: a jackknife distance algorithm, a Kriging algorithm, and a chip volatility heartbeat check algorithm.

[0090] In one embodiment, an adapter is deployed in the task scheduling platform. Accordingly, the method further includes: preprocessing the scheduling tasks of different data sources using a pre-configured adapter through the task scheduling platform; wherein the preprocessing includes: processing the scheduling tasks of different data sources into a unified format by configuring a preset language program; different data sources include: data types, data file paths, and file addresses provided by different tenants.

[0091] In one embodiment, the construction of the distributed storage platform includes: using the container engine Docker to run the MinIO instance, using the StatefulSet controller of Kubernetes to deploy the MinIO cluster, and performing the MinIO cluster orchestration; configuring the Kubernetes storage class; configuring the JuiceFS file system, configuring the MinIO cluster as the backend storage, configuring the encryption policy for MinIO, and configuring the SSL certificate in the MinIO instance; integrating the key management system in the MinIO cluster, using the preset external key to encrypt / decrypt the data processing results stored in the object storage, and ensuring the high security of the stored data; configuring the NetworkPolicy of JuiceFS to ensure that only authorized tenants can access and operate the JuiceFS file system;

[0092] The construction of the resource computing platform includes: configuring a distributed computing engine; wherein the distributed computing engine includes at least Spark on K8s and Flink on K8s; using the Pod elastic scaler function of Kubernetes to automatically expand and reduce computing resources according to the load of the scheduled tasks;

[0093] The construction of the task scheduling platform includes: configuring Helm Chart or Kubernetes YAML files to deploy and manage the scheduling platform, and the configuration content includes at least: name, description, type, version, keywords, resources, open source tools, and custom comments.

[0094] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0095] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An intelligent big data analysis public cloud for the semiconductor industry, characterized in that: The intelligent big data analysis public cloud includes: a pre-built task scheduling platform, a resource computing platform, and a distributed storage platform with encryption features; The scheduling platform pre-configures a task configuration table; the task configuration table includes at least: task ID, task type, task triggering method, task priority, task resource configuration, dependency, attribute information of task input data and output data, task retry strategy after execution failure, and task log configuration; The task scheduling platform is used to receive the scheduling tasks issued by different tenants and perform task scheduling according to the task characteristics corresponding to the scheduling tasks and the task configuration table; The resource computing platform is used to call a pre-configured integrated algorithm according to the task scheduling to perform task processing and obtain corresponding data processing results; The distributed storage platform is used to encrypt the data processing results and store the encrypted data processing results in the object storage space.

2. The intelligent big data analysis public cloud according to claim 1, characterized in that: The distributed storage platform includes a data warehouse, which is divided into a data collection layer, a data storage layer, a data processing layer, an intelligent analysis layer, and an application service layer; data transmission and interaction are performed between the layers through standardized interfaces; the data warehouse is configured with role-based access control permissions; The data storage layer includes an object storage space; in the object storage space, an independent data storage space is configured for each tenant, and the data processing results corresponding to each tenant are stored in the configured data storage space, so as to isolate the data between the tenants; The data storage space corresponding to each tenant corresponds to a corresponding tenant ID, and access control authority is pre-configured in the data storage space so that each tenant can access the corresponding data storage space through the tenant ID.

3. The intelligent big data analysis public cloud according to claim 1, characterized in that: The distributed storage platform also includes: a deployed log monitoring platform; The log monitoring platform is used to collect the audit logs collected by the preset audit log collector tool and the platform log data reported by the task scheduling platform, resource computing platform, and distributed storage platform, so as to centrally manage the platform log data.

4. The intelligent big data analysis public cloud according to claim 1, characterized in that: The distributed storage platform further includes: a data modeling module; The data modeling module is used to model the processed data processing results based on preset tenant requirements to form at least two microservices with data functional characteristics, so that the tenants use the microservices to perform demand processing according to the data functional characteristics respectively corresponding to the at least two microservices to obtain corresponding processing results, and use the preset container orchestration tool to enable the microservices to be automatically expanded and deployed; Among them, each microservice corresponds to a data function feature; wherein, the data function features at least include: yield anomaly comparison, wafer special pattern judgment, continuous abnormal failure judgment, equipment site yield and distribution anomaly, and test distribution anomaly detection.

5. The intelligent big data analysis public cloud according to claim 4, characterized in that: The distributed storage platform further includes: a unified data interface module; The unified data interface module is used for each of the at least two microservices to use the API gateway to uniformly manage the data access interface, and to automatically deploy and update each microservice and the API gateway to uniformly manage the data access interface through a preset CI / CD pipeline.

6. The intelligent big data analysis public cloud according to claim 4, characterized in that: The distributed storage platform further includes: a monitoring module; The monitoring module is used to configure a pre-trained data quality monitoring model for the at least two microservices according to the type of the microservice, so as to perform data anomaly monitoring on each microservice through the data quality monitoring model to obtain a monitoring result, and to use a pre-integrated monitoring tool to determine whether the data quality indicator of the monitoring result exceeds a preset monitoring indicator, and to automatically trigger an alarm when the data quality indicator exceeds the preset monitoring indicator.

7. The intelligent big data analysis public cloud according to claim 1, characterized in that: The integrated algorithms pre-configured in the resource computing platform include at least: a jackknife algorithm, a Kriging algorithm, and a chip volatility heartbeat check.

8. The intelligent big data analysis public cloud according to claim 1, characterized in that: The adapter is deployed in the task scheduling platform. Accordingly, the task scheduling platform is also used for: Use a pre-configured adapter to pre-process the scheduling tasks of different data sources; wherein the pre-processing includes: processing the scheduling tasks of different data sources into a unified format by configuring a preset language program; different data sources include: data types, data file paths, and file addresses provided by different tenants.

9. The intelligent big data analysis public cloud according to claim 1, characterized in that: The construction of the distributed storage platform includes: using the container engine Docker to run the MinIO instance, using the StatefulSet controller of Kubernetes to deploy the MinIO cluster, and performing the MinIO cluster orchestration; configuring the Kubernetes storage class; configuring the JuiceFS file system, configuring the MinIO cluster as the backend storage, configuring the encryption policy for MinIO, and configuring the SSL certificate in the MinIO instance; integrating the key management system in the MinIO cluster, using the preset external key to encrypt / decrypt the data processing results stored in the object storage to ensure the high security of the stored data; configuring the NetworkPolicy of JuiceFS to enable authorized tenants to access and operate the JuiceFS file system; The construction of the resource computing platform includes: configuring a distributed computing engine; wherein the distributed computing engine includes at least Spark on K8s and Flink on K8s; using the Pod elastic scaler function of Kubernetes to automatically expand and reduce computing resources according to the load of the scheduled tasks; The construction of the task scheduling platform includes: configuring Helm Chart or Kubernetes YAML files to deploy and manage the scheduling platform, and the configuration content includes at least: name, description, type, version, keywords, resources, open source tools, and custom comments.

10. An intelligent big data analysis method for the semiconductor industry, characterized in that: An intelligent big data analysis public cloud applied to the semiconductor industry, the intelligent big data analysis public cloud includes: a pre-built task scheduling platform, a resource computing platform, and a distributed storage platform with encryption features, which serve as the basic platform of the intelligent data analysis system; The scheduling platform pre-configures a task configuration table; the task configuration table includes at least: task ID, task type, task triggering method, task priority, task resource configuration, dependency, attribute information of task input data and output data, task retry strategy after execution failure, and task log configuration; The intelligent big data analysis method comprises: Receiving scheduling tasks issued by different tenants through the task scheduling platform, and performing task scheduling according to task features corresponding to the scheduling tasks and the task configuration table; The resource computing platform calls a pre-configured integrated algorithm according to the task scheduling to perform task processing to obtain corresponding data processing results; The data processing result is encrypted through the distributed storage platform, and the encrypted data processing result is stored in the object storage space.

11. The intelligent big data analysis method according to claim 10, characterized in that: The performing task scheduling according to the task characteristics corresponding to the scheduled task and the task configuration table includes: Matching the corresponding task configuration information in the task configuration table according to the task ID of the scheduled task and the task feature corresponding to the scheduled task; Task scheduling is performed according to the task configuration information.

Images