Data processing method and device, equipment, medium and product
By determining the data processing tasks and their dependencies and executing the tasks in sequence, the problems of excessive storage space and low efficiency in semiconductor test data processing are solved, and efficient processing of massive data is achieved.
Patent Information
- Application Number
- CN202411181710.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-27
AI Technical Summary
When facing massive data, existing semiconductor test data processing solutions have low real-time analysis efficiency and offline analysis occupies too much storage space, so they cannot effectively process large-scale data.
By obtaining the data set, the original test items, the calculation test items and the split test items, determine the data processing tasks and their dependencies, and execute the tasks in turn to reduce the storage space occupancy.
It effectively reduces storage space usage, improves data processing efficiency, and is suitable for the analysis of massive semiconductor test data.
Smart Images

Figure CN119066098A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the field of semiconductor technology, and in particular, to a data processing method, device, equipment, medium and product. Background Art
[0002] Semiconductor test data is usually large in scale and has rich and diverse test results. In addition to the original test data, the original test data is also used to calculate some statistical indicators, parameters that meet industry standards, etc. Existing data processing solutions include: real-time analysis and offline analysis.
[0003] Real-time analysis will calculate in real time when data is needed. It avoids large-scale data storage. However, real-time analysis is easily affected by the amount of data. When the amount of data is large, the data processing efficiency is low and real-time analysis cannot be achieved. For semiconductor test data, when a chip on a wafer is used as the analysis dimension, the amount of data usually reaches more than 10 billion. Real-time analysis cannot be used for the analysis of such massive data.
[0004] Offline analysis is to calculate and cache the original test data offline. The amount of semiconductor test data is large, so it will take up too much storage space. Summary of the invention
[0005] Embodiments of the present invention provide a data processing method, apparatus, device, medium and product, which can reduce the amount of occupied storage space.
[0006] According to one aspect of the present invention, there is provided a data processing method, comprising:
[0007] Acquire a data set, at least one original test item, at least one calculated test item composed of the original test items, and at least one split test item composed of the original test items, wherein the data set includes at least one wafer identification;
[0008] Determine a plurality of data processing tasks and dependencies between the data processing tasks according to at least one wafer identification, at least one original test item, at least one calculated test item composed of the original test item, and at least one split test item composed of the original test item;
[0009] According to the dependency relationship between the data processing tasks and the data set, the data processing tasks are executed in sequence to obtain the data processing results corresponding to the data processing tasks.
[0010] According to another aspect of the present invention, there is provided a data processing device, the data processing device comprising:
[0011] An acquisition module, configured to acquire a data set, at least one original test item, at least one calculated test item composed of the original test items, and at least one split test item composed of the original test items, wherein the data set includes at least one wafer identification;
[0012] A task generation module, used to determine a plurality of data processing tasks and dependencies between the data processing tasks according to at least one wafer identification, at least one original test item, at least one calculation test item composed of the original test item, and at least one split test item composed of the original test item;
[0013] The task execution module is used to execute each data processing task in sequence according to the dependency relationship between the data processing tasks and the data set, and obtain the data processing results corresponding to each data processing task.
[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0015] at least one processor; and
[0016] a memory communicatively connected to the at least one processor; wherein,
[0017] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the data processing method described in any embodiment of the present invention.
[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method described in any embodiment of the present invention when executed.
[0019] According to another aspect of the present invention, a computer program product is provided. When the computer program is executed by a processor, the computer program implements the data processing method as described in any one of the embodiments of the present invention.
[0020] The embodiment of the present invention acquires a data set, at least one original test item, at least one calculated test item composed of the original test items, and at least one split test item composed of the original test items, wherein the data set includes at least one wafer identification; determines multiple data processing tasks and dependencies between the data processing tasks based on the at least one wafer identification, at least one original test item, at least one calculated test item composed of the original test items, and at least one split test item composed of the original test items; executes each data processing task in sequence based on the dependencies between the data processing tasks and the data set to obtain data processing results corresponding to each data processing task, thereby reducing the amount of storage space occupied.
[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0023] Figure 1 is a flow chart of a data processing method in an embodiment of the present invention;
[0024] Figure 2 is a flow chart of another data processing method in an embodiment of the present invention;
[0025] Figure 3 is a structural schematic diagram of a data processing device in an embodiment of the present invention;
[0026] Figure 4 It is a structural schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION
[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0029] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0030] Embodiment 1
[0031] Figure 1 A flowchart of a data processing method provided by an embodiment of the present invention. This embodiment is applicable to data processing. The method can be executed by a data processing device in an embodiment of the present invention. The device can be implemented in software and / or hardware. Figure 1 As shown, the method specifically comprises the following steps:
[0032] S110, acquiring a data set, at least one original test item, at least one calculated test item composed of the original test items, and at least one split test item composed of the original test items.
[0033] The data set includes at least one wafer identification. The data set also includes: semiconductor test results corresponding to at least one wafer identification. For example, the data set includes: semiconductor test results of multiple test items corresponding to wafer identification A, semiconductor test results of multiple test items corresponding to wafer identification B, and semiconductor test results of multiple test items corresponding to wafer identification C.
[0034] The at least one original test item may be: at least one original test item identifier, and the original test item identifier may be an original test item name. The calculation test item composed of the original test items is used to represent: calculating multiple original test items. For example, the calculation test item may be: the name of original test item 1 + the name of original test item 2. The split test item composed of the original test items is used to represent: splitting multiple original test items. For example, the split test item may be: split(Test Item,':',2)as num.
[0035] The data set is used to represent a predefined data range. Specifically, the data set can be obtained by querying the total semiconductor test data (test results) according to the key information input by the user to obtain the data set corresponding to the key information.
[0036] Optionally, obtain a dataset, including:
[0037] In response to a preset trigger operation, displaying the main interface;
[0038] In response to a trigger operation on a new data set control in the main interface, displaying a data set selection interface;
[0039] In response to a trigger operation on a preview control in a data set selection interface, a data set corresponding to the test attribute information input by a user is obtained, and the data set is displayed in the data set selection interface, wherein the test attribute information includes at least one of: a test phase, a test time, a batch material number, a test batch, and a wafer identification.
[0040] Among them, the main interface may include: a data set list, a batch deletion control, a filter list control and a new data set control. The data set list includes: data set name, data table source, test phase, creator, creation time, modifier, modification time and corresponding operation controls (operation controls may include: enter icon analysis control, data set setting control, delete control and more controls).
[0041] Among them, the data set selection interface includes: a data range selection area and a data selection overview area. The data range selection area includes: a preview control, a reset control, and a test attribute information input area. The data selection overview area is used to display data set details (data set details include: test stage, batch material number, test batch, wafer group identification, wafer identification, total number of chips, number of chips passed the test, test pass rate, etc.). The user can directly enter the test attribute information in the test attribute information input area, and the user can also select the test attribute information through the drop-down box.
[0042] It should be noted that a batch number contains multiple wafer groups, a wafer group contains multiple wafers, and a wafer contains multiple chips. The minimum granularity for defining a data set is the wafer.
[0043] Optionally, obtain at least one original test item, including:
[0044] In response to a trigger operation on a next step control in the data set selection interface, a test item selection interface is displayed, wherein the test item selection interface includes: a query area and a test item list display area;
[0045] In response to a triggering operation on a determination control in the query area, updating the test item list in the test item list display area according to the query information input by the user;
[0046] In response to a selection operation on a test item in the test item list, the selected test item is determined as an original test item.
[0047] The query area includes: a query information editing area. The test item list includes: a test item name and a test item type.
[0048] It should be noted that querying test items is to query test items based on the test item names. The query area may also include: test item selection methods, such as customizable (directly click on the required test items from the total list), or by entering regular expressions (filtering the list through regular expressions to obtain the required test items. For example, it may be: test items that do not contain XXXX in the test item name, or test items that contain XXX in the test item name), which is not limited in this embodiment of the present invention.
[0049] It should be noted that after determining the data set, that is, after defining the wafer range, the test item range is further defined. Each chip on each wafer may undergo multiple tests. When processing data, only some test items can be selected. Optionally, at least one calculated test item composed of the original test items is obtained, including:
[0050] In response to a triggering operation on a next step control in a test item selection interface, displaying a calculation test item setting interface;
[0051] In response to a triggering operation on a newly added calculation test item control in a calculation test item setting interface, a calculation test item editing interface is displayed, wherein the calculation test item editing interface includes: a first editing area, a first test item list display area, and a first function display area;
[0052] In response to a triggering operation on a test item in the first test item list display area, the triggered test item is displayed in the first editing area; in response to a triggering operation on a function in the first function display area, the triggered function is displayed in the first editing area;
[0053] In response to a triggering operation on a determination control in the computing test item editing interface, a computing test item is generated according to the triggered test item and the triggered function displayed in the first editing area.
[0054] The calculation test item setting interface includes: a calculation test item list display area and a new calculation test item control. The calculation test item editing interface includes: a calculation test item name editing area, a first editing area, a first test item list display area and a first function display area.
[0055] The first test item list display area is used to display the test item list selected in the previous step. The first function display area is used to display the calculation formula list, for example, the calculation formula list can be as shown in the first column of Table 1 (the second column of Table 1 is used to explain the meaning of each calculation formula):
[0056] Table 1
[0057]
[0058]
[0059] Specifically, in response to a trigger operation on a test item in the first test item list display area, the triggered test item is displayed in the first editing area, and in response to a trigger operation on a function in the first function display area, the triggered function is displayed in the first editing area. The method may be: when a user double-clicks a test item in the first test item list, the test item is triggered, and the triggered test item is displayed in the first editing area. After the function is triggered, the function is displayed after the pre-triggered test item. For example, after the user double-clicks test item 1 in the first test item list, the user double-clicks the summation function, and the user double-clicks test item 2 in the first test item list, then the following is displayed in the first editing area: test item 1 + test item 2.
[0060] Specifically, the method of generating the calculation test item according to the triggered test items and the triggered functions displayed in the first editing area may be: according to the user's triggering order, the triggered test items and functions are sequentially spliced to obtain the calculation test item. For example, after the user double-clicks the test item 1 in the first test item list, the user double-clicks the summation function, and the user double-clicks the test item 2 in the first test item list, then the calculation test item is: test item 1 + test item 2.
[0061] Optionally, obtain at least one split test item composed of the original test item, including:
[0062] In response to a triggering operation of a next step control in the calculation test item setting interface, displaying a split test item setting interface;
[0063] In response to a triggering operation on a newly added split test item control in the split test item setting interface, a split test item editing interface is displayed, wherein the split test item editing interface includes: a second editing area, a second test item list display area, and a second function display area;
[0064] In response to a triggering operation on a test item in the second test item list display area, displaying the triggered test item in the second editing area, and in response to a triggering operation on a function in the second function display area, displaying the triggered function in the editing area;
[0065] In response to a triggering operation on a determination control in the split test item editing interface, a split test item is generated according to the triggered test item and the triggered function displayed in the second editing area.
[0066] The split test item setting interface includes: a split test item list display area and a new split test item control. The split test item editing interface includes: a split test item name editing area, a second editing area, a second test item list display area, and a second function display area.
[0067] The second test item list display area is used to display the test item list selected in the previous step. The second function display area is used to display the calculation formula list, for example, the calculation formula list can be as shown in the first column of Table 2 (the second column of Table 2 is used to explain the meaning of each calculation formula):
[0068] Table 2
[0069]
[0070]
[0071] It should be noted that if the test item name is 123:test, after calculation by formula 1, the test item name is changed to test, and after calculation by the second formula, 123 is used as an attribute of this test item. Some companies will name the test items as: voltage test: 5v, voltage test: 10v, etc. After the test items are split, the name can be unified as: voltage test, and the attributes are: 5v, 10v.
[0072] S120, determining multiple data processing tasks and dependencies between the data processing tasks based on at least one wafer identification, at least one original test item, at least one calculation test item composed of the original test item, and at least one split test item composed of the original test item.
[0073] The data processing tasks may include: a first type of data processing task and a second type of data processing task. The dependency relationship between the data processing tasks may be: the first type of data processing task is a predecessor task of the second type of data processing task.
[0074] Specifically, according to at least one wafer identification, at least one original test item, at least one computing test item composed of the original test items, and at least one split test item composed of the original test items, a method for determining multiple data processing tasks and dependencies between the data processing tasks can be: according to each computing test item and at least one wafer identification, a plurality of first-type data processing tasks are generated, wherein the total number of the first-type data processing tasks is the same as the number of computing test items; according to each split test item and at least one wafer identification, a plurality of second-type data processing tasks are generated, wherein the total number of the second-type data processing tasks is the same as the number of split test items, and the first-type data processing tasks are predecessor tasks of the second-type data processing tasks.
[0075] Optionally, determining a plurality of data processing tasks and dependencies between the data processing tasks according to at least one wafer identification, at least one original test item, at least one calculation test item composed of the original test item, and at least one split test item composed of the original test item includes:
[0076] generating a plurality of first-type data processing tasks according to each computing test item and at least one wafer identification, wherein the total number of the first-type data processing tasks is the same as the number of the computing test items;
[0077] A plurality of second-type data processing tasks are generated according to each split test item and at least one wafer identification, wherein the total number of the second-type data processing tasks is the same as the number of the split test items, and the first-type data processing tasks are predecessor tasks of the second-type data processing tasks.
[0078] It should be noted that the first type of data processing task is a predecessor task of the second type of data processing task, that is, the first type of data processing task needs to be processed in advance, and when processing the second type of data processing task, the processing result of the first type of data processing task is needed.
[0079] Specifically, multiple first-type data processing tasks are generated according to each computing test item and at least one wafer identification. The method can be: one computing test item generates one first-type data processing task, and each first-type data processing task is a data processing task for all selected wafer identifications.
[0080] Specifically, multiple second-type data processing tasks are generated according to each split test item and at least one wafer identification. The method can be: one split test item generates one second-type data processing task, and each second-type data processing task is a data processing task for all selected wafer identifications.
[0081] Optionally, executing each data processing task in sequence according to the dependency relationship between the data processing tasks and the data set to obtain a data processing result corresponding to each data processing task includes:
[0082] After executing the first type of data processing task, executing the second type of data processing task to obtain a processing result list;
[0083] Among them, executing the first type of data processing task includes: obtaining each original test item contained in the calculation test item corresponding to the first type of data processing task; querying the data set to obtain the test results of each original test item corresponding to the wafer identification; performing a first calculation on the test results of each original test item corresponding to the wafer identification according to the function in the calculation test item to obtain the processing result, and generating a first list according to the processing result;
[0084] Executing the second type of data processing task includes: obtaining each original test item contained in the split test item corresponding to the second type of data processing task; performing a second calculation on the identification information of each original test item according to the function in the split test item to obtain attribute information, adding the attribute information to the first list to obtain a processing result list.
[0085] The identification information of the original test item may be the name of the original test item, and the identification information of the original test item may carry attribute information. For example, the name of the original test item may be: voltage test: 5v. The 5v in the name of the original test item is attribute information.
[0086] It should be noted that during the execution of the first type of data processing task, the method for querying the data set and obtaining the test results of each original test item corresponding to the wafer identification can be: querying the data set according to the wafer identification and the test item name to obtain the test results of each original test item corresponding to the wafer identification.
[0087] For example, the calculation test items corresponding to the first type of data processing task may include: original test item 1 and original test item 2, the selected wafers are identified as wafer A and wafer B, and the data set is queried to obtain the test results of wafer A when performing original test item 1, the test results of wafer B when performing original test item 1, the test results of wafer A when performing original test item 2, and the test results of wafer B when performing original test item 2. A first calculation is performed based on the test results of wafer A when performing original test item 1, the test results of wafer B when performing original test item 1, the test results of wafer A when performing original test item 2, and the test results of wafer B when performing original test item 2, and the function in the calculation test item to obtain a processing result, and a first list is generated based on the processing result.
[0088] It should be noted that, in the process of executing the second type of data processing task, the identification information of the test item is split, and the information obtained after the split is determined as an attribute. For example, if the identification information of the test item is the name of the test item, the names of the test items are: voltage test: 5v, voltage test: 10v. The names of the test items are split, and after the test items are split, the names of the test items can be unified as: voltage test, and the attribute information is 5v and 10v respectively.
[0089] S130, executing each data processing task in sequence according to the dependency relationship between the data processing tasks and the data set, and obtaining a data processing result corresponding to each data processing task.
[0090] Specifically, according to the dependency relationship and data set between the data processing tasks, the data processing tasks are executed in sequence, and the data processing results corresponding to the data processing tasks can be obtained by: slicing each data processing task to obtain multiple data processing task slices; determining the dependency relationship between multiple data processing task slices according to the dependency relationship between the data processing tasks; executing each data processing task slice in sequence according to the dependency relationship and data set of each data processing task slice to obtain the data processing results corresponding to each data processing task slice; determining the data processing results corresponding to each data processing task according to the data processing results corresponding to each data processing task slice and the dependency relationship between each data processing task slice. According to the dependency relationship and data set between the data processing tasks, the data processing tasks are executed in sequence, and the data processing results corresponding to each data processing task can also be obtained by: submitting each data processing task to the message queue, waiting for the executor to fetch the message, and performing the next step of processing. Multiple executors can be deployed on different machines. The executor monitors the message queue. When a task to be processed is found, the message will be retrieved and processed.
[0091] Optionally, executing each data processing task in sequence according to the dependency relationship between the data processing tasks and the data set to obtain a data processing result corresponding to each data processing task includes:
[0092] Slicing each data processing task according to the total number of wafer identifications to obtain a plurality of data processing task slices and dependencies between the data processing task slices, wherein the total number of the data processing task slices is the same as the total number of wafer identifications;
[0093] According to the dependency relationship and data set of each data processing task slice, each data processing task slice is executed in sequence to obtain the data processing results corresponding to each data processing task slice;
[0094] The data processing results corresponding to each data processing task slice are determined based on the data processing results corresponding to each data processing task slice and the dependency relationship between the data processing task slices.
[0095] Among them, the dependency relationship between the data processing task slices can be determined by the data processing task identifier carried by each data processing task slice. For example, if the data processing task identifiers carried by each data processing task slice are the same, that is, each data processing task slice belongs to the same data processing task. If the data processing task identifiers carried by each data processing task slice are different. The dependency relationship between the data processing task slices carrying the same data processing task identifier is determined to belong to the same data processing task.
[0096] Specifically, each data processing task is sliced according to the total number of wafer identifiers to obtain multiple data processing task slices and the dependency relationship between each data processing task slice can be: if the total number of wafer identifiers is M, then each data processing task is sliced to obtain M data processing task slices, and the dependency relationship between each data processing task slice belonging to the same data processing task is the same as that between the data processing tasks. The dependency relationship between each data processing task slice belonging to different data processing tasks is the same as the dependency relationship between data processing tasks, where M is a positive integer.
[0097] In a specific example, if the total number of wafer identifications is 4, the data processing tasks include: data processing task R and data processing task T. Data processing task R is the predecessor task of data processing task T. Data processing task R is sliced to obtain 4 data processing task slices r. Data processing task T is sliced to obtain 4 data processing task slices t. The dependency between each data processing task slice r is that they belong to the same data processing task. The dependency between each data processing task slice t is that they belong to the same data processing task. The dependency between data processing task slice r and data processing task slice t is the same as the dependency between data processing task R and data processing task T, that is, data processing task slice r is the predecessor task of data processing task t.
[0098] Specifically, according to the dependency relationship and data set of each data processing task slice, each data processing task slice is executed in sequence to obtain the data processing results corresponding to each data processing task slice: if each data processing task slice belongs to the same data processing task, the data processing task slices are executed in parallel; if each data processing task slice belongs to different data processing tasks, first, according to the data set, each data processing task slice belonging to the predecessor task is executed, and then, according to the data set, each data processing task slice belonging to the successor task is executed to obtain the data processing results corresponding to each data processing task slice.
[0099] Specifically, according to the data processing results corresponding to each data processing task slice and the dependency relationship between each data processing task slice, the method for determining the data processing results corresponding to each data processing task can be: after obtaining the data processing results corresponding to each data processing task slice, according to the dependency relationship between each data processing task slice, splicing the data processing results corresponding to each data processing task slice to obtain the data processing results corresponding to each data processing task.
[0100] In a specific example, if the total number of wafer identifications is 4, the data processing tasks include: data processing task R and data processing task T, data processing task R is the predecessor task of data processing task T, data processing task R is sliced to obtain 4 data processing task slices r, data processing task T is sliced to obtain 4 data processing task slices t, the 4 data processing task slices r are first executed in parallel to obtain data processing results corresponding to the 4 data processing task slices r, and then the 4 data processing task slices t are executed in parallel to obtain data processing results corresponding to the 4 data processing task slices t. The data processing results corresponding to the 4 data processing task slices r are spliced to obtain the data processing results corresponding to the data processing task R, and the data processing results corresponding to the 4 data processing task slices t are spliced to obtain the data processing results corresponding to the data processing task T.
[0101] It should be noted that when slicing the data processing task, the data processing task slices belonging to the same data processing task can be executed in parallel. If two data processing task slices belong to different data processing tasks and there is a dependency relationship between the two data processing tasks, the two data processing task slices need to be executed serially.
[0102] Optionally, the data processing tasks include: first type data processing tasks and second type data processing tasks;
[0103] Each data processing task is sliced according to the total number of wafer identifiers to obtain a plurality of data processing task slices and dependencies between the data processing task slices, including:
[0104] Slicing each first-type data processing task according to the total number of wafer identifications to obtain a plurality of data processing task slices corresponding to the first-type data processing task;
[0105] Slicing each second-type data processing task according to the total number of wafer identifications to obtain a plurality of data processing task slices corresponding to the second-type data processing task;
[0106] According to the dependency relationship between the first type of data processing tasks and the second type of data processing tasks, the dependency relationship between multiple data processing task slices corresponding to the first type of data processing tasks and multiple data processing task slices corresponding to the second type of data processing tasks is determined.
[0107] It should be noted that, when slicing the first type of data processing task, the number of data processing task slices obtained is the same as the total number of wafer identifications. When slicing the second type of data processing task, the number of data processing task slices obtained is the same as the total number of wafer identifications. Specifically, according to the dependency relationship between the first type of data processing task and the second type of data processing task, the method for determining the dependency relationship between multiple data processing task slices corresponding to the first type of data processing task and multiple data processing task slices corresponding to the second type of data processing task can be: if the first data processing task slice belongs to the first type of data processing task, and the second data processing task slice belongs to the second type of data processing task, then the dependency relationship between the first data processing task slice and the second data processing task slice is determined as the first data processing task slice being the predecessor task of the second data processing task slice.
[0108] Optionally, executing each data processing task in sequence according to the dependency relationship between the data processing tasks and the data set to obtain a data processing result corresponding to each data processing task includes:
[0109] If all predecessor tasks of the current data processing task have been processed, the current data processing task is executed based on the data set to obtain the data processing result corresponding to the current data processing task, wherein the predecessor task is the data processing task on which the current data processing task depends.
[0110] In a specific example, Figure 2 As shown, an embodiment of the present invention provides a task scheduling system, which includes: a task scheduler, a slicer, an executor and a task monitor, wherein the task scheduler is used to determine which data processing task to execute first (execute the predecessor task first, then the successor task) according to the dependency relationship between the data processing tasks. After the predecessor task is executed, the task scheduler will automatically trigger the execution of the successor task. For example: first check whether the predecessor task of the task to be processed is executed successfully. If it is executed successfully, execute the task to be processed.
[0111] After the task scheduler starts a task, the slicer first processes the data (such as Figure 2 For example, slices are made according to the total number of wafer identifiers v to obtain v task slices. The slicer ensures that the results of summarizing each slice after slicing are equivalent to the results obtained by processing the task as a whole according to the original data range.
[0112] In order to improve the concurrency of the system, the embodiment of the present invention uses a message queue to decouple the executor. The task slices cut by the slicer are uniformly submitted to the message queue, waiting for the executor to retrieve the message and proceed to the next step of processing.
[0113] The executor is a data processing module in the embodiment of the present invention. Since the message queue is used for decoupling, multiple executors can be deployed and can be deployed on different machines, so that the concurrency of processing can be flexibly controlled to achieve the management of computing power. In the embodiment of the present invention, V executors can be deployed, each for processing V task slices. The executor will listen to the message queue, and when it finds a slice to be processed, it will retrieve the message and process it. The slicer will store the processed data and the test result data that has not been processed in the cache table. Each data processing task will have a unique cache table, and each executor will share this table.
[0114] The task monitor monitors the health of data processing tasks in real time. If a task times out or fails, it will be retried a limited number of times. After the maximum number of retries, the task will enter a final failure state. In a group of data processing tasks, once a task enters a final failure state, the group of data processing tasks is considered to have failed. The task monitor, as a compensation mechanism in an embodiment of the present invention, can improve the stability of the system.
[0115] In a specific example, the data processing method provided by the embodiment of the present invention includes the following steps:
[0116] 1. Define the data scope and design the processing plan (calculation test items composed of original test items and / or split test items composed of original test items)
[0117] The user first needs to define the data range that needs to be processed. The embodiment of the present invention does not impose strict restrictions on the amount of data, and needs to design a specific processing plan. The embodiment of the present invention supports operations such as splitting, modifying, and combining the original data to obtain new indicators. There is a predecessor and successor relationship between the operations, and the successor task can further process the results of the predecessor task.
[0118] 2. Submit the task
[0119] The data range and processing plan will be converted into a set of processing tasks and submitted to the system to which the embodiment of the present invention belongs. The task scheduler will determine which task to execute first. After the predecessor task is successfully executed, the subsequent task will be automatically scheduled for execution.
[0120] 3. Data range slicing and distribution
[0121] The slicer divides the task into slices and distributes them using the message queue.
[0122] 4. Execute Slicing
[0123] Several executors will monitor the message queue and start specific data processing after receiving the message.
[0124] The embodiment of the present invention improves the parallelism of calculations through a distributed system, and designs a monitoring program as a compensation mechanism to improve the robustness of the system. The processing results are stored in a database cache table as a whole, which is convenient for subsequent use. The embodiment of the present invention makes it possible to process and store massive test results, which has a driving effect on semiconductor data analysis and production decision-making.
[0125] The technical solution of this embodiment obtains a data set, at least one original test item, at least one calculated test item composed of the original test items, and at least one split test item composed of the original test items, wherein the data set includes at least one wafer identification; determines multiple data processing tasks and dependencies between the data processing tasks based on at least one wafer identification, at least one original test item, at least one calculated test item composed of the original test items, and at least one split test item composed of the original test items; executes each data processing task in sequence based on the dependencies between the data processing tasks and the data set to obtain data processing results corresponding to each data processing task, thereby reducing the amount of storage space occupied.
[0126] Embodiment 2
[0127] Figure 3 This is a schematic diagram of the structure of a data processing device provided by an embodiment of the present invention. This embodiment is applicable to data processing. The device can be implemented in software and / or hardware. The device can be integrated in any device that provides data processing functions, such as Figure 3 As shown, the data processing device specifically includes: an acquisition module 310, a task generation module 320 and a task execution module 330.
[0128] Wherein, the acquisition module is used to acquire a data set, at least one original test item, at least one calculated test item composed of the original test items, and at least one split test item composed of the original test items, wherein the data set includes at least one wafer identification;
[0129] A task generation module, used to determine a plurality of data processing tasks and dependencies between the data processing tasks according to at least one wafer identification, at least one original test item, at least one calculation test item composed of the original test item, and at least one split test item composed of the original test item;
[0130] The task execution module is used to execute each data processing task in sequence according to the dependency relationship between the data processing tasks and the data set, and obtain the data processing results corresponding to each data processing task.
[0131] The above-mentioned product can execute the method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0132] Embodiment 3
[0133] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0134] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0135] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0136] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a data processing method.
[0137] In some embodiments, the data processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).
[0138] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0139] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0140] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0141] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0142] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0143] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0144] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0145] An embodiment of the present invention further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the data processing method according to any embodiment of the present invention is implemented.
[0146] In the process of implementation, the computer program product can be written in one or more programming languages or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).
[0147] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A data processing method, characterized in that: include: Acquire a data set, at least one original test item, at least one calculated test item composed of the original test items, and at least one split test item composed of the original test items, wherein the data set includes at least one wafer identification; Determine a plurality of data processing tasks and dependencies between the data processing tasks according to at least one wafer identification, at least one original test item, at least one calculated test item composed of the original test item, and at least one split test item composed of the original test item; According to the dependency relationship between the data processing tasks and the data set, the data processing tasks are executed in sequence to obtain the data processing results corresponding to the data processing tasks.
2. The method according to claim 1, characterized in that Determining a plurality of data processing tasks and dependencies between the data processing tasks according to at least one wafer identification, at least one original test item, at least one calculated test item composed of the original test item, and at least one split test item composed of the original test item includes: generating a plurality of first-type data processing tasks according to each computing test item and at least one wafer identification, wherein the total number of the first-type data processing tasks is the same as the number of the computing test items; A plurality of second-type data processing tasks are generated according to each split test item and at least one wafer identification, wherein the total number of the second-type data processing tasks is the same as the number of the split test items, and the first-type data processing tasks are predecessor tasks of the second-type data processing tasks.
3. The method according to claim 1, characterized in that According to the dependency relationship between the data processing tasks and the data set, the data processing tasks are executed in sequence to obtain the data processing results corresponding to the data processing tasks, including: Slicing each data processing task according to the total number of wafer identifications to obtain a plurality of data processing task slices and dependencies between the data processing task slices, wherein the total number of the data processing task slices is the same as the total number of wafer identifications; According to the dependency relationship and data set of each data processing task slice, each data processing task slice is executed in sequence to obtain the data processing results corresponding to each data processing task slice; The data processing results corresponding to each data processing task slice are determined based on the data processing results corresponding to each data processing task slice and the dependency relationship between the data processing task slices.
4. The method according to claim 3, characterized in that The data processing tasks include: first type data processing tasks and second type data processing tasks; Each data processing task is sliced according to the total number of wafer identifiers to obtain a plurality of data processing task slices and dependencies between the data processing task slices, including: Slicing each first-type data processing task according to the total number of wafer identifications to obtain a plurality of data processing task slices corresponding to the first-type data processing task; Slicing each second-type data processing task according to the total number of wafer identifications to obtain a plurality of data processing task slices corresponding to the second-type data processing task; According to the dependency relationship between the first type of data processing tasks and the second type of data processing tasks, the dependency relationship between multiple data processing task slices corresponding to the first type of data processing tasks and multiple data processing task slices corresponding to the second type of data processing tasks is determined.
5. The method according to claim 1, characterized in that Get the dataset, including: In response to a preset trigger operation, displaying the main interface; In response to a trigger operation on a new data set control in the main interface, displaying a data set selection interface; In response to a trigger operation on a preview control in a data set selection interface, a data set corresponding to the test attribute information input by a user is obtained, and the data set is displayed in the data set selection interface, wherein the test attribute information includes at least one of: a test phase, a test time, a batch material number, a test batch, and a wafer identification.
6. The method according to claim 5, characterized in that Get at least one original test item, including: In response to a trigger operation on a next step control in the data set selection interface, a test item selection interface is displayed, wherein the test item selection interface includes: a query area and a test item list display area; In response to a triggering operation on a determination control in the query area, updating the test item list in the test item list display area according to the query information input by the user; In response to a selection operation on a test item in the test item list, the selected test item is determined as an original test item.
7. The method according to claim 6, characterized in that Get at least one computed test item consisting of original test items, including: In response to a triggering operation on a next step control in a test item selection interface, displaying a calculation test item setting interface; In response to a triggering operation on a newly added calculation test item control in a calculation test item setting interface, a calculation test item editing interface is displayed, wherein the calculation test item editing interface includes: a first editing area, a first test item list display area, and a first function display area; In response to a triggering operation on a test item in the first test item list display area, the triggered test item is displayed in the first editing area; in response to a triggering operation on a function in the first function display area, the triggered function is displayed in the first editing area; In response to a triggering operation on a determination control in the computing test item editing interface, a computing test item is generated according to the triggered test item and the triggered function displayed in the first editing area.
8. The method according to claim 7, characterized in that Get at least one split test item that is composed of the original test item, including: In response to a triggering operation of a next step control in the calculation test item setting interface, displaying a split test item setting interface; In response to a triggering operation on a newly added split test item control in the split test item setting interface, a split test item editing interface is displayed, wherein the split test item editing interface includes: a second editing area, a second test item list display area, and a second function display area; In response to a triggering operation on a test item in the second test item list display area, displaying the triggered test item in the second editing area, and in response to a triggering operation on a function in the second function display area, displaying the triggered function in the editing area; In response to a triggering operation on a determination control in the split test item editing interface, a split test item is generated according to the triggered test item and the triggered function displayed in the second editing area.
9. The method according to claim 2, characterized in that: According to the dependency relationship between the data processing tasks and the data set, the data processing tasks are executed in sequence to obtain the data processing results corresponding to the data processing tasks, including: After executing the first type of data processing task, executing the second type of data processing task to obtain a processing result list; Among them, executing the first type of data processing task includes: obtaining each original test item contained in the calculation test item corresponding to the first type of data processing task; querying the data set to obtain the test results of each original test item corresponding to the wafer identification; performing a first calculation on the test results of each original test item corresponding to the wafer identification according to the function in the calculation test item to obtain the processing result, and generating a first list according to the processing result; Executing the second type of data processing task includes: obtaining each original test item contained in the split test item corresponding to the second type of data processing task; performing a second calculation on the identification information of each original test item according to the function in the split test item to obtain attribute information, adding the attribute information to the first list to obtain a processing result list.
10. The method according to claim 1, characterized in that According to the dependency relationship between the data processing tasks and the data set, the data processing tasks are executed in sequence to obtain the data processing results corresponding to the data processing tasks, including: If all predecessor tasks of the current data processing task have been processed, the current data processing task is executed based on the data set to obtain the data processing result corresponding to the current data processing task, wherein the predecessor task is the data processing task on which the current data processing task depends.
11. A data processing device, characterized in that: include: An acquisition module, configured to acquire a data set, at least one original test item, at least one calculated test item composed of the original test items, and at least one split test item composed of the original test items, wherein the data set includes at least one wafer identification; A task generation module, used to determine a plurality of data processing tasks and dependencies between the data processing tasks according to at least one wafer identification, at least one original test item, at least one calculation test item composed of the original test item, and at least one split test item composed of the original test item; The task execution module is used to execute each data processing task in sequence according to the dependency relationship between the data processing tasks and the data set, and obtain the data processing results corresponding to each data processing task.
12. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the data processing method according to any one of claims 1 to 10.
13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method according to any one of claims 1 to 10 when executed.
14. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the data processing method according to any one of claims 1 to 10.
Citation Information
Patent Citations
Data processing method, device and equipment and computer readable storage medium
CN108628675A
Processing method and system for wafer detection data and storage medium
CN115269175A
Micro-service testing method and device, electronic equipment and storage medium
CN116737583A
System for automated capture and analysis of business information for security and client-facing infrastructure reliability
US20170124501A1
Flexible task scheduler for multiple parallel processing of database data
US20170228422A1
