A data aggregation method and aggregation system

Abstract

The application provides a data aggregation method, comprising: obtaining data to be aggregated for aggregation, and triggering batch processing of the data to be aggregated based on a plurality of fields according to memory usage, dividing the data to be aggregated into a plurality of batches, and aggregating the data to be aggregated in batches in the memory; moving aggregated result data of a batch that has completed aggregation from the memory into an aggregated result file, continuing aggregation of data of the next batch in the memory, until aggregation of all the data to be aggregated is completed, and obtaining a final aggregated result file. The method can combine the characteristics of column-oriented storage data query convenience, and effectively solve the problem of insufficient memory when large amounts of data are aggregated by reading multiple times in batches. The application also provides a data aggregation system, which has corresponding advantages because it can execute the data aggregation method of the application.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of semiconductor design and production, and particularly relates to a data aggregation method and a corresponding data aggregation system. BACKGROUND

[0002] In the whole life cycle of advanced process, there is an important risk factor affecting product yield: the process development stage as a relatively simple environment, its main purpose is to develop a platform process for a certain process node, while the product introduction and mass production stage as a relatively complex and changeable environment, is oriented to the actual product results. Chip products are diversified, and the design maturity and process sensitivity of each chip are different, and even there may be great differences, so in the process of chip introduction, there will be many various problems that are not found or not paid attention to in the process development stage, resulting in the disconnection between process development and product introduction.

[0003] Under the traditional test chip process, customers can only test the test structure in the test chip to infer the state of the corresponding device in the product chip. However, with the continuous evolution of process nodes, the differences between the physical environment faced by the devices in the product chip and the test structures in the test chip gradually manifest. Therefore, how to use the real product chip to design the built-in test chip to realize the test of the key devices in the product chip in the real physical environment has great significance for the improvement of chip product yield.

[0004] In the design of built-in test chip, how to select the devices that need to be tested according to the huge device information is a crucial step, which involves the aggregation processing of device data. Device data is generally stored in the form of two-dimensional row list (device list), and the storage mode includes row-oriented and column-oriented, and the two storage modes have advantages and disadvantages. Row-oriented: device data is stored by row, which can efficiently add data, especially in real-time data storage, but it is inefficient and slow when querying data, such as when querying a row of data, it needs to read each column field in the row. Column-oriented: device data is stored by column, which is not efficient in data addition, especially real-time data addition, but it has an advantage in executing set range operations on the entire two-dimensional table, and can more accurately and quickly respond to data queries.

[0005] For example, a device list as shown in Table 1, some rows in the table have the same content, and the aggregation count will count how many times each different row appears. The aggregation result is shown in Table 2, where the Length, Width, and Model columns are the aggregation conditions, and the last column is the count field representing the aggregation result. The aggregation process effectively reduces the data amount, from 28 rows before aggregation to 8 rows after aggregation.

[0006] Table 1. A device list

[0007]

[0008]

[0009] Table 2. Aggregation result

[0010]

[0011]

[0012] The aggregation does not necessarily apply to all columns, but can also apply to part of the columns, as shown in Table 3, which is the result of aggregation only on the Model column, i.e., the aggregation condition is only the Model column.

[0013] Table 3. Aggregation result of the Model column

[0014] Model Count Nch08 14 Pch08 14

[0015] The aggregation does not necessarily apply to the original column, but can also apply to the derived column, as shown in Table 4, which is the result of aggregation on the Length x Width≥0.0008 and Length x Width≤0.001.

[0016] Table 4. Aggregation result of the derived column

[0017] Length x Width Count 0.0008 2 0.00096 8 0.0008 2 0.00096 8

[0018] In practical applications, the total number of rows of the device list before aggregation can reach the order of billions, and the total number of rows of the result after aggregation can reach the order of millions. In the aggregation process, the aggregation result needs to be saved in the memory all the time, and saving millions of rows of aggregation results in the memory all the time will bring great pressure to the memory, and it may even be impossible to complete. Therefore, how to solve the problem of excessive memory pressure of large amounts of information in the aggregation process is of great significance for completing the aggregation of large amounts of device data and then completing the selection of the tested device. SUMMARY

[0019] The application provides a data aggregation method, which provides a solution to the problem of excessive memory pressure in the aggregation process of large amounts of information.

[0020] Other purposes and advantages of the application can be further understood from the technical features disclosed in the application.

[0021] To achieve one or part or all of the above purposes or other purposes, a technical solution of the application provides a data aggregation method, defining each row of to-be-aggregated data as a piece of data and each column as a field; the data aggregation method comprises: obtaining to-be-aggregated data for aggregation, and triggering batch processing of the to-be-aggregated data based on a plurality of fields according to memory usage, dividing the to-be-aggregated data into a plurality of batches, and aggregating the to-be-aggregated data in batches in the memory according to the batches; moving the aggregation result data of a completed batch into an aggregation result file, continuing to aggregate data of the next batch in the memory, until the aggregation of all the to-be-aggregated data is completed, and obtaining a final aggregation result file; wherein the aggregation result data comprises an aggregation condition and a count field, and the count field is used to represent the number of data pieces meeting the aggregation condition. Through real-time batch processing and batch aggregation of to-be-aggregated data, the memory pressure of large amounts of information in the aggregation process can be effectively relieved.

[0022] The to-be-aggregated data is stored in a column-oriented manner; the to-be-aggregated data is obtained by reading the fields at the corresponding positions in columns to obtain a piece of data for aggregation.

[0023] The aggregation condition is a plurality of fields used for aggregation, including a plurality of original columns or / and a plurality of derived columns; the original column is a field of a column in the to-be-aggregated data, and the derived column is a field generated by logically combining fields in a plurality of columns in the to-be-aggregated data.

[0024] A preset memory is used to store an upper limit of the number of rows of aggregation result data; the data aggregation method further comprises: judging whether the currently stored aggregation result data in the memory reaches the upper limit, and if so and there is unaggregated data, triggering batch processing of the to-be-aggregated data.

[0025] When aggregating in batches, the aggregation result data stored in the memory that no longer belongs to the batch currently being aggregated is temporarily stored in a temporary file; and after the aggregation of the current batch is completed, the aggregation result data in the temporary file is moved into the memory.

[0026] The upper limit is calculated in the following manner: the current available memory bytes are obtained and denoted as Mem; the bytes occupied by each piece of data in the data to be aggregated are obtained and denoted as Agg; M, i.e., the maximum number of rows, is calculated by the formula M = [Mem x a / Agg]; wherein the symbol [] in the formula represents taking the integer of the calculation result; and a is a preset proportion value of the available memory of the processor.

[0027] Suppose that the total number of rows of the data to be aggregated is N, and the maximum number of rows of the memory used to store the aggregated result data is M; the execution process of the data aggregation method comprises the following steps: step 1): initializing i = 1 and j = 0; wherein i ∈ [1, N] and j ∈ [0, M]; setting a filter used to represent a filtering condition, and the number m of filtering conditions in the filter; initializing the filter to be empty (which means that any data satisfies the filtering condition) and m = 0; creating a temporary file; and setting an aggregation starting position set A; step 2): reading the i-th field value in each column of data used as an aggregation condition in the data to be aggregated to form a piece of data and denoted as R; judging whether R satisfies the filter: if yes, going to step 3); if no, going to step 5); step 3): judging whether the memory already stores aggregated result data with the same aggregation condition as R: if yes, adding 1 to the count field of the aggregated result data, and going to step 5); if no, adding a piece of aggregated result data to the memory, wherein the aggregation condition of the aggregated result data is the aggregation condition of R, and the count field of the aggregated result data is 1; and setting j = j + 1; step 4): judging whether j is equal to M: if no, going directly to step 5); if yes, setting m = m + 1, obtaining all field values of the m-th column, and dividing all the data to be aggregated into batches by using the field values, determining one batch as a current aggregation batch, adding the field values used to filter the current batch to the filter as the latest filtering condition, and moving the aggregated result data in the memory that does not satisfy the filter to the temporary file, and setting element a m = i, recording the aggregation state of the batch corresponding to the element, setting element a mStep 5): judging whether i is equal to N: if not, setting i=i+1, and going to step 2); if yes, writing all the aggregation result data in the memory into the aggregation result file, deleting the latest filter condition in the filter, setting m=m-1, and going to step 6); Step 6): judging whether the aggregation start position set A has no element: if yes, completing the aggregation of the data to be aggregated, and obtaining the aggregation result file; if not, obtaining the latest element in the aggregation start position set A, judging whether the element has corresponding un-aggregated batches: if yes, setting the element to i, determining a current batch in the un-aggregated batches corresponding to the element, adding the field value used for screening the current batch as the latest filter condition into the filter, setting m=m+1, moving the aggregation result data belonging to the current batch in the temporary file into the memory, and changing the value of j to the number of the aggregation result data in the memory, and going to step 2); if not, going to step 7); Step 7): deleting the latest element in the aggregation start position set A, and going to step 6).

[0028] The data to be aggregated, the aggregation result file and the temporary file are stored in the same or different storage devices, and the storage devices are one of a database, a disk and a hard disk.

[0029] The application further provides a data aggregation system, which comprises a storage device, and the storage device stores a plurality of instructions which are loaded by a processor and execute the data aggregation method.

[0030] Compared with the prior art, the application has the following beneficial effects:

[0031] 1. The data aggregation method can utilize the convenient characteristics of column-oriented storage data query, and effectively solve the insufficient memory problem of large data aggregation through batch reading and filter setting, thereby providing a feasible solution for efficient completion of large device data aggregation and further efficient and reliable completion of device selection.

[0032] 2. The data aggregation system has the corresponding advantages because it can execute the data aggregation method. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings described below are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort.

[0034] Figure 1 A data aggregation method process diagram of an embodiment of the present application.

[0035] Figure 2 A data aggregation flowchart of an embodiment of the present application. DETAILED DESCRIPTION

[0036] The foregoing and other technical contents, features and effects of the present application will be clearly presented in the following detailed description of preferred embodiments in conjunction with the drawings.

[0037] In the data aggregation method of the embodiment example of the present application, each row of the data to be aggregated is defined as a piece of data, and each column is defined as a field. As shown in the following table, the data to be aggregated includes a plurality of fields, and each field is a column in the table. Figure 1 The data aggregation method includes: obtaining the data to be aggregated for aggregation, and triggering batch processing of the data to be aggregated based on a plurality of fields according to memory usage; dividing the data to be aggregated into a plurality of batches according to the memory usage, and aggregating the data to be aggregated in batches in the memory; moving the aggregated result data of the completed batch from the memory to the aggregated result file, continuing to aggregate the data of the next batch in the memory, until the aggregation of all the data to be aggregated is completed, and obtaining the final aggregated result file; wherein the aggregated result data includes an aggregation condition and a count field, and the count field is used to represent the number of data pieces satisfying the aggregation condition.

[0038] In some specific embodiments, the data to be aggregated is stored in a column-oriented manner; and the data to be aggregated is obtained by reading the fields at corresponding positions by column to obtain a piece of data for aggregation.

[0039] The aggregation condition includes a plurality of fields for aggregation, including a plurality of native columns or / and a plurality of derived columns; the native column is a field of a column in the data to be aggregated, and the derived column is a field generated by logically combining fields in a plurality of columns in the data to be aggregated.

[0040] In some specific examples, a preset upper limit of the number of rows of the memory for storing the aggregated result data is provided. The data aggregation method further includes: judging whether the currently stored aggregated result data in the memory reaches the upper limit, and if so and there is unaggregated data, triggering batch processing of the data to be aggregated. In the data aggregation method, the aggregation and the real-time judgment of whether batch processing is needed are performed simultaneously. When batch processing is performed, the aggregated result data stored in the memory which no longer belongs to the current batch being aggregated is temporarily stored in a temporary file; and after the aggregation of the current batch is completed, the aggregated result data in the temporary file is moved into the memory. The batch processing is performed on all the data, and even if some data has been aggregated after batch processing, it will also be moved to the temporary file, and will be stored in the memory and then moved to the aggregated result data when the batch data is processed.

[0041] The upper limit of the example is calculated in the following manner: the current available memory in bytes is obtained and denoted as Mem; the number of bytes occupied by each piece of data in the data to be aggregated is obtained and denoted as Agg. In the example, the number of bytes occupied by each piece of data is the same. The maximum number of rows M is calculated by the formula M = [Mem x a / Agg], where the symbol [] represents rounding the calculation result to an integer; a is a preset proportion of the available memory of the processor. In a better example, a = 80%.

[0042] In some embodiments, as shown in Figure 2 the total number of rows of the data to be aggregated is N, and the maximum number of rows of the memory used to store the aggregated result data is M; the specific execution process of the data aggregation method is as follows: step 1): initialize i = 1, j = 0; where i ∈ [1, N], j ∈ [0, M]; set filter used to represent the filtering condition, and the number of filtering conditions m in filter; initialize filter to be empty (filter being empty means that any data satisfies the filtering condition), m = 0; create a temporary file; set the aggregation starting position set A; step 2): read the i-th field value in each column of data as the aggregation condition in the data to be aggregated to form a piece of data and denote it as R; judge whether R satisfies filter: if yes, go to step 3); if no, go to step 5); step 3): judge whether the memory has stored aggregated result data with the same aggregation condition as R: if yes, add 1 to the count field of the aggregated result data, and go to step 5); if no, add a piece of aggregated result data to the memory, the aggregation condition of the aggregated result data is the aggregation condition of R, and the count field of the aggregated result data is 1; let j = j + 1; step 4): judge whether j is equal to M: if no, go directly to step 5); if yes, let m = m + 1, obtain all field values of the m-th column, and divide all the data to be aggregated into several batches by using these field values, determine one of the batches as the current aggregation batch, and add the field values used to filter the current batch to filter as the latest filtering condition (all field values of the field are obtained, and the batch is divided into two or more batches, rather than using a certain field value to divide the batch into two, so the position may be backtracked several times for aggregation of different batches); move the aggregated result data in the memory that does not satisfy filter to the temporary file, and let a m = i (the subscript m relates to the current value of m, for example, the first batch, here a1 = i, which is used to record the position of the i-th row of data), and record the aggregation state of the batch corresponding to the element, and let a mAs the latest one element is added to the aggregation starting position set A; step 5): judging whether i is equal to N: if not, i is set to i+1, and goes to step 2); if yes, all the aggregation result data in the memory are written into the aggregation result file, the latest one filter condition in the filter is deleted, m is set to m-1, and goes to step 6); step 6): judging whether there is no element in the aggregation starting position set A: if yes, the aggregation of the data to be aggregated is completed, and the aggregation result file is obtained; if not, the latest one element in the aggregation starting position set A is obtained, and it is judged whether the element has corresponding un-aggregated batches: if yes, the element is assigned to i, a current batch is determined in the un-aggregated batch corresponding to the element, the field value used for screening the current batch is added to the filter as the latest one filter condition, m is set to m+1, the aggregation result data belonging to the current batch in the temporary file are all moved into the memory, the value of j is changed to the number of the aggregation result data in the memory, and goes to step 2); if not, goes to step 7); step 7): the latest one element in the aggregation starting position set A is deleted, and goes to step 6).

[0043] The data aggregation system provided in the embodiment includes a storage device, and a plurality of instructions are stored in the storage device, the plurality of instructions are loaded by a processor and execute the data aggregation method of the embodiment. The example data to be aggregated, the aggregation result file and the temporary file can be respectively stored in different storage devices or stored in one storage device, which is not limited. The storage device can be a database, a disk, a hard disk, etc. The example using a database for storage can be cloud storage or distributed storage, which is not limited.

[0044] The following embodiments can enable those skilled in the art to more fully understand the specific aggregation process of the present application, but do not limit the present application in any way. It should be noted that the parameters used in this embodiment are small for ease of description. The example parameters are not limited to the actual situation.

[0045] The following is an example of N=28 data to be aggregated. For ease of description, as shown in Table 5 below, an additional column representing the row number is added to the leftmost side of Table 5, which can or can not be stored in the actual database. Set M=6, i.e., the maximum number of rows of aggregation result data stored in the memory is 6. The aggregation Length (length), Width (width), and Model (device model) columns, and the sa in the table refer to the distance from the stress (stress) to the transistor a side.

[0046] Table 5. Data to be aggregated

[0047]

[0048]

[0049] Read the field values of Length, Width, Model three columns by column to obtain a piece of data for aggregation, at this time the sa column is ignored, only consider three columns; until i = 20, at this time the memory stores the aggregation result data as shown in Table 6, has reached the maximum number of rows of memory, and there is no aggregation result data in the memory with the same aggregation condition as the i = 20th row data.

[0050] Table 6. Aggregation result data

[0051] Length Width Model Count 0.010 0.060 Nch08 3 0.010 0.080 Nch08 2 0.016 0.060 Nch08 8 0.016 0.080 Nch08 1 0.010 0.060 Pch08 3 0.010 0.080 Pch08 2

[0052] At this time, j = M is satisfied, Length = 0.010 is added to the filter condition filter, and the two rows of Length = 0.016 in the memory are written to the temporary file, at this time the memory data is as shown in Table 7.

[0053] Table 7. Memory data

[0054] Length Width Model Count 0.010 0.060 Nch08 3 0.010 0.080 Nch08 2 0.016 0.060 Nch08 0 0.016 0.080 Nch08 0 0.010 0.060 Pch08 3 0.010 0.080 Pch08 2

[0055] At this time, the parameter i = 20, and there is only one condition Length = 0.010 in the filter. Since Length is 0.016 from i = 20 to 28, aggregation can be performed, that is, after the aggregation of i = 28 is completed, all data that meet the filter condition have been aggregated, so it can be directly written to the file F, that is, the file F as shown in Table 8 is obtained, and the memory data becomes as shown in Table 9.

[0056] Table 8. File F

[0057] Length Width Model Count 0.010 0.060 Nch08 3 0.010 0.080 Nch08 2 0.010 0.060 Pch08 3 0.010 0.080 Pch08 2

[0058] Table 9. Memory data

[0059] Length Width Model Count 0.010 0.060 Nch08 0 0.010 0.080 Nch08 0 0.016 0.060 Nch08 0 0.016 0.080 Nch08 0 0.010 0.060 Pch08 0 0.010 0.080 Pch08 0

[0060] After that, the two rows in the temporary file are moved into the memory, and the memory data is as shown in Table 10.

[0061] Table 10. Memory data

[0062] Length Width Model Count 0.016 0.060 Nch08 8 0.016 0.080 Nch08 1 0.016 0.060 Nch08 0 0.016 0.080 Nch08 0 0.010 0.060 Pch08 0 0.010 0.080 Pch08 0

[0063] At this time, the filter is actually equivalent to Length not equal to 0.010, return to i = 20, continue to aggregate until i = 28 to complete all data aggregation, write the aggregation result data in the memory to the file F, obtain the final aggregation result file as shown in Table 11, complete the aggregation of the data to be aggregated as shown in Table 5 above.

[0064] Table 11. Aggregation result file

[0065]

[0066]

[0067] The above description is merely that of the preferred embodiments of the application and is not intended to limit the scope of the application being claimed. By means of simple equivalent changes and modifications, the embodiments disclosed in the claims and specification can be implemented in various other forms, all of which should be construed to fall within the scope of the application. In addition, any embodiment or claim of the application need not achieve all the purposes or advantages disclosed in the specification. Furthermore, the abstract and title are intended to assist with patent file indexing, and are not intended to limit the scope of the application.

Claims

1. A data aggregation method, characterized by, Each row of the data to be aggregated is defined as a piece of data, and each column is a field; the data aggregation method comprises: The data to be aggregated is aggregated, and based on the memory usage, the data to be aggregated is divided into batches based on a plurality of fields, the data to be aggregated is divided into a plurality of batches, and the data to be aggregated is batch-aggregated in the memory according to the batches; During the batch aggregation, the aggregated result data stored in the memory which no longer belongs to the batch currently being aggregated is temporarily stored in a temporary file; and after the aggregation of the current batch is completed, the aggregated result data in the temporary file is moved into the memory; The aggregated result data of the batch which has completed the aggregation is moved from the memory into an aggregated result file, and the aggregation of the next batch of data is continued in the memory until the aggregation of all the data to be aggregated is completed, and a final aggregated result file is obtained; The aggregated result data comprises an aggregation condition and a count field, and the count field is used to represent the number of data pieces satisfying the aggregation condition; The data to be aggregated is stored in a column-oriented manner; the data to be aggregated is aggregated by reading the field at the corresponding position by column to obtain a piece of data; Supposing that the total number of rows of the data to be aggregated is N, and the maximum number of rows of the memory used for storing the aggregated result data is M; the execution process of the data aggregation method comprises: Step 1): initializing i = 1 and j = 0; wherein i ∈ [1, N] and j ∈ [0, M]; setting a filter used to represent a filtering condition, and the number m of filtering conditions in the filter; initializing the filter to be empty and m = 0; creating a temporary file; and setting an aggregation starting position set A; Step 2): reading the i-th field value in each column of the data to be aggregated as an aggregation condition to form a piece of data and record it as R; judging whether R satisfies the filter: if yes, going to step 3); if no, going to step 5); Step 3): judging whether the memory already stores aggregated result data with the same aggregation condition as R: If yes, the count field of the aggregated result data is incremented by 1, and going to step 5); If no, a new piece of aggregated result data is added to the memory, the aggregation condition of the aggregated result data is the aggregation condition of R, and the count field of the aggregated result data is 1; and j = j + 1; Step 4): judging whether j is equal to M: If no, going directly to step 5); If yes, m = m + 1, all field values of the m-th column are obtained, all the data to be aggregated is divided into a plurality of batches by using the field values, one of the batches is determined as a current aggregation batch, the field value used to filter the current batch is added to the filter as the latest filtering condition, the aggregated result data in the memory which does not satisfy the filter is moved into the temporary file, element am = i is recorded, the aggregation state of the batch corresponding to the element is recorded, and the element am is added to the aggregation starting position set A as the latest element; Step 5): judging whether i is equal to N: if not, setting i = i + 1, and going to step 2); if yes, writing all the aggregation result data in the memory into the aggregation result file, deleting the latest filter condition in the filter, setting m = m - 1, and going to step 6); Step 6): judging whether the aggregation start position set A has no element: if yes, completing the aggregation of the data to be aggregated, and obtaining the aggregation result file; if no, obtaining the latest element in the aggregation start position set A, judging whether the element has corresponding un-aggregated batches: if yes, setting the element to i, determining a current batch in the un-aggregated batches corresponding to the element, adding the field value used for filtering the current batch as the latest filter condition into the filter, setting m = m + 1, moving the aggregation result data belonging to the current batch in the temporary file into the memory, changing the value of j to the number of aggregation result data in the memory, and going to step 2); if no, going to step 7); Step 7): deleting the latest element in the aggregation start position set A, and going to step 6).

2. The data aggregation method of claim 1, wherein, The aggregation conditions are a plurality of fields used for aggregation, including a plurality of original columns or / and a plurality of derived columns; the original column is a field of a column in the data to be aggregated, and the derived column is a field generated by logically combining fields in a plurality of columns in the data to be aggregated.

3. The data aggregation method of claim 1, wherein, A preset memory is used to store an upper limit of the number of rows of aggregation result data. The data aggregation method further includes: judging whether the currently stored aggregation result data in the memory reaches the upper limit, if yes and there is un-aggregated data, triggering the batch processing of the data to be aggregated.

4. The data aggregation method of claim 3, wherein, The calculation method of the upper limit is specifically as follows: obtaining the number of currently available memory bytes, denoted as Mem; obtaining the number of bytes occupied by each piece of data in the data to be aggregated, denoted as Agg; calculating M, i.e. the maximum number of rows, through the formula M = [Mem x a / Agg]; wherein the symbol [] in the formula represents taking the integer of the calculation result; and a is a preset processor available memory occupancy proportion value.

5. The data aggregation method of claim 1, wherein, The data to be aggregated, the aggregation result file and the temporary file are stored in the same or different storage devices, and the storage device is one of a database, a disk and a hard disk.

6. A data aggregation system comprising a storage device having stored therein a number of instructions, wherein, The plurality of instructions are loaded and executed by a processor to implement the data aggregation method in any one of claims 1 to 5.

Images