Data lake schema categorization method, apparatus, device, medium, and product

Abstract

The application relates to a data lake schema classification method, device, equipment, medium and product. The method comprises the following steps: creating a queue and initializing; acquiring all fields in at least two table schemas and storing the fields in the queue; sequentially performing field type and field name judgment on the fields in a partition schema and the at least two table schemas in the queue; sequentially performing distance calculation on the fields in the partition schema and the same fields in the at least two table schemas in the queue; determining the table schema in the queue corresponding to the minimum distance, and merging the partition schema into the table schema. The application solves the problem that when there are at least two compatible table schemas, it is difficult to determine to which table schema the classification is performed. The application has the effect of quickly merging the partition schema into the table schema.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data classification, in particular to a data lake schema classification method and device, equipment, medium and product. BACKGROUND

[0002] In an unmanned system, raw collected sensor data, labeled data, log data, etc. are included, and these different data constitute a data lake. Each kind of data is generated every day, and in order to facilitate management, it is usually stored in different partitions according to date, vehicle, etc. The table structure description information corresponding to the data lake is usually called table schema, and the table structure description information corresponding to different partitions in the data lake is called partition schema. With the passage of time, data is constantly evolving, and different partitions have different partition schemas.

[0003] In order to facilitate use, compatible partition schemas in different partition schemas can be constructed into a unified table schema. The table schema includes which fields and the type of each field. Different partition schemas have inconsistent field types, and the fields of a partition schema are added or deleted from the fields of another partition schema. Each partition schema can also form a table schema. Among them, the partition schema corresponding to the addition of fields or the deletion of fields can maintain compatibility with the old table schema, so it can be merged into the table schema.

[0004] For the related technologies in the above, the inventor believes that when there are at least two table schemas compatible with the partition schema, it is difficult to determine which table schema the partition schema is classified into. SUMMARY

[0005] In order to quickly merge the partition schema corresponding to the addition of fields or the deletion of fields into the compatible table schema, the present application provides a data lake schema classification method, device, equipment, medium and product.

[0006] In a first aspect, the present application provides a data lake schema classification method, which has the characteristics of quickly merging the partition schema corresponding to the addition of fields or the deletion of fields into the compatible table schema.

[0007] The present application is realized by the following technical solutions:

[0008] A data lake schema classification method includes the following steps:

[0009] Create a queue for storing table schemas and initialize;

[0010] all fields in at least two table schemas in the queue are obtained and stored;

[0011] the fields in the partition schema are sequentially compared with the fields in at least two table schemas in the queue in terms of field type and field name to obtain the same fields in the queue;

[0012] the fields in the partition schema are sequentially compared with the same fields in at least two table schemas in the queue in terms of distance;

[0013] the table schema in the queue corresponding to the minimum distance is determined, and the partition schema is merged into the table schema.

[0014] In a preferred example, the application can be further configured such that the step of sequentially comparing the fields in the partition schema with the same fields in at least two table schemas in the queue in terms of distance comprises:

[0015] the value of the field stored in the queue is initialized as a first threshold value;

[0016] when the field in the partition schema is the same as the field in the queue, the value of the corresponding field in the queue is added by a first preset value, and the calculation result is updated as the value of the field;

[0017] when the field in the partition schema is different from the field in the queue, the field in the partition schema is newly added relative to the field in the table schema in the queue, the corresponding field in the partition schema is added to the queue, and the value of the added field is a second threshold value, which is not equal to the first threshold value;

[0018] the values of the fields in the same table schema are sequentially accumulated to obtain the distance value of the table schema.

[0019] In a preferred example, the application can be further configured such that the step of merging the corresponding field in the partition schema with the corresponding field in the table schema comprises:

[0020] when the field in the partition schema is the same as the field in the table schema, the value of the corresponding field in the table schema is added by a second preset value;

[0021] When the field in the partition schema is different from the field in the table schema, the field in the partition schema is added to the table schema, and the value of the added field is a third preset value, which is different from the initial value of the field in the table schema.

[0022] The application can be further configured in a preferred example to further include the following steps:

[0023] All fields in the table schema are reserved.

[0024] The application can be further configured in a preferred example to include the following steps in the step of making the field in the partition schema sequentially perform field type judgment with at least two table schemas in the queue:

[0025] The field in the partition schema and the field of any table schema are added to the same hash table, and it is judged whether the field in the hash table meets the condition.

[0026] When the field meets the condition, the field type in the partition schema is the same as the field type of the corresponding field in at least two table schemas in the queue.

[0027] The application can be further configured in a preferred example to include the following steps while performing the step of making the field in the partition schema sequentially perform field type judgment with at least two table schemas in the queue to obtain the same field in the queue:

[0028] When the field type in the partition schema is different from the field type in the queue, the distance value between the field in the partition schema and the corresponding field in the table schema is positive infinity.

[0029] The application can be further configured in a preferred example to further include the following steps:

[0030] The field in the partition schema corresponding to the positive infinity distance value is stored in the queue.

[0031] In the second aspect, the application provides a data lake schema classification device, which has the characteristics of quickly merging the partition schema corresponding to the addition of the field or the deletion of the field to the compatible table schema.

[0032] The application is achieved by the following technical solutions:

[0033] A data lake schema classification device comprises:

[0034] A queue module is configured to create a queue for storing table schemas and initialize the queue;

[0035] A benchmark module is configured to obtain all types of fields in at least two table schemas and store the fields in the queue;

[0036] A type module is configured to sequentially compare the fields in a partition schema with the fields in at least two table schemas in the queue in terms of field type and field name, and obtain the same fields in the queue;

[0037] A distance module is configured to sequentially calculate the distance between the fields in a partition schema and the same fields in at least two table schemas in the queue;

[0038] A merging module is configured to determine the table schema in the queue corresponding to the minimum distance, and merge the corresponding fields in the partition schema and the corresponding fields in the table schema.

[0039] In a third aspect, the present application provides a computer device having the feature that the partition schema corresponding to the addition of a field or the deletion of a field is quickly merged into a compatible table schema.

[0040] The present application is achieved by the following technical solutions:

[0041] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above-mentioned data lake schema classification method when executing the computer program.

[0042] In a fourth aspect, the present application provides a computer readable storage medium having the feature that the partition schema corresponding to the addition of a field or the deletion of a field is quickly merged into a compatible table schema.

[0043] The present application is achieved by the following technical solutions:

[0044] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the above-mentioned data lake schema classification method.

[0045] In a fifth aspect, the present application provides a computer program product having the feature that the partition schema corresponding to the addition of a field or the deletion of a field is quickly merged into a compatible table schema.

[0046] The present application is achieved by the following technical solutions:

[0047] A computer program product comprising a computer program which, when executed by a processor, implements the steps of the above-described data lake schema classification method.

[0048] In summary, the present application includes at least one of the following beneficial technical effects:

[0049] 1. A data lake schema classification method for partition schema with field addition or deletion operation, using clustering algorithm, the partition schema is added to the old compatible and shortest distance table schema for merging, so that the partition schema can be quickly merged into the table schema to speed up the merging speed; at the same time, the number of table schemas in the system is reduced, the total classification is reduced, the complexity of using data is reduced, the purpose of quickly classifying table schema in data lake is realized, and the table schema is automatically managed; it can also be used to automatically detect whether there is incompatible partition schema in the system, which is beneficial to reduce the destructive effect of incompatible partition schema on upper application programs;

[0050] 2. When merging, the union set of partition schema and table schema is calculated, that is, only the fields in the original table schema are added, and no fields are deleted, so as to ensure that the final table schema can cover all previous partition schemas;

[0051] 3. When the field type in the partition schema is different from the field type in the queue, the distance value between the field in the partition schema and the corresponding field in the table schema is positive infinity, and the field in the partition schema corresponding to the distance value of positive infinity is stored in the queue to form a new table schema in the queue, expand the merging range of the queue, so that the queue can merge more partition schemas, and the applicability is stronger. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is a whole flow chart of a data lake schema classification method according to an embodiment of the present application.

[0053] Figure 2 is a step flow chart of distance calculation between fields in the partition schema and the same fields in the table schema in the queue.

[0054] Figure 3 is a step flow chart of merging the corresponding fields in the partition schema and the corresponding fields in the table schema.

[0055] Figure 4is a structural block diagram of a data lake schema classification device according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] The specific embodiments are merely illustrative of the present application, and are not intended to limit the present application. Those skilled in the art can make modifications to the embodiments according to the present application without creative effort, and such modifications shall fall within the scope of the present application as long as they are within the scope of the claims of the present application.

[0057] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort shall fall within the scope of the present application.

[0058] In addition, the term "and / or" in the present application is only used to describe the association relationship of the associated objects, and can represent three relationships, for example, A and / or B can represent three cases of existence of A alone, existence of A and B simultaneously, and existence of B alone. In addition, the character " / " in the present application generally represents an "or" relationship between the associated objects, unless otherwise specified.

[0059] The different partition schemas have two types: inconsistent field types, for example, the type of field A is int in partition 1, and the type of field A is long in partition 2, that is, the field types are inconsistent; and addition and deletion of fields, that is, the field types of the partition schema are consistent with the field types of the table schema.

[0060] The field types of the partition schema inconsistent must be divided into different table schemas. The partition schema with added or deleted fields can be added to the old table schema.

[0061] The embodiments of the present application will be described below with reference to the drawings of the specification.

[0062] Referring to Figure 1 The embodiments of the present application provide a data lake schema classification method, and the main steps of the method are described as follows.

[0063] S1: creating a queue for storing table schemas and initializing;

[0064] S2: obtaining all fields in at least two table schemas and storing them in the queue;

[0065] S31: judging the field type and the field name of the fields in the partition schema with at least two table schemas in the queue in sequence to obtain the same fields in the queue;

[0066] S4: calculating the distance of the fields in the partition schema with the same fields in at least two table schemas in the queue in sequence;

[0067] S5: determining the table schema in the queue corresponding to the minimum distance, and merging the partition schema into the table schema, and updating the merging result to the queue.

[0068] Further, the step of S31: judging the field type of the fields in the partition schema with at least two table schemas in the queue in sequence to obtain the same fields in the queue further comprises the following steps:

[0069] S32: when the field type in the partition schema is different from the field type in the queue, returning the distance value of the field in the partition schema corresponding to the field in the table schema as positive infinity;

[0070] S321: storing the field in the partition schema corresponding to the distance value of positive infinity to the queue.

[0071] Until all the fields in all the partition schemas complete the field type judgment with at least two table schemas in the queue, at this time, all the table schemas in the updated queue are the final merging result.

[0072] Referring to Figure 2 Further, the step of S4: calculating the distance of the fields in the partition schema with the same fields in at least two table schemas in the queue in sequence comprises:

[0073] S41: initializing the value of the field stored in the queue as a first threshold value;

[0074] S421: when the field in the partition schema is the same as the field in the queue, adding a first preset value to the value of the corresponding field in the queue, and updating the calculation result as the value of the field;

[0075] S422: when the field in the partition schema is different from the field in the queue, the field in the partition schema is a new field relative to the field in the table schema in the queue, adding the corresponding field in the partition schema to the queue, and setting the value of the added field as a second threshold value, the second threshold value is not equal to the first threshold value;

[0076] S43: sequentially accumulate the values of the fields in the same table schema to obtain the distance value of the table schema.

[0077] Referring to Figure 3 Further, S5: the step of merging the corresponding fields in the partition schema and the corresponding fields in the table schema comprises:

[0078] S51: when the field in the partition schema is the same as the field in the table schema, adding a second preset value to the value of the corresponding field in the table schema;

[0079] S52: when the field in the partition schema is different from the field in the table schema, the field in the partition schema is newly added relative to the field in the table schema, the corresponding field in the partition schema is added to the table schema, and the value of the added field is a third preset value, and the third preset value is not equal to the initial value of the field in the table schema;

[0080] S53: at the same time, all fields in the table schema are retained.

[0081] Further, S3: the step of sequentially performing field type judgment on the fields in the partition schema and at least two table schemas in the queue comprises:

[0082] adding the fields in the partition schema and the fields of any table schema to the same hash table, and judging whether the fields in the hash table meet the condition;

[0083] When the fields meet the condition, the field type in the partition schema is the same as the field type of the corresponding field in at least two table schemas in the queue.

[0084] The specific flow steps of each embodiment are described as follows:

[0085] When the number of old table schemas that can be joined by the partition schema with newly added or deleted fields is at least two, for example, for two incompatible table schemas A and B, there is a new partition schema C, which only adds some fields relative to A and B, so it is compatible with A and B, at this time there are two division methods:

[0086] 1. A+C, B;

[0087] 2. A, B+C.

[0088] In order to quickly merge the partition schema corresponding to the addition of fields or the deletion of fields into the compatible table schema, the application provides a data lake schema classification method.

[0089] A queue for storing table schemas is created first, and initialized to create an empty queue to save the clustering results of partition schemas. All fields of a partition schema are first added to the empty queue as a table schema in the queue, and then each new partition schema is compared with the table schema in the queue in terms of field type and field name; if there is a same field, i.e., the table schema in the queue is compatible, the new partition schema is merged into the table schema in the queue; otherwise, the fields of the new partition schema are added to the queue to form another table schema; the next partition schema is then compared with the table schema in the queue in terms of field type and field name; if there are two compatible table schemas, the table schema with the smallest distance to the partition schema is found from the compatible table schemas for merging; otherwise, the fields of the partition schema are added to the queue to form another table schema; and the process is repeated until the merging of all partition schemas is completed.

[0090] In this embodiment, all fields in at least two table schemas are acquired and stored in the queue to add all old table schemas to the queue as a reference for subsequent distance comparison. In this embodiment, all fields of two table schemas are acquired and stored in the queue. For example, the fields contained in table schema 1 are {“x”: int, “y”: float},

[0091] the fields contained in table schema 2 are {“x”: int, “y”: float, “z”: double},

[0092] All fields of the table schema 1 and all fields of the table schema 2 are stored into a queue, i.e. the queue stores the field "x: int" and the field "y: float" of the table schema 1, the field "x: int", the field "y: float" and the field "z: double" of the table schema 2. The fields in the partition schema are sequentially compared with at least two table schemas in the queue in terms of field type and field name. In this embodiment, the fields in the partition schema are sequentially compared with two table schemas in the queue in terms of field type and field name. For example, the fields contained in the partition schema can be { "y": float, "z": double}. The fields "x" and "y" of the table schema 1 are first added into the same hash table, and the hash table is initialized, so the hash table is { "x": (int, -1), "y": (float, -1)}. The field "y" in the partition schema is first added into the hash table, so the hash table is { "x": (int, -1), "y": (float, 0)}. The hash table is scanned, and it is determined whether there is a field with a field value equal to 0 in the hash table, i.e. it is determined whether the field in the hash table satisfies the condition. Since "y": (float, 0), the condition is satisfied, and the field "y" in the partition schema has the same type as the field "y" in the table schema 1. The field "z" in the partition schema is added into the initialized hash table, so the hash table is { "x": (int, -1), "y": (float, -1), "z": (double, -1)}. Since there is no field with a field value equal to 0 in the hash table, the condition is not satisfied, and the field "z" in the partition schema has different types from all fields in the table schema 1.

[0093] Similarly, the fields of "x", "y" and "z" of the table schema2 are also added into the same hash table, the hash table is initialized as { "x" : (int, -1), "y" : (float, -1), "z" : (double, -1)}, the field "y" in the partition schema is added into the hash table first, at this time, the hash table is { "x" : (int, -1), "y" : (float, 0), "z" : (double, -1)}, the hash table is scanned to determine whether there is a field with a value equal to 0 in the hash table, and then the field "y" in the partition schema is determined to be of the same type as the field "y" in the table schema2; the field "z" in the partition schema is added into the initialized hash table, at this time, the hash table is { "x" : (int, -1), "y" : (float, -1), "z" : (double, 0)}, and thus the field "z" in the partition schema is determined to be of the same type as the field "z" in the table schema2.

[0094] The fields in the partition schema are sequentially compared with the same fields in at least two table schemas in the queue to perform distance calculation, so as to compare all the fields in each table schema in the queue. In this embodiment, the fields in the partition schema are sequentially compared with the same fields in two table schemas in the queue to perform distance calculation. For example, the field "y" in the partition schema is compared with the field "y" in the table schema1 and the field "y" in the table schema2 to perform distance calculation.

[0095] Specifically, the values of the fields stored in the queue are initialized as a first threshold value. In this embodiment, the first threshold value can be -1. For example:

[0096] { [ "x" : (int, -1), "y" : (float, -1) ], [ "x" : (int, -1), "y" : (float, -1), "z" : (double, -1) ]}.

[0097] Since the field "y" in the partition schema is the same as the field "y" in the table schema1 and the field "y" in the table schema2 in the queue, the values of the corresponding fields in the queue are added by a first preset value, and in this embodiment, the first preset value can be 1, that is:

[0098] { [ "x" : (int, -1), "y" : (float, 0) ], [ "x" : (int, -1), "y" : (float, 0), "z" : (double, -1) ]}.

[0099] When the field in the partition schema is different from the field in the queue, the field in the partition schema is added to the queue, and the value of the added field is a second threshold value, which is different from the first threshold value. In this embodiment, the second threshold value can be 1.

[0100] The values of the fields in the same table schema are sequentially accumulated, that is, the initial value of the distance is 0, the value is 1 when there is an addition or deletion, and the value is 0 when the fields are completely the same. For example, the distance value of the table schema 1 is |-1|+0=1, and the distance value of the table schema 2 is |-1|+0+|-1|=2, so that the distance value of the table schema 1 and the table schema 2 is obtained.

[0101] The table schema in the queue corresponding to the minimum distance is determined to find the table schema with the minimum distance to be merged, so that the change is the least, the merging is faster, and the corresponding field in the partition schema and the corresponding field in the table schema are merged. In this embodiment, the table schema in the queue corresponding to the minimum distance is the table schema 1.

[0102] The partition schema is merged into the corresponding table schema, that is, the corresponding field in the partition schema and the corresponding field in the table schema 1 are merged, which specifically includes that when the field in the partition schema is the same as the field in the table schema 1, the value of the corresponding field in the table schema 1 is added by a second preset value, and in this embodiment, the second preset value can be 1; when the field in the partition schema is different from the field in the table schema 1, the corresponding field in the partition schema is added to the table schema 1, and the value of the added field is a third preset value, which is different from the initial value of the field in the table schema, and in this embodiment, the initial value of the field can be -1, and the third preset value can be 1. For example, the field “y” in the partition schema is merged into the table schema 1, and at this time, the table schema 1 is {“x”:(int,-1),“y”:(float,0)}, and then for the system, the table schema can be used as much as possible, so as to facilitate management and reduce the complexity of data use.

[0103] In this embodiment, the fields in the partition schema can also be sorted with the fields of at least two table schemas respectively, a two-way merging algorithm is adopted to judge the length of the merged table schema, and the table schema with the smaller length is determined as a target table schema, so as to find the table schema with the minimum distance to be merged with the partition schema.

[0104] Further, all fields in the table schema schema1 are reserved, i.e. the partition schema and the table schema are merged to form a union set, and only fields are added to the original table schema, and no field is deleted, so that the final table schema can cover all previous partition schemas.

[0105] Meanwhile, the merging result: the table schema schema1 {“x”:(int,-1),“y”:(float,0)} is updated to the queue, so that the table schemas in the queue are updated in real time, and the fields in the queue include the fields after merging.

[0106] Therefore, all fields with a value of 1 in the queue after merging represent added fields, all fields with a value of 0 represent the same fields, and all fields with a value of -1 represent deleted fields.

[0107] Further, the fields in the partition schema are sequentially subjected to field type judgment with at least two table schemas in the queue, and the step of obtaining the same fields in the queue further includes the following steps:

[0108] When the field type in the partition schema is different from the field type in the queue, the distance value between the field in the partition schema and the corresponding field in the table schema is returned as positive infinity, indicating that the field in the partition schema and all fields in the queue are incompatible.

[0109] The field in the partition schema corresponding to the distance value of positive infinity is stored in the queue to form a new table schema, which expands the merging range of the queue, so that the queue can merge more partition schemas, and the applicability is stronger.

[0110] For example, the field “z” in the partition schema is different from all field types in the table schema schema1, and positive infinity is returned, and then the field “z” in the partition schema is stored in the table schema schema1 in the queue.

[0111] Finally, all table schemas in the queue are the results after merging the partition schemas.

[0112] Further, a data lake schema classification method adds or deletes a partition schema for a field, adopts a clustering algorithm, and merges the partition schema into an old compatible and shortest distance table schema to speed up the merging speed. At the same time, the number of table schemas in the system is reduced, the total classification is reduced, the complexity of using data is reduced, the purpose of quickly classifying table schemas in the data lake is achieved, and the table schema is automatically managed. It can also be used to automatically detect whether there is an incompatible partition schema in the system. Because the incompatible partition schema may be destructive to the upper application program, it is beneficial to reduce the destructive effect of the incompatible partition schema on the upper application program.

[0113] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0114] Referring to Figure 4 The embodiments of the present application also provide a data lake schema classification device, which corresponds to the data lake schema classification method described above. The data lake schema classification device comprises:

[0115] A queue module is configured to create a queue for storing table schemas and initialize the queue;

[0116] A reference module is configured to obtain all fields in at least two table schemas and store the fields in the queue;

[0117] A type module is configured to sequentially compare the fields in the partition schema with the fields in at least two table schemas in the queue in terms of field type and field name, and obtain the same fields in the queue;

[0118] A distance module is configured to sequentially calculate the distance between the fields in the partition schema and the same fields in at least two table schemas in the queue;

[0119] A merging module is configured to determine the table schema in the queue corresponding to the shortest distance, and merge the corresponding fields in the partition schema and the table schema, and update the merging result to the queue.

[0120] Further, the data lake schema classification device further comprises:

[0121] An extension module is configured to store the field in the partition schema corresponding to the positive infinite distance value into the queue.

[0122] Further, the distance module comprises:

[0123] An initialization unit is configured to initialize the value of the field stored in the queue as the first threshold value.

[0124] An overlay unit is configured to, when the field in the partition schema is the same as the field in the queue, add the first preset value to the value of the corresponding field in the queue, and update the calculation result as the value of the field.

[0125] An adding unit is configured to, when the field in the partition schema is different from the field in the queue, add the corresponding field in the partition schema into the queue, and make the value of the added field the second threshold value, which is different from the first threshold value.

[0126] A calculation unit is configured to sequentially accumulate the values of the fields in the same table schema to obtain the distance value of the table schema.

[0127] Further, the merging module comprises:

[0128] A first unit is configured to, when the field in the partition schema is the same as the field in the table schema, add the second preset value to the value of the corresponding field in the table schema.

[0129] A second unit is configured to, when the field in the partition schema is different from the field in the table schema, add the corresponding field in the partition schema into the table schema, and make the value of the added field the third preset value, which is different from the initial value of the field in the table schema.

[0130] A third unit is configured to retain all the fields in the table schema.

[0131] Further, the type module comprises:

[0132] A judging unit is configured to add the field in the partition schema and the field of any table schema into the same hash table, and judge whether the field in the hash table meets the condition.

[0133] A same type unit is configured to, when the condition is met, obtain the fields of at least two table schemas in the queue to obtain the same field in the queue.

[0134] The type different unit is configured to, when the judgment result does not satisfy the condition, acquire fields of at least two table schemas in the queue to determine fields of different types in the queue, and return a distance value between the field in the partition schema and the corresponding field of the table schema as positive infinity.

[0135] The specific limitation of the data lake schema classification device can refer to the limitation of the data lake schema classification method in the above, which will not be repeated here. Each module in the data lake schema classification device can be realized by software, hardware, and a combination thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above modules.

[0136] In an embodiment, a computer device, which can be a server, is provided. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a data lake schema classification method.

[0137] In an embodiment, a computer readable storage medium is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor implements the following steps when executing the computer program:

[0138] S1: creating a queue for storing table schemas and initializing;

[0139] S2: acquiring all fields in at least two table schemas and storing them in the queue;

[0140] S31: sequentially judging the fields in the partition schema with at least two table schemas in the queue in terms of field type and field name to obtain the same fields in the queue;

[0141] S4: sequentially calculating the distance between the fields in the partition schema and the same fields of at least two table schemas in the queue;

[0142] S5: determining the table schema in the queue corresponding to the minimum distance, and merging the partition schema into the table schema, and updating the merging result to the queue.

[0143] In one embodiment, a computer program product is provided, which includes a computer program, the computer program being executed by a processor to implement a data lake schema categorization method.

[0144] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the system is divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A data lake schema categorization method, characterized in that, The method comprises the following steps: creating a queue for storing table schemas and initializing; obtaining all fields in at least two table schemas and storing them in the queue; judging the field types and field names of the fields in the partition schema and the at least two table schemas in the queue in sequence to obtain the same fields in the queue; initializing the values of the fields stored in the queue as a first threshold value; when the field in the partition schema is the same as the field in the queue, adding a first preset value to the value of the corresponding field in the queue and updating the calculation result as the value of the field; when the field in the partition schema is different from the field in the queue, the field in the partition schema is newly added relative to the field in the queue, the corresponding field in the partition schema is added to the queue, and the value of the added field is a second threshold value, which is different from the first threshold value; sequentially accumulating the values of the fields in the same table schema to obtain the distance value of the table schema; determining the table schema in the queue corresponding to the minimum distance value and merging the partition schema into the table schema.

2. The data lake schema classification method of claim 1, wherein, The step of merging the corresponding field in the partition schema and the corresponding field in the table schema comprises: when the field in the partition schema is the same as the field in the table schema, adding a second preset value to the value of the corresponding field in the table schema; when the field in the partition schema is different from the field in the table schema, the field in the partition schema is newly added relative to the field in the queue, the corresponding field in the partition schema is added to the table schema, and the value of the added field is a third preset value, which is different from the initial value of the field in the table schema.

3. The data lake schema classification method of claim 2, wherein, The method further comprises the following steps: retaining all fields in the table schema.

4. The data lake schema classification method of claim 1, wherein, The step of judging the field types of the fields in the partition schema and the at least two table schemas in the queue in sequence comprises: adding the fields in the partition schema and the fields of any table schema to the same hash table and judging whether the fields in the hash table meet the condition; when the fields meet the condition, the field type in the partition schema is the same as the field type of the corresponding field in the at least two table schemas in the queue.

5. The data lake schema classification method of any one of claims 1-4, wherein, The step of judging the field types of the fields in the partition schema and the at least two table schemas in the queue in sequence to obtain the same fields in the queue further comprises the following steps: when the field type in the partition schema is different from the field type in the queue, returning the distance value of the field in the partition schema and the corresponding field in the table schema as positive infinity.

6. The data lake schema classification method of claim 5, wherein, The method further comprises the following steps: storing the field in the partition schema corresponding to the positive infinity distance value into the queue.

7. A data lake schema categorization apparatus, characterized by, The method comprises the following steps: a queue module is used to create a queue for storing table schemas and initialize; a benchmark module is used to obtain all types of fields in at least two table schemas and store them in the queue; a type module is used to sequentially compare the fields in the partition schema with the fields in the at least two table schemas in the queue in terms of field type and field name, and obtain the same fields in the queue; a distance module is used to initialize the value of the field stored in the queue as a first threshold value; when the field in the partition schema is the same as the field in the queue, the value of the corresponding field in the queue is increased by a first preset value, and the calculation result is updated as the value of the field; when the field in the partition schema is different from the field in the queue, the field in the partition schema is newly added relative to the field in the table schema in the queue, the corresponding field in the partition schema is added to the queue, and the value of the added field is a second threshold value, which is not equal to the first threshold value; the values of the fields in the same table schema are sequentially accumulated to obtain the distance value of the table schema; a merging module is used to determine the table schema in the queue corresponding to the minimum distance, and merge the partition schema into the table schema.

8. A computer device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the steps of the method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the steps of the method in any one of claims 1-6.

10. A computer program product, characterised in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the steps of the method in any one of claims 1-6.

Images