Data processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method and device, equipment and a storage medium, and relates to the technical field of big data analysis, the method comprises the steps of obtaining a query flow or a data writing completion notification message, determining that new semi-structured data is written into a data lake according to the query flow, or writing the new semi-structured data into the data lake according to the query flow; determining that new semi-structured data is written into the data lake according to the data writing completion notification message, obtaining a target data table where the new semi-structured data is located, a first file in the target data table where the new semi-structured data is located and a target field of the first file, and determining that the target data table meets an optimization condition, and if yes, carrying out storage space optimization on a target field of the first file in the target data table to obtain an optimized first file, and writing data in the optimized first file into a new second file. Therefore, the storage efficiency of the semi-structured data can be improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of big data analysis, and in particular to a data processing method and device, equipment and a storage medium. BACKGROUND

[0002] In the field of Internet and artificial intelligence, business requirements or use scenarios have the characteristics of high frequency changes, and the generated data also inherits the related characteristics, which cannot be stored using a predefined data structure. Therefore, in business, a semi-structured data such as a JSON string is generally used to store related data to avoid changes in data types or data structures. Directly using a JSON string for storage requires complete JSON parsing of the entire JSON string, which is inefficient and inefficient in extracting subfields of JSON data.

[0003] For such semi-structured data, one storage method in the prior art is to continue storing JSON strings nakedly and use high-level compression levels to compress the data to optimize the storage space.

[0004] However, in the above method, there is still room for optimization of storage space, and the storage efficiency of semi-structured data is low. SUMMARY

[0005] The present application provides a data processing method, device, equipment and storage medium, which can improve the storage efficiency of semi-structured data.

[0006] In a first aspect, embodiments of the present application provide a data processing method, comprising:

[0007] Obtaining a query stream or a data write completion notification message;

[0008] According to the query stream, it is determined that new semi-structured data is written into a data lake, or according to the data write completion notification message, it is determined that new semi-structured data is written into a data lake, obtaining a target data table where the new semi-structured data is located, a first file in the target data table and a target field of the first file, and the data lake has multiple data tables;

[0009] If it is determined that the target data table meets the optimization condition, the target field of the first file in the target data table is stored in the storage space, and an optimized first file is obtained;

[0010] Data in the optimized first file is written into a newly created second file.

[0011] In a second aspect, embodiments of the present application provide a data processing device, comprising:

[0012] The first obtaining module is configured to obtain a query flow or a data write completion notification message.

[0013] The second obtaining module is configured to determine, according to the query flow, that new semi-structured data is written into a data lake, or determine, according to the data write completion notification message, that new semi-structured data is written into the data lake, obtain a target data table in which the new semi-structured data is located, a first file in the target data table and a target field of the first file to which the new semi-structured data belongs, and the data lake includes a plurality of data tables.

[0014] The processing module is configured to determine that the target data table satisfies an optimization condition, and perform storage space optimization on the target field of the first file in the target data table to obtain an optimized first file.

[0015] The processing module is further configured to write data in the optimized first file into a newly created second file.

[0016] In an embodiment, the second obtaining module is configured to:

[0017] According to the query flow, obtain access feature information of data tables in a current data lake, and the data lake includes a plurality of data tables.

[0018] According to the access feature information of the data tables, determine that new semi-structured data is written.

[0019] In an embodiment, the query flow includes query records corresponding to query requests in a first time period, and the second obtaining module is configured to:

[0020] According to the query records corresponding to the query requests in the first time period, obtain an identifier of a data table accessed in the first time period, a field in the data table accessed in the first time period and an identifier of a file in which the field is located.

[0021] The identifier of the data table accessed in the first time period, the field in the data table accessed in the first time period and the identifier of the file in which the field is located are determined as the access feature information of the data tables in the current data lake.

[0022] In an embodiment, the second obtaining module is configured to:

[0023] If the field in the data table accessed in the first time period is a semi-structured field, it is determined that new semi-structured data is written.

[0024] In an embodiment, the second obtaining module is specifically configured to:

[0025] The field in the data table accessed in the access feature information is determined as the target field.

[0026] determining a data table in which the target field is located as the target data table, and determining an identity of a file to which the target field belongs as the identity of the first file.

[0027] In an embodiment, the data write completion notification message indicates that semi-structured data writing is completed, and the second obtaining module is configured to:

[0028] determine a field to which the new semi-structured data belongs as the target field;

[0029] determine a data table in which the target field is located as the target data table, and determine an identity of a file to which the target field belongs as the identity of the first file.

[0030] In an embodiment, the processing module is configured to:

[0031] create a temporary table;

[0032] store data of a part of files in the target data table into the temporary table;

[0033] optimize storage space by performing a preset processing on the data in the temporary table, and obtain optimization index information of the temporary table after storage space optimization, the optimization index information including time consumption of storage space optimization, improvement rate of storage space, and improvement rate of query speed;

[0034] if the optimization index information satisfies at least one of the following conditions, it is determined that the target data table satisfies an optimization condition:

[0035] the time consumption of storage space optimization is less than a preset time, the improvement rate of storage space is greater than a preset threshold, and the improvement rate of query speed is greater than a preset value.

[0036] In an embodiment, the processing module is configured to:

[0037] perform column splitting and rewriting on semi-structured data of the target field to obtain a target field after column splitting and rewriting;

[0038] obtain the first file after optimization according to fields other than the target field in the first file and the target field after column splitting and rewriting.

[0039] In an embodiment, the processing module is specifically configured to:

[0040] perform parsing on semi-structured data of the target field, and determine the same part of each row subfield according to the parsing result;

[0041] rewriting the semi-structured data of the target field according to the same part by column, and writing a non-same part in the semi-structured data of the target field into an encoded field;

[0042] combining the column-rewritten field and the encoded field to obtain the column-rewritten target field.

[0043] In an embodiment, the apparatus further comprises:

[0044] a query module configured to, after receiving a query request, parse the query request to obtain a field to be queried;

[0045] determine a first field from the field to be queried, the first field being a subfield of a semi-structured field or all fields of a semi-structured field;

[0046] if it is determined by a reader that a file in which the first field is located has been optimized in storage space, read the first field from the newly created second file;

[0047] if it is determined by the reader that the file in which the first field is located has not been optimized in storage space, read the first field from the first file;

[0048] obtain a query result according to the first field.

[0049] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, the memory being configured to store a computer program, and the processor being configured to invoke and run the computer program stored in the memory to execute the method in the first aspect.

[0050] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium comprising instructions which, when executed on a computer, cause the computer to execute the method in the first aspect.

[0051] In a fifth aspect, an embodiment of the present application provides a computer program product comprising computer program instructions, the computer program instructions causing a computer to execute the method in the first aspect.

[0052] In a sixth aspect, an embodiment of the present application provides a computer program, the computer program causing a computer to execute the method in the first aspect.

[0053] In summary, in the embodiments of the present application, when it is determined according to the query flow that there is new semi-structured data writing, or when it is determined according to the data writing completion notification message that there is new semi-structured data writing, the target data table in which the new semi-structured data is located, the first file in the target data table, and the target field of the first file to which the new semi-structured data belongs are obtained. There are multiple data tables in the data lake. When it is determined that the target data table meets the optimization condition, the target field of the first file in the target data table is optimized in storage space, an optimized first file is obtained, and the data in the optimized first file is written into a newly created second file. The newly written semi-structured data is asynchronously optimized in storage space, which does not affect the data writing process, thereby improving the storage efficiency of semi-structured data. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0055] Figure 1 An application scenario of the embodiments of the present application is shown in a schematic diagram;

[0056] Figure 2 A flowchart of a data processing method provided by the embodiments of the present application is shown;

[0057] Figure 3 A system architecture diagram of a data processing method provided by the embodiments of the present application is shown;

[0058] Figure 4 A process schematic diagram of a data processing method provided by the embodiments of the present application is shown;

[0059] Figure 5 A data query process schematic diagram provided by the embodiments of the present application is shown;

[0060] Figure 6 A schematic block diagram of a data processing apparatus of the embodiments of the present application is shown;

[0061] Figure 7 A schematic block diagram of a data processing apparatus of the embodiments of the present application is shown;

[0062] Figure 8 A schematic block diagram of an electronic device 30 provided by the embodiments of the present application is shown. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0064] It should be understood that, in the embodiments of the present application, "B corresponding to A" means that B is associated with A. In an implementation, B can be determined according to A. However, it should also be understood that determining B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information.

[0065] In the description of the present application, "at least one" means one or more, and "multiple" means two or more than two, unless otherwise specified. The character " / " generally represents that the objects before and after are in an "or" relationship. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0066] It should be noted that the terms "first", "second", and the like in the embodiments of the present application are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0067] It should also be understood that the specific features, structures or characteristics related to the embodiments in the description are included in at least one embodiment of the present application. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner.

[0068] First, the related terms involved in the embodiments of the present application are introduced.

[0069] 1. Data Lake: a storage or system that stores data in raw format, which stores data as it is without prior structured processing of data. Structured data, semi-structured data, unstructured data are all included in the data storage layer, and this raw data storage area is called Data Lake.

[0070] 2. Open table format (Iceberg): a data organization format between the computing engine and the data storage format, which organizes data and metadata in a specific way. It is a Lakehouse technology that targets structured data in the data lake, helping users to build high-performance data warehouses in the data lake, and realizes lake-warehouse integration.

[0071] 3. Semi-structured data: data between structured data (such as data in a relational database) and completely unstructured data (images, videos, etc.), which does not have a clear data structure determined in advance, but has similar data structures between multiple rows of data. Common semi-structured data includes JSON (JavaScript Object Notation) and XML.

[0072] In related technologies, for semi-structured data, when storing, store the JSON string directly, and use a high-level compression level to compress the data to optimize the storage space. However, there is still room for optimization of storage space, and it is not possible to put all the data into a compressed block for compression, the repetition in semi-structured data cannot be fully utilized, and the storage efficiency is low.

[0073] To solve this problem, the embodiment of the present application determines that there is new semi-structured data writing according to the query stream or according to the data writing completion notification message, obtains the target data table where the new semi-structured data is located, the first file in the target data table, and the target field of the first file, and when it is determined that the target data table meets the optimization condition, the target field of the first file in the target data table is stored. Space optimization, get the optimized first file, write the data in the optimized first file to the newly created second file. Asynchronously optimize the storage space of the newly written semi-structured data, without affecting the data writing process, thereby improving the storage efficiency of semi-structured data.

[0074] Further, in related technologies, only storage space optimization is performed, and when multiple subfields are extracted repeatedly, the subfield extraction efficiency is poor, the query speed is slow, and the query efficiency is low. In the embodiment of the present application, when the newly written semi-structured data is optimized for storage space, the semi-structured data of the target field is specifically rewritten, and the rewritten target field is obtained. According to the fields in the first file except the target field and the rewritten target field, the optimized first file is obtained. Since the column-stored data has better compression and column separation, space saving and query speedup can be obtained. While improving the storage efficiency, the subsequent subfield extraction can directly read the column-separated fields, improving the subfield extraction efficiency and thus improving the query efficiency.

[0075] The embodiment of the application can be applied to the technical field of big data analysis. The embodiment of the application can be applied to various scenarios, including but not limited to an online transaction processing (OLAP) scenario, and can solve the problems of low storage efficiency, poor sub-field extraction efficiency and low query efficiency existing in semi-structured data storage in a data lake.

[0076] Figure 1 An example of an application scenario of the embodiment of the application is shown.

[0077] As Figure 1 shown, the application scenario involves a server 1 and a terminal device 2. The terminal device 2 can communicate data with the server 1 through a communication network. The server 1 can be a background server of the terminal device 2.

[0078] For example, the terminal device 2 can refer to a kind of device with rich human-computer interaction mode, internet access capability, various operating systems, strong processing capability, etc. The terminal device can be a smart phone, a tablet computer, a portable notebook computer, a desktop computer, a wearable device, a vehicle-mounted device, etc., but is not limited thereto. Optionally, in the embodiment of the application, the terminal device 2 is installed with various client application programs (APPs) of video applications, including but not limited to a search APP, an APP with information query function, etc.

[0079] The server 1 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. The server can also be a node of a blockchain.

[0080] The server can be one or more. When the server is multiple, at least two servers are used to provide different services, and / or at least two servers are used to provide the same service, such as providing the same service in a load balancing manner, which is not limited in the embodiment of the application.

[0081] The terminal device and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the application. The number of servers or terminal devices is not limited in the application. The scheme provided in the application can be completed independently by the server, and can also be completed by the terminal device and the server, which is not limited in the application.

[0082] It should be understood that,Figure 1 The above is only an example and does not specifically limit the application scenarios of the embodiments of the present application. For example, Figure 1 The terminal device and the server are exemplarily shown, and actually, other numbers of terminal devices and servers can be included, which are not limited by the present application.

[0083] The data processing method provided by the embodiments of the present application can be executed by Figure 1 The server shown in the figure, specifically, can be executed by an Auto Optimizer arranged in the server, which can determine the semi-structured data that can be optimized in the data lake, and the target data table and the target field where the semi-structured data is located, and then optimize the storage space of the semi-structured data stored in the target data table and the target field. The Auto Optimizer can be a background resident service for automatically optimizing data access and storage space in the open table format data lake.

[0084] The technical solutions of the embodiments of the present application will be described in detail below through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes can not be described in detail in some embodiments.

[0085] Figure 2 A flowchart of a data processing method provided by the embodiments of the present application is shown in the figure. The method can be executed by any electronic device with data processing capability, for example, the electronic device can be implemented as a server, for example, can be implemented as Figure 1 the server 1 in the figure, which is not limited by the present application. As shown in the figure, Figure 2 the method of the present embodiment can include:

[0086] S101, acquiring a query flow or a data write completion notification message.

[0087] Specifically, the business side data is written into the data lake by the computing engine and stored in the data table of the open table format (Iceberg). The business side data is stored in the data table of the open table format (Iceberg) in the data lake, and specifically can be stored in the semi-structured data column of the first file of the data table, for example, stored in the original JSON string.

[0088] When data is queried, the object is used to initiate a query of semi-structured data (such as using get_json_object) by a client to perform subfield extraction. For example, the client sends a query request to the computing engine, and the computing engine performs a query according to the query request to obtain corresponding query records. The query records include the identifier of the accessed table, the field in the accessed table, and the file identifier where the field is located. The query records in a period of time form a query log. Optionally, in an embodiment, the query log can be acquired periodically, and the query log is acquired when a preset time is reached.

[0089] In an embodiment, the object side can mark semi-structured data that can be optimized for storage space on demand. The specific implementation is that the computing engine sends a data write completion notification message when semi-structured data writing is completed. When the server acquires the data write completion notification message, it can determine that there is new semi-structured data writing according to the data write completion notification message.

[0090] S102, according to the query log, determine that there is new semi-structured data writing to the data lake, or, according to the data write completion notification message, determine that there is new semi-structured data writing to the data lake, acquire the target data table where the new semi-structured data is located, the first file in the target data table, and the first file target field to which it belongs. The data lake has multiple data tables.

[0091] Specifically, there are two ways to determine that there is new semi-structured data writing to the data lake in this embodiment, one is to determine according to the query log, and the other is to determine according to the data write completion notification message.

[0092] Optionally, in an implementable manner, S102 determines that there is new semi-structured data writing according to the query log, which can be specifically:

[0093] S1021, according to the query log, acquire the access feature information of the data table in the current data lake. The data lake includes multiple data tables.

[0094] Specifically, the query log includes query records corresponding to query requests in a period of time. The query records include the identifier of the accessed table, the field in the accessed table, and the file identifier where the field is located. In this embodiment, the format of the data table can be Iceberg, which is a data organization format between the computing engine and the data storage format. It organizes data and metadata through a specific way. The data lake stores multiple data tables.

[0095] Optionally, in an implementable manner, S1021 can be specifically:

[0096] S10211, acquiring, according to the query record corresponding to the query request in the first time period, the identifier of the data table accessed in the first time period, the field in the data table accessed and the identifier of the file where the field is located.

[0097] Specifically, for each query record corresponding to the query request in the first time period, the identifier of the accessed table, the field in the accessed table and the identifier of the file where the field is located can be acquired according to one query record. For example, there are 5 query records in the first time period, and 5 identifiers of the data tables accessed, fields in the data tables accessed and identifiers of the files where the fields are located can be acquired. Accordingly, the identifier of the data table accessed in the first time period, the field in the data table accessed and the identifier of the file where the field is located are 5.

[0098] S10212, determining the identifier of the data table accessed in the first time period, the field in the data table accessed and the identifier of the file where the field is located as the access feature information of the data table in the current data lake.

[0099] S1022, determining that there is new semi-structured data writing according to the access feature information of the data table.

[0100] Optionally, in an implementable manner, S1022 can be specifically:

[0101] S10221, if the field in the data table accessed in the first time period is a semi-structured field, it is determined that there is new semi-structured data writing to the data lake.

[0102] Optionally, in an implementable manner, the target data table where the new semi-structured data is located, the first file in the target data table and the target field belonging to the first file in S102 can be specifically:

[0103] S1, determining the field in the data table accessed in the access feature information as the target field.

[0104] S2, determining the data table where the target field is located as the target data table, and determining the identifier of the file to which the target field belongs as the identifier of the first file.

[0105] It can be understood that the number of fields in the data table accessed in the access feature information which are semi-structured fields can be one or more, and the target field includes one or more.

[0106] In another implementable manner, the data writing completion notification message indicates that the semi-structured data writing is completed, and the target data table where the new semi-structured data is located, the first file in the target data table and the target field belonging to the first file in S102 can be specifically:

[0107] S11, determine the field to which the new semi-structured data belongs as a target field.

[0108] S12, determine the data table in which the target field is located as a target data table, and determine the identity of the file to which the target field belongs as the identity of the first file.

[0109] Specifically, the computing engine sends a data write completion notification message when the semi-structured data writing is completed, and the server can determine that there is new semi-structured data writing according to the data write completion notification message. Optionally, the data write completion notification message can carry the field to which the new semi-structured data belongs, so the field to which the new semi-structured data belongs can be determined as the target field, the data table in which the target field is located can be determined as the target data table, and the identity of the file to which the target field belongs can be determined as the identity of the first file.

[0110] S103, determining that the target data table satisfies an optimization condition, and performing storage space optimization on the target field of the first file in the target data table to obtain an optimized first file.

[0111] Specifically, the target data table is the target data table, and determining that the target data table satisfies the optimization condition can be an A / B test on the target data table. The optimization index information of the target data table is obtained through the A / B test, and the optimization index information includes the time consumption of storage space optimization, the improvement rate of storage space, and the improvement rate of query speed. When the optimization index information meets the preset index requirement, it is determined that the optimization condition is satisfied. A specific A / B test implementation is shown below.

[0112] Specifically, in an implementable manner, the determination that the target data table satisfies the optimization condition in S103 can be specifically:

[0113] S1031, creating a temporary table.

[0114] Specifically, a temporary data table is created.

[0115] S1032, storing data of a part of files in the target data table into the temporary table.

[0116] S1033, optimizing the storage space by performing a preset processing on the data in the temporary table, and obtaining optimization index information of the temporary table after the storage space optimization. The optimization index information includes the time consumption of storage space optimization, the improvement rate of storage space, and the improvement rate of query speed.

[0117] The storage space can be optimized by preset processing of the data in the temporary table, which can be the same optimization method as the storage space optimization of the target field. After the storage space optimization is completed, the optimization index information of the temporary table after the storage space optimization can be obtained. The improvement rate of the storage space refers to the improvement rate of the storage space of the temporary table after the storage space is optimized by preset processing of the data in the temporary table. The improvement rate of the query speed can be the improvement rate of the query speed of the temporary table after the storage space is optimized by preset processing of the data in the temporary table.

[0118] S1034, if the optimization index information satisfies at least one of the following conditions, it is determined that the target data table satisfies the optimization condition:

[0119] The time consumption of the storage space optimization is less than the preset time, the improvement rate of the storage space is greater than the preset threshold, and the improvement rate of the query speed is greater than the preset value.

[0120] For example, the optimization index information includes the time consumption of the storage space optimization, the improvement rate of the storage space, and the improvement rate of the query speed. When the time consumption of the storage space optimization is less than the preset time, the improvement rate of the storage space is greater than the preset threshold, and the improvement rate of the query speed is greater than the preset value, it is determined that the target data table satisfies the optimization condition. Alternatively, when any one of the three conditions is satisfied, it is determined that the target data table satisfies the optimization condition, such as when the time consumption of the storage space optimization is less than the preset time, or the improvement rate of the storage space is greater than the preset threshold, or the improvement rate of the query speed is greater than the preset value.

[0121] Optionally, after it is determined that the optimization condition is satisfied, the target field in the target data table is subjected to storage space optimization, and the temporary table is deleted.

[0122] Specifically, in an implementable manner, the target field in the target data table is subjected to storage space optimization in S103, and an optimized first file is obtained, which can be specifically:

[0123] S31, the semi-structured data of the target field is subjected to column splitting and rewriting to obtain the target field after column splitting and rewriting.

[0124] Specifically, the column splitting and rewriting of semi-structured data refers to a process of reorganizing data with partial organizational characteristics but not conforming to a strict relational model into a structured format of standardized, easy-to-analyze and store data by sampling and deriving a common structure. This process is transparent to the upper layer user.

[0125] Optionally, in an implementable manner, the semi-structured data of the target field is subjected to column splitting and rewriting in S31 to obtain the target field after column splitting and rewriting, which can be specifically:

[0126] S311, parse the semi-structured data of the target field, and determine the same part of each row subfield according to the parsing result.

[0127] Specifically, the semi-structured data is data between structured data (such as data in a relational database) and completely unstructured data (images, videos, etc.), which does not have a clear data structure determined in advance, but has similar data structures between multiple rows of data. The target field stores semi-structured data, and the semi-structured data of the target field is parsed, and the same part of each row subfield can be determined according to the parsing result. Specifically, the same part of each row subfield can be calculated, that is, the data structure of the same part.

[0128] Specifically, the semi-structured data of the target field is parsed, which can be data parsing according to the type of semi-structured data. For example, the semi-structured data is the following JSON data:

[0129]

[0130] According to the JSON data type, the data is parsed to obtain the parsing result, that is, the content of each row subfield. According to the parsing result, the same part of each row subfield is determined.

[0131] S312, rewrite the semi-structured data of the target field according to the same part by column, and write the non-same part of the semi-structured data of the target field into the encoded field.

[0132] Specifically, the semi-structured data of the target field is rewritten into a table form according to the same part by column, and the non-same part of the semi-structured data of the target field is written into the encoded field.

[0133] For example, in an example, the above example content is rewritten into the following table form:

[0134] User.id User.name User.contact.email User.contact.phone 123 XX XXXX.com ]]> ​ 10XXXXX

[0135] S313, combine the column-rewritten field and the encoded field to obtain the column-rewritten target field.

[0136] S32, obtain the optimized first file according to the field other than the target field in the first file and the column-rewritten target field.

[0137] In this embodiment, by performing storage space optimization on newly written semi-structured data, specifically, by performing columnar rewriting on semi-structured data of the target field to obtain the columnar rewritten target field, and according to the fields in the first file except the target field and the columnar rewritten target field, the optimized first file is obtained. Since the columnar stored data has better compression and columnar, space saving and query speedup can be obtained. While improving storage efficiency, subsequent subfield extraction can directly read the columnar field, improving the subfield extraction efficiency and further improving the query efficiency.

[0138] S104, write the data in the optimized first file into the newly created second file.

[0139] Optionally, after writing the data in the optimized first file into the newly created second file, the data in the original first file can be deleted. The second file is used to store semi-structured data that has been optimized in storage space.

[0140] In this embodiment, since the newly written semi-structured data is optimized in storage space, further corresponding optimization processing of the query is also performed. Optionally, in an embodiment, the method of the present embodiment can further include S105-S109 (not shown in the figure) after S104: Figure 2

[0141] S105, after receiving the query request, the query request is parsed to obtain the field to be queried.

[0142] Specifically, the query request carries code data, and the code data can be parsed to obtain the field to be queried.

[0143] S106, the first field is determined from the field to be queried, and the first field is a subfield of the semi-structured field or all fields of the semi-structured field.

[0144] S107, if it is determined by the reader that the file where the first field is located is optimized in storage space, the first field is read from the newly created second file.

[0145] S108, if it is determined by the reader that the file where the first field is located is not optimized in storage space, the first field is read from the first file.

[0146] S109, the query result is obtained according to the first field.

[0147] Optionally, after the columnar rewriting of the semi-structured data of the target field is completed, the query on the columnar rewritten semi-structured data can be performed by the server or the computing engine, and the optimizer of the computing engine can be automatically rewritten to query the subcolumn or fallback to the binary encoded query. In an embodiment, S105-S109 can be specifically executed.​

[0148] The data processing method provided by the embodiment can obtain a target data table in which new semi-structured data is located, a first file in the target data table, and a target field of the first file, when it is determined that there is new semi-structured data writing according to a query stream or when it is determined that there is new semi-structured data writing according to a data writing completion notification message. When it is determined that the target data table meets an optimization condition, the target field in the target data table is subjected to storage space optimization, an optimized first file is obtained, and data in the optimized first file is written into a newly created second file. The newly written semi-structured data is subjected to storage space optimization asynchronously, and the data writing process is not affected, so that the storage efficiency of semi-structured data can be improved.

[0149] The processing procedure of the data processing method will be described in detail below with reference to a specific embodiment. Figures 3 to 5

[0150] The system architecture diagram of the data processing method provided by the embodiment is shown in FIG. 1. Figure 3 The system includes an automatic optimizer 10, a first computing engine 20, and a second computing engine 30. In the embodiment, the execution subject of the method shown in FIG. 1 is taken as an example, and the optimizer and the computing engine can be deployed on different servers. There can be one or more computing engines, and two computing engines are taken as an example in the embodiment. Figure 3 Figure 2 The automatic optimizer 10 can be a background resident service for automatically optimizing data access and storage space of a data table in an open table format in a data lake. The first computing engine 20 and the second computing engine 30 are units for reading and writing data. Specifically, the flow of the data processing method in the embodiment includes the following steps.

[0151] The automatic optimizer 10 can be a background resident service for automatically optimizing data access and storage space of a data table in an open table format in a data lake. The first computing engine 20 and the second computing engine 30 are units for reading and writing data. Specifically, the flow of the data processing method in the embodiment includes the following steps.

[0152] S201, semi-structured data on the business side is written into a data lake through the first computing engine and the second computing engine, and is stored in a data table in an open table format.

[0153] Specifically, data on the business side is written into a data lake through a computing engine, and is stored in a data table in an open table format (Iceberg). The data on the business side is stored in a data table in an open table format (Iceberg) in a data lake, and can be specifically stored in a semi-structured data column of a first type file of the data table, for example, in a raw JSON string.

[0154] S202, a client initiates a query request through the first computing engine and the second computing engine to perform subfield extraction.

[0155] ​Specifically, when querying data, the object initiates a query of semi-structured data (such as using get_json_object) through the client to perform subfield extraction. For example, the client sends a query request to the computing engine, the computing engine performs a query according to the query request, and obtains corresponding query records, which include the identifier of the accessed table, the field in the accessed table, and the file identifier where the field is located. The query records in a period of time form a query log.

[0156] S203, the automatic optimizer obtains the query log or the data write completion notification message.

[0157] Optionally, in an embodiment, the automatic optimizer obtains the query log, which can be obtained periodically, and obtains the query log when a preset time is reached.

[0158] Optionally, in an embodiment, the object side can mark semi-structured data that can be optimized for storage on demand. Specifically, the computing engine sends a data write completion notification message when semi-structured data is written, and the automatic optimizer obtains the data write completion notification message to determine that new semi-structured data is written.

[0159] S204, the automatic optimizer determines that new semi-structured data is written to the data lake according to the query log, or determines that new semi-structured data is written to the data lake according to the data write completion notification message, obtains the target data table where the new semi-structured data is located, the first file in the target data table, and the target field of the first file, and performs the semi-structured data storage space optimization process.

[0160] Specifically, the first file is a first type file, and in this embodiment, there are two ways to determine that new semi-structured data is written to the data lake, one is to determine according to the query log, and the other is to determine according to the data write completion notification message.

[0161] Optionally, in an implementable manner, S204 determines that new semi-structured data is written to the data lake according to the query log, which can be specifically as follows:

[0162] S2041, according to the query log, access feature information of data tables in the current data lake is obtained, and the data lake includes a plurality of data tables.

[0163] Specifically, the query stream includes query records corresponding to query requests in a period of time, and the query records include an identifier of an accessed table, a field in the accessed table, and an identifier of a file where the field is located. In this embodiment, the format of the data table can be an open table format (Iceberg). The Iceberg is a data organization format between a computing engine and a data storage format, which organizes data and metadata in a specific manner. The data lake stores a plurality of data tables.

[0164] Optionally, in an implementable manner, S2041 can be specifically:

[0165] S20411, according to the query records corresponding to the query requests in the first time period, obtaining the identifiers of the data tables accessed in the first time period, the fields in the data tables accessed, and the identifiers of the files where the fields are located.

[0166] Specifically, for each query record corresponding to a query request in the first time period, the identifier of the accessed table, the field in the accessed table, and the identifier of the file where the field is located can be obtained according to one query record. For example, there are 5 query requests in the first time period, and 5 identifiers of the data tables accessed, the fields in the data tables accessed, and the identifiers of the files where the fields are located can be obtained. Accordingly, the identifiers of the data tables accessed in the first time period, the fields in the data tables accessed, and the identifiers of the files where the fields are located are 5.

[0167] S20412, determining the identifiers of the data tables accessed in the first time period, the fields in the data tables accessed, and the identifiers of the files where the fields are located as the access feature information of the data tables in the current data lake.

[0168] S2042, determining that new semi-structured data is written into the data lake according to the access feature information of the data tables.

[0169] Optionally, in an implementable manner, S2042 can be specifically:

[0170] S20421, if the fields in the data tables accessed in the first time period are semi-structured fields, it is determined that new semi-structured data is written into the data lake.

[0171] Optionally, in an implementable manner, the target data table where the new semi-structured data is located, the first file in the target data table where the new semi-structured data is located, and the target field belonging to the first file in S204 can be specifically:

[0172] S21, determining the fields in the data tables accessed in the access feature information as the target fields.

[0173] S22, determine the data table in which the target field is located as a target data table, and determine the identity of the file to which the target field belongs as the identity of the first file.

[0174] It can be understood that the number of fields in the accessed data table in the accessed feature information can be one or more, and the target field includes one or more.

[0175] In another implementable manner, the data write completion notification message indicates that the semi-structured data write is completed, and the target data table in which the new semi-structured data is located, the target field, and the first file to which the target field belongs are obtained in S204. Specifically, the target data table in which the new semi-structured data is located, the target field, and the first file to which the target field belongs can be determined as follows:

[0176] S31, determine the field to which the new semi-structured data belongs as the target field.

[0177] S32, determine the data table in which the target field is located as a target data table, and determine the identity of the file to which the target field belongs as the identity of the first file.

[0178] Specifically, the computing engine sends a data write completion notification message when the semi-structured data write is completed, and the server can determine that there is new semi-structured data written into the data lake according to the data write completion notification message. Optionally, the data write completion notification message can carry the field to which the new semi-structured data belongs, so the field to which the new semi-structured data belongs can be determined as the target field, the data table in which the target field is located can be determined as the target data table, and the identity of the file to which the target field belongs can be determined as the identity of the first file.

[0179] Specifically, the process of optimizing the semi-structured data storage space in S204 can be as follows:

[0180] S41, determine that the target data table meets the optimization condition, and then optimize the storage space of the target field of the first file in the target data table to obtain an optimized first file.

[0181] S42, write the data in the optimized first file into a newly created second file.

[0182] Optionally, after writing the data in the optimized first file into the newly created second file, the data in the original first file can be deleted. The second file is used to store the semi-structured data after the storage space optimization. The second file is a second type file.

[0183] Specifically, the target data table is a target data table, and it is determined that the target data table meets the optimization condition. The target data table can be subjected to an A / B test, and optimization index information of the target data table is obtained through the A / B test. The optimization index information includes time consumption of storage space optimization, an improvement rate of storage space, and an improvement rate of query speed. When the optimization index information meets preset index requirements, it is determined that the optimization condition is met. The following will be described in combination with Figure 4 An embodiment of performing semi-structured data storage space optimization is shown.

[0184] Figure 4 A process schematic diagram of a data processing method provided by the embodiment is shown in FIG. 1. The method of the embodiment can include the following steps. Figure 4

[0185] S301, creating a temporary table.

[0186] Specifically, a temporary data table is created.

[0187] S302, storing data of a part of files sampled from a target data table into the temporary table.

[0188] S303, optimizing storage space by preset processing of data in the temporary table, and obtaining optimization index information of the temporary table after storage space optimization. The optimization index information includes time consumption of storage space optimization, an improvement rate of storage space, and an improvement rate of query speed.

[0189] The optimization of storage space by preset processing of data in the temporary table can be the same optimization mode as the storage space optimization of the target field. For example, the data in the temporary table can be subjected to column splitting and rewriting. After the storage space optimization is completed, the optimization index information of the temporary table after storage space optimization can be obtained. The improvement rate of storage space refers to the improvement rate of storage space of the temporary table after the optimization of storage space by preset processing of data in the temporary table. The improvement rate of query speed refers to the improvement rate of query speed of the temporary table after the optimization of storage space by preset processing of data in the temporary table.

[0190] S304, determining whether the optimization condition is met according to the optimization index information.

[0191] Specifically, when it is determined that the optimization index information meets preset index requirements, it is determined that the optimization condition is met, and S305 is performed. When it is determined that the optimization index information does not meet the preset index requirements, it is determined that the optimization condition is not met, and S306 is performed.

[0192] Specifically, if the optimization index information meets at least one of the following conditions, it is determined that the target data table meets the optimization condition:

[0193] ​The time consumption of the storage space optimization is less than a preset time, the improvement rate of the storage space is greater than a preset threshold, and the improvement rate of the query speed is greater than a preset value.

[0194] For example, the optimization index information includes the time consumption of the storage space optimization, the improvement rate of the storage space, and the improvement rate of the query speed. When the time consumption of the storage space optimization is less than a preset time, the improvement rate of the storage space is greater than a preset threshold, and the improvement rate of the query speed is greater than a preset value, it is determined that the target data table meets the optimization condition. Alternatively, when any one of the three conditions is met, it is determined that the target data table meets the optimization condition, such as the time consumption of the storage space optimization being less than a preset time, or the improvement rate of the storage space being greater than a preset threshold, or the improvement rate of the query speed being greater than a preset value.

[0195] S305, performing storage space optimization on a target field in a target data table to obtain an optimized first file.

[0196] S306, deleting the temporary table.

[0197] Specifically, in an implementable manner, the storage space optimization on the target field in the target data table in S305 obtains an optimized first file, which can specifically be:

[0198] S3051, performing column splitting and rewriting on semi-structured data of the target field to obtain the target field after column splitting and rewriting.

[0199] Optionally, in an implementable manner, the column splitting and rewriting on the semi-structured data of the target field in S3051 obtains the target field after column splitting and rewriting, which can specifically be:

[0200] S51, analyzing the semi-structured data of the target field, and determining the same part of each row subfield according to the analysis result.

[0201] Specifically, semi-structured data is data between structured data (such as data in a relational database) and completely unstructured data (images, videos, etc.), which does not have a clear data structure determined in advance, but has similar data structures between multiple rows of data. The target field stores semi-structured data, and the semi-structured data of the target field is analyzed, and the same part of each row subfield can be determined according to the analysis result, which can specifically calculate the same part of each row subfield, that is, the data structure of the same part.

[0202] S52, rewriting the semi-structured data of the target field according to the same part, and writing the non-same part of the semi-structured data of the target field into the encoded field.

[0203] S53, combine the column-rewritten field and the encoded field to obtain a target field after column-rewriting.

[0204] In this embodiment, based on the characteristics of atomicity, consistency, isolation and durability of the data lake, the rewriting operation of the data file does not affect the normal query.

[0205] S54, obtain an optimized first file according to the fields in the first file except the target field and the target field after column-rewriting.

[0206] In the embodiments of the present application, the above-mentioned processing process and data writing flow for optimizing the semi-structured data storage space are performed asynchronously, and the data writing flow is normally performed, so that the original semi-structured data can still be written into the data lake and stored in the data table in the open table format.

[0207] In the embodiments of the present application, in addition to the storage space optimization of the newly written semi-structured data, the automatic optimizer also performs rewriting optimization on the historical data, thereby reducing the storage space occupation of the historical data.

[0208] In the embodiments of the present application, since the storage space optimization is performed on the newly written semi-structured data, further corresponding optimization processing of the query is also performed. The processing process of the query optimization will be described in detail below. Figure 5

[0209] Figure 5 A data query process schematic diagram is provided in the embodiments of the present application, and the method of the embodiments can be executed by a server, and specifically can be executed by a computing engine, as shown in the figure. Figure 5 The method of the embodiments can include the following steps.

[0210] S401, receive a query request.

[0211] S402, parse the query request to obtain a field to be queried.

[0212] S403, determine a first field from the field to be queried, and the first field is a subfield of a semi-structured field or all fields of the semi-structured field.

[0213] S404, determine whether the file where the first field is located is optimized in storage space by a reader.

[0214] If yes, execute S405, and if no, execute S406.

[0215] S405, construct a reader corresponding to each field according to the field to be queried, and read the first field from a newly created second file.

[0216] ​S406, reading the original semi-structured data, according to the field to be queried, falling back to the original extraction logic, reading the first field from the first file.

[0217] S407, obtaining the query result according to the first field.

[0218] In the embodiment, the data in the optimized first file is written into the newly created second file due to the storage space optimization of the newly written semi-structured data. When receiving a query request, the query request is analyzed to obtain the field to be queried, and the first field is determined from the field to be queried. The first field is a subfield of the semi-structured field or all fields of the semi-structured field. It is determined by the reader whether the file where the first field is located is optimized in storage space. If yes, the first field is read from the newly created second file. If not, the first field is read from the first file. The optimization of the query is realized, the query can be performed according to the file optimized in storage space, and the query efficiency is improved.

[0219] The data processing method provided in the embodiment, when it is determined that there is new semi-structured data written into the data lake according to the query flow or when it is determined that there is new semi-structured data written into the data lake according to the data write completion notification message, the target data table where the new semi-structured data is located, the first file in the target data table, and the target field of the first file are obtained. There are multiple data tables in the data lake. When it is determined that the target data table meets the optimization condition, the storage space optimization is performed on the target field in the target data table, the optimized first file is obtained, and the data in the optimized first file is written into the newly created second file. The storage space optimization of the newly written semi-structured data is performed asynchronously, which does not affect the data write process, thereby improving the storage efficiency of the semi-structured data. Moreover, when the storage space optimization of the newly written semi-structured data is performed, the semi-structured data of the target field is specifically disassembled and rewritten to obtain the disassembled and rewritten target field. The optimized first file is obtained according to the fields in the first file except the target field and the disassembled and rewritten target field. Since the column storage data has better compression and disassembly, space saving and query speedup can be obtained. While improving the storage efficiency, the subsequent subfield extraction can directly read the disassembled field, thereby improving the subfield extraction efficiency and further improving the query efficiency.

[0220] In the embodiment of the application, the data is used for testing and verification. After using the data processing method provided in the application, the storage space is reduced by 50%-60%, and the query efficiency is improved by more than 5 times. The data processing method of the embodiment realizes the optimization of the storage space and the improvement of the query efficiency while having no perceptible intervention for the upstream and downstream users. The user does not need to adjust or adjust the parameters, thereby reducing the use or access cost of the user.

[0221] The specific embodiments of the present application are described in detail above with reference to the accompanying drawings, but the present application is not limited to the specific details in the above-described embodiments. Within the technical concept of the present application, various simple modifications can be made to the technical solutions of the present application, and these simple modifications all belong to the protection scope of the present application. For example, in the above-described specific embodiments, various specific technical features are described, and in the case of no contradiction, any suitable combination can be made. In order to avoid unnecessary repetition, various possible combination manners are not described again in the present application. For another example, various different embodiments of the present application can also be combined arbitrarily, as long as it does not deviate from the idea of the present application, and it should also be considered as the disclosed content of the present application.

[0222] It should also be understood that in various method embodiments of the present application, the size of the serial number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. It should be understood that these serial numbers can be interchanged under appropriate circumstances, so that the described embodiments of the present application can be implemented in an order other than those illustrated or described.

[0223] The method embodiments of the present application are described in detail above, and the device embodiments of the present application are described in detail below. Figures 6 to 7

[0224] Figure 6 is a schematic block diagram of a data processing device of an embodiment of the present application. As shown in Figure 6 , the device can include a first acquisition module 11, a second acquisition module 12, and a processing module 13.

[0225] The first acquisition module 11 is configured to acquire a query log or a data write completion notification message.

[0226] The second acquisition module 12 is configured to determine that there is new semi-structured data written into a data lake according to the query log, or determine that there is new semi-structured data written into the data lake according to the data write completion notification message, acquire a target data table where the new semi-structured data is located, a first file in the target data table where the new semi-structured data is located, and a target field of the first file to which the new semi-structured data belongs, and the data lake has a plurality of data tables.

[0227] The processing module 13 is configured to determine that the target data table satisfies an optimization condition, and then perform storage space optimization on the target field of the first file in the target data table to obtain an optimized first file.

[0228] The processing module 13 is further configured to write data in the optimized first file into a newly created second file.

[0229] In an embodiment, the second acquisition module 12 is configured to:​

[0230] According to the query stream, access feature information of the data table in the current data lake is acquired, and the data lake includes a plurality of data tables;

[0231] According to the access feature information of the data table, it is determined that new semi-structured data is written into the data lake.

[0232] In an embodiment, the query stream includes query records corresponding to query requests in a first time period, and the second acquisition module 12 is configured to:

[0233] According to the query records corresponding to the query requests in the first time period, an identifier of a data table accessed in the first time period, a field in the data table accessed in the first time period, and an identifier of a file in which the field is located are acquired.

[0234] The identifier of the data table accessed in the first time period, the field in the data table accessed in the first time period, and the identifier of the file in which the field is located are determined as the access feature information of the data table in the current data lake.

[0235] In an embodiment, the second acquisition module 12 is configured to:

[0236] If the field in the data table accessed in the first time period is a semi-structured field, it is determined that new semi-structured data is written.

[0237] In an embodiment, the second acquisition module 12 is specifically configured to:

[0238] The field in the data table accessed in the access feature information is determined as a target field.

[0239] The data table in which the target field is located is determined as a target data table, and the identifier of the file to which the target field belongs is determined as an identifier of a first file.

[0240] In an embodiment, the data write completion notification message indicates that semi-structured data writing is completed, and the second acquisition module 12 is configured to:

[0241] The field to which the new semi-structured data belongs is determined as a target field.

[0242] The data table in which the target field is located is determined as a target data table, and the identifier of the file to which the target field belongs is determined as an identifier of a first file.

[0243] In an embodiment, the processing module 11 is configured to:

[0244] Create a temporary table;

[0245] Sample data of part of the file in the target data table into the temporary table;

[0246] The storage space is optimized by preset processing on data in the temporary table, and optimization index information of the temporary table after storage space optimization is obtained, the optimization index information including time consumption of storage space optimization, improvement rate of storage space and improvement rate of query speed;

[0247] If it is determined according to the optimization index information that the preset index requirement is met, it is determined that the optimization condition is met.

[0248] In an embodiment, the processing module 11 is configured to:

[0249] The semi-structured data of the target field is columned and rewritten to obtain the target field after columned and rewritten;

[0250] The first file after optimization is obtained according to the fields other than the target field in the first file and the target field after columned and rewritten.

[0251] In an embodiment, the processing module 11 is specifically configured to:

[0252] The semi-structured data of the target field is parsed, and the same part of each row subfield is determined according to the parsing result;

[0253] The semi-structured data of the target field is columned and rewritten according to the same part, and the non-same part in the semi-structured data of the target field is written into the encoded field;

[0254] The columned and rewritten field and the encoded field are combined to obtain the target field after columned and rewritten.

[0255] Figure 7 is a schematic block diagram of a data processing apparatus of an embodiment of the present application. As shown in Figure 7 the apparatus of the present embodiment is based on the apparatus shown in Figure 6 further, the apparatus can further include a query module 14.

[0256] The query module 14 is configured to: after receiving a query request, parse the query request to obtain a field to be queried;

[0257] The first field is determined from the field to be queried, and the first field is a subfield of a semi-structured field or all fields of a semi-structured field;

[0258] If it is determined by the reader that the file where the first field is located has been optimized for storage space, the first field is read from the newly created second file;

[0259] If it is determined by the reader that the file where the first field is located has not been optimized for storage space, the first field is read from the first file;

[0260] The query result is obtained according to the first field.

[0261] It should be understood that the device embodiments and the method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, no further description is given here. Specifically, Figure 5 , Figure 6 or Figure 7 The device shown in the above formula can perform the method embodiments described above, and the foregoing and other operations and / or functions of each module in the device are respectively for realizing the corresponding flow in the method embodiments described above, and for the sake of brevity, no further description is given here.

[0262] The device of the embodiments of the present application is described above in conjunction with the functional modules. It should be understood that the functional modules can be realized by hardware, or by instructions in the form of software, or by a combination of hardware and software modules. Specifically, each step of the method embodiments in the embodiments of the present application can be completed by the integrated logic circuit of hardware in the processor and / or instructions in the form of software, and the steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processor execution completion, or executed by a combination of hardware and software modules in the code processor. Alternatively, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory, and combines the hardware to complete the steps in the method embodiments described above.

[0263] Figure 8 is a schematic block diagram of an electronic device 30 provided by the embodiments of the present application.

[0264] As shown in Figure 8 , the electronic device 30 can include:

[0265] The memory 31 and the processor 32, the memory 31 is used to store computer programs and transmit the program code to the processor 32. In other words, the processor 32 can call and run the computer program from the memory 31 to realize the method in the embodiments of the present application.

[0266] For example, the processor 32 can be used to execute the method embodiments described above according to the instructions in the computer program.

[0267] In some embodiments of the present application, the processor 32 can include but is not limited to:

[0268] A general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc.

[0269] In some embodiments of the present application, the memory 31 includes, but is not limited to:

[0270] volatile memory and / or non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM can be used, such as a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a synch link DRAM (SLDRAM), and a Direct Rambus RAM (DR RAM).

[0271] In some embodiments of the present application, the computer program can be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to complete the method provided by the present application. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the electronic device.

[0272] As shown in Figure 8 , the electronic device 30 can further include:

[0273] A transceiver 33, which can be connected to the processor 32 or the memory 31.

[0274] The processor 32 can control the transceiver 33 to communicate with other devices, specifically, can send information or data to other devices, or receive information or data sent by other devices. The transceiver 730 can include a transmitter and a receiver. The transceiver 33 can further include an antenna, and the number of antennas can be one or more.

[0275] It should be understood that various components in the electronic device are connected through a bus system, which includes a data bus, a power supply bus, a control bus, and a state signal bus in addition to a data bus.

[0276] The present application also provides a computer storage medium, which stores a computer program, and the computer program enables the computer to execute the method of the above method embodiments when executed by the computer. Alternatively, the present application embodiment also provides a computer program product containing instructions, which enables the computer to execute the method of the above method embodiments when executed by the computer.

[0277] When implemented by using software, it can be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer program instructions produce the flow or function of the present application embodiment in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media sets. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital video disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0278] It can be understood that in the specific embodiments of the present application, when the above embodiments of the present application are applied to specific products or technologies and related data such as user information, the user's permission or consent needs to be obtained, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards.

[0279] Those skilled in the art can appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0280] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed each other can be indirect coupling or communication connection through some interface, device or module, and can be electrical, mechanical or other forms.

[0281] The modules described as separate components can or can not be physically separated, and the components shown as modules can or can not be physical modules, i.e. they can be located in one place or distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. For example, the functional modules in each embodiment of the present application can be integrated into a processing module, or each module can be physically present separately, or two or more modules can be integrated into one module.

[0282] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data processing method, characterized in that, include: Retrieve query logs or data write completion notification messages; Based on the query flow, it is determined that new semi-structured data has been written to the data lake; or, based on the data write completion notification message, it is determined that new semi-structured data has been written to the data lake. The target data table where the new semi-structured data is located, the first file in the target data table, and the target field of the first file to which it belongs are obtained. There are multiple data tables in the data lake. If the target data table is determined to meet the optimization conditions, then the storage space of the target field of the first file in the target data table is optimized to obtain the optimized first file; Write the data from the optimized first file into a newly created second file.

2. The method according to claim 1, characterized in that, The step of determining that new semi-structured data has been written to the data lake based on the query flow includes: Based on the query flow, access characteristic information of the data tables in the current data lake is obtained, and the data lake includes multiple data tables; Based on the access characteristics of the data table, it is determined that new semi-structured data has been written to the data lake.

3. The method according to claim 2, characterized in that, The query log includes query records corresponding to query requests within a first time period. The step of obtaining access characteristic information of data tables in the current data lake based on the query log includes: Based on the query records corresponding to the query requests within the first time period, obtain the identifiers of the data tables accessed within the first time period, the fields in the accessed data tables, and the identifiers of the files containing the fields; The identifiers of the data tables accessed during the first time period, the fields in the accessed data tables, and the identifiers of the files containing the fields are determined as the access characteristic information of the data tables in the current data lake.

4. The method according to claim 3, characterized in that, The step of determining that new semi-structured data has been written to the data lake based on the access characteristic information of the data table includes: If the fields in the data tables accessed during the first time period are semi-structured fields, then it is determined that new semi-structured data has been written to the data lake.

5. The method according to claim 3, characterized in that, The step of obtaining the target data table containing the new semi-structured data, the first file in the target data table, and the target fields of the first file includes: The fields in the accessed data tables that are semi-structured fields are identified as the target fields. The data table containing the target field is determined as the target data table, and the identifier of the file to which the target field belongs is determined as the identifier of the first file.

6. The method according to claim 1, characterized in that, The data write completion notification message indicates that the semi-structured data write is complete. The step of obtaining the target data table containing the new semi-structured data, the first file within the target data table, and the target fields of the first file includes: The field to which the new semi-structured data belongs is determined as the target field; The data table containing the target field is determined as the target data table, and the identifier of the file to which the target field belongs is determined as the identifier of the first file.

7. The method according to any one of claims 1-6, characterized in that, Determining that the target data table meets the optimization conditions includes: Create a temporary table; Data from a portion of the files sampled from the target data table is stored in the temporary table; The storage space is optimized by performing preset processing on the data in the temporary table, and the optimization index information of the temporary table after storage space optimization is obtained. The optimization index information includes the time consumed by storage space optimization, the storage space improvement rate, and the query speed improvement rate. If the optimization index information satisfies at least one of the following, then the target data table is determined to meet the optimization conditions: The time taken for storage space optimization is less than a preset time, the improvement rate of storage space is greater than a preset threshold, and the improvement rate of query speed is greater than a preset value.

8. The method according to any one of claims 1-6, characterized in that, The step of optimizing the storage space of the target field of the first file in the target data table to obtain the optimized first file includes: The semi-structured data of the target field is split and rewritten to obtain the target field after splitting and rewriting. The optimized first file is obtained based on the fields in the first file other than the target field and the target field after splitting and rewriting.

9. The method according to claim 8, characterized in that, The step of splitting and rewriting the semi-structured data of the target field to obtain the split and rewritten target field includes: The semi-structured data of the target field is parsed, and the common parts of each row of sub-fields are determined based on the parsing results; The semi-structured data of the target field is rewritten column by column according to the same part, and the non-identical part of the semi-structured data of the target field is written into the encoded field. The column-rewritten fields are combined with the encoded fields to obtain the target fields after column splitting and rewriting.

10. The method according to any one of claims 1-6, characterized in that, After writing the optimized data from the first file into the newly created second file, the method further includes: Upon receiving a query request, the query request is parsed to obtain the fields to be queried; The first field is determined from the fields to be queried, where the first field is either a subfield of a semi-structured field or all fields of a semi-structured field. If the reader determines that the file containing the first field has undergone storage space optimization, then the first field is read from the newly created second file; If the reader determines that the file containing the first field has not undergone storage space optimization, then the first field is read from the first file. The query results are obtained based on the first field.

11. A data processing apparatus, characterized in that, include: The first acquisition module is used to acquire query logs or data write completion notification messages; The second acquisition module is used to determine, based on the query flow, that new semi-structured data has been written to the data lake, or, based on the data writing completion notification message, that new semi-structured data has been written to the data lake, and to acquire the target data table where the new semi-structured data is located, the first file in the target data table, and the target field of the first file to which it belongs. The data lake contains multiple data tables. The processing module is used to determine if the target data table meets the optimization conditions, and then optimize the storage space of the target field of the first file in the target data table to obtain the optimized first file; The processing module is also used to: write the data from the optimized first file into a newly created second file.

12. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores instructions, and when the processor executes the instructions, it causes the processor to perform the method according to any one of claims 1-10.

13. A computer storage medium, characterized in that, Used to store a computer program, the computer program including a method for performing any one of claims 1-10.

14. A computer program product, characterized in that, It includes computer program code that, when executed by an electronic device, causes the electronic device to perform the method of any one of claims 1-10.

Images