Data processing method, device, system and storage medium based on data lake

Abstract

A data processing method, device, system and storage medium based on a data lake are disclosed, wherein the method includes: obtaining source data to be stored in the data lake; in response to the source data being unstructured data, parsing the unstructured data to determine label data corresponding to the unstructured data; binarizing the unstructured data to obtain binary data corresponding to the source data; storing the binary data and the label data in the data lake. The disclosed embodiment realizes the unified storage of unstructured data and label data in the data lake, which can improve the retrieval speed and reading speed of the stored unstructured data and ensure the use of the unstructured data stored in the data lake.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing method, device, system and storage medium based on a data lake. Background Art

[0002] In the process of autonomous driving, such as assisted driving and unmanned driving, the autonomous driving data collected by cameras, lidar and other equipment plays an indispensable role. For example, it assists autonomous driving vehicles in judging road conditions. Therefore, the storage and retrieval of autonomous driving data is very important.

[0003] In the related technology, unstructured autonomous driving data is stored in different directories of different storage spaces (buckets) of object storage according to their types. As the amount of autonomous driving data increases, the number of buckets and directory levels also increase, making data retrieval more difficult. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure provides a data processing method, device, system and storage medium based on a data lake. By storing unstructured data and corresponding label data in a data lake, it helps to retrieve unstructured data through label data, reduces the difficulty of retrieving unstructured data, and improves the retrieval efficiency of unstructured data.

[0005] A first aspect of the present disclosure provides a data processing method based on a data lake, comprising: acquiring source data to be stored in a data lake; in response to the source data being unstructured data, parsing the unstructured data to determine label data corresponding to the unstructured data; binarizing the unstructured data to obtain binary data corresponding to the source data; and storing the binary data and the label data in the data lake.

[0006] A second aspect of the present disclosure provides a data processing device based on a data lake, including: an acquisition unit, used to acquire source data to be stored in the data lake; a label generation unit, used to parse the unstructured data in response to the source data being unstructured data, and determine label data corresponding to the unstructured data; a binarization unit, used to binarize the unstructured data to obtain binary data corresponding to the source data; and a storage unit, used to store the binary data and the label data in the data lake.

[0007] A third aspect of the present disclosure provides a data management system, including a service layer, a gateway layer and a storage layer; the gateway layer receives source data sent by a user and stored in a data lake; the service layer obtains the source data to be stored in the data lake; in response to the source data being unstructured data, the unstructured data is parsed and processed to determine label data corresponding to the unstructured data; the unstructured data is binarized to obtain binary data corresponding to the source data; the binary data and the label data are sent to the storage layer; the storage layer is used to store the binary data and the label data.

[0008] A fourth aspect of the present disclosure provides a computer-readable storage medium, which stores computer program instructions. When the computer program instructions are executed, the data processing method based on the data lake proposed in the embodiment of the first aspect of the present disclosure is implemented.

[0009] Based on the data processing method, device, system and storage medium based on the data lake provided by the embodiments of the present disclosure, when the source data needs to be stored in the data lake, if the source data is unstructured data, such as images, videos, point clouds and other data, the unstructured data can be first parsed and binarized to obtain the label data and binary data corresponding to the source data, and the label data and binary data are stored in the data lake. Therefore, by converting the unstructured data into binary data and obtaining the label data of the unstructured data, the unstructured data and the label data are uniformly stored in the data lake, which helps to retrieve the unstructured data through the label data, without the need to retrieve the unstructured data in different directories of the storage space, thereby improving the retrieval speed and reading speed of the unstructured data and accelerating the response speed of using the unstructured data. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is an exemplary application scenario diagram of data processing based on a data lake according to the present disclosure;

[0011] Figure 2 is a flowchart of a data processing method based on a data lake provided by an exemplary embodiment of the present disclosure;

[0012] Figure 3 is a flowchart of step 202 provided by an exemplary embodiment of the present disclosure;

[0013] Figure 4 is a flowchart of storing binary data and label data in a data lake provided by an exemplary embodiment of the present disclosure;

[0014] Figure 5 is a flowchart of step 402 provided by an exemplary embodiment of the present disclosure;

[0015] Figure 6 is a schematic diagram of a query process for unstructured data provided by an exemplary embodiment of the present disclosure;

[0016] Figure 7 is a query schematic diagram of unstructured data provided by an exemplary embodiment of the present disclosure;

[0017] Figure 8 is a structural diagram of a data management system provided by an exemplary embodiment of the present disclosure;

[0018] Fig. 9 is a structural diagram of a data processing device based on a data lake provided by an exemplary embodiment of the present disclosure;

[0019] Fig.10 is a structural schematic diagram of a data processing device based on a data lake provided by another exemplary embodiment of the present disclosure;

[0020] Fig.11 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0022] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0023] SUMMARY OF THE DISCLOSURE

[0024] In the process of realizing the present disclosure, the inventors have found through research that in the field of autonomous driving, the amount of unstructured data collected by a large number of cameras, laser radars and other equipment is growing, and the difficulty of storing and managing the unstructured data is also increasing. In the related art, the unstructured autonomous driving data is stored in different directories of different storage spaces (buckets) of the object storage according to the type, but with the increase in the amount of autonomous driving data, the number and directory levels of data stored in this way are also increasing, and the difficulty of data retrieval and management is increasing.

[0025] In order to reduce the difficulty of retrieving unstructured data, the inventor proposed the technical solution disclosed in the present invention.

[0026] Exemplary Systems

[0027] Figure 1FIG. 1 is an exemplary application scenario diagram of data processing based on a data lake according to the present disclosure. Figure 1 As shown, each unstructured data collection device 11, such as a camera 111 and a laser radar 112, can collect surrounding environment data in real time, and store the collected data in a data lake system 13 through a network 12; or other storage systems 14, such as a relational database, a data warehouse, a distributed database (such as a Mango database), and source data in an analytical database can also be stored in the data lake system 13 through a network 12 or a wired connection. When storing the source data in the data lake system 13, the data processing method based on the data lake disclosed in the present invention can be used to store the unstructured data and its corresponding label data in the data lake system 13.

[0028] The data processing method based on the data lake disclosed in the present invention can parse and obtain the label data of each unstructured data according to the data type of the unstructured data collected by different devices, and by binarizing the unstructured data, the binary data of the unstructured data can be obtained, thereby realizing the unified storage of the binary data and label data of the unstructured data in the data lake system 13. The present invention facilitates the retrieval of unstructured data through label data by storing the unstructured data in a structured manner, improves the retrieval speed and reading speed of the unstructured data, and accelerates the response speed of using the unstructured data.

[0029] In the present disclosure, label data is data used to describe unstructured data. The data lake system can effectively manage and maintain unstructured data based on label data, which is convenient for subsequent query and management. The data lake system may include a data processing server 131, through which the unstructured data collected by each collection device can be parsed to obtain corresponding label data, and the unstructured data can be binarized to obtain corresponding binary data, and then the label data and binary data can be written into the data table of the data lake respectively.

[0030] The collection device 11 is used to indicate a device installed at each traffic intersection or traffic road for collecting surrounding environment data. In addition to the camera device 111 and the laser radar 112, the collection device 11 can also be other sensor devices capable of collecting unstructured data. The present disclosure does not limit the type, quantity, installation location, and viewing angle range of the collection device.

[0031] Exemplary Methods

[0032] Figure 2 is a flowchart of a data processing method based on a data lake provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 2 As shown, the following steps are included:

[0033] Step 201: Obtain source data to be stored in the data lake.

[0034] Among them, a data lake is a storage system that can store data of various data types and formats. The data lake is used to store structured data, such as tables in relational databases, and also to store unstructured data, such as image data, point cloud data, document data, email data, report data, etc.

[0035] In the present disclosure, source data refers to the original data collected by the device without being processed or simplified, and may include point cloud data, image data, video data, etc.

[0036] In some implementations, source data that has been stored in other storage systems can be obtained, such as obtaining source data to be stored in the data lake from data sources such as relational databases, data warehouses, distributed databases (such as Mango databases), and analytical databases (such as Doris databases).

[0037] In other optional implementations, the source data to be stored in the data lake may also be obtained from various data collection devices, such as cameras, radar equipment, and the like.

[0038] Step 202: In response to the source data being unstructured data, the unstructured data is parsed and processed to determine label data corresponding to the unstructured data.

[0039] Among them, unstructured data is data with irregular or incomplete structure, no predefined data model, and is not easy to express using database binary logic.

[0040] In this embodiment, when obtaining label data of unstructured data, the type label of the unstructured data can be determined based on the data type of the unstructured data, such as image, video, text, point cloud; and the unstructured data can be parsed to obtain a content label that can characterize the content of the unstructured data, and the label data of the unstructured data can be obtained by combining the above-mentioned type label and content label.

[0041] In specific implementation, for unstructured data of different data types, corresponding deep learning models can be used to learn content labels that can characterize the content of the unstructured data. For example, the content label of an image can be identified through an image recognition model, and the label data can reflect the key features of the essential content of the image data.

[0042] Step 203, binarization processing is performed on the unstructured data to obtain binary data corresponding to the source data.

[0043] Among them, the binarization of unstructured data refers to the process of converting unstructured data into binary form, which helps to simplify the data structure and facilitate storage and subsequent processing. For example, each pixel of an image can be converted to 0 or 1 to achieve the binarization of image data.

[0044] In this embodiment, the binarization of unstructured data can be implemented by a specific binarization algorithm or tool. For example, the binarization of the image is implemented by the image processing tool PIL (Python Image Library) or OpenCV library in Python, and the image is converted into binary data. For the binarization of video data, the video data can be first segmented into separate video frames or short video clips, and then the segmented video frames or short video clips are binarized according to the image binarization algorithm.

[0045] Step 204: Store the binary data and label data in the data lake.

[0046] Among them, binary data can be stored in the data table of the data lake, and label data can be stored in the label table of the data lake.

[0047] In specific implementation, the label data and the data identification information corresponding to the label data can be stored in the label table, and the data identification information and the binary data corresponding to the data identification information can be stored in the data table. As a result, the label table and the data table have the same field "data identification information". Based on the field "data identification information", the association between the label data and the binary data can be achieved.

[0048] In some implementations, when storing label data in the data lake, the corresponding label data can be stored in the label table of the corresponding business type based on the business type of the unstructured data; in other implementations, the corresponding label data can also be stored in the label table of the corresponding data type based on the data type of the unstructured data. By storing label data in different ways, the retrieval speed and reading speed of the label data of the stored unstructured data can be improved, the use of the unstructured data stored in the data lake can be guaranteed, and the response speed when using the unstructured data can be accelerated.

[0049] In some implementations, when storing binary data in a data lake, the binary data can be stored in an Iceberg format table. The Iceberg format is an open table format for large analytical data that can support data operations on multiple storage backends, facilitating scalable, efficient, and secure data storage and query solutions.

[0050] In some implementations, when storing binary data in a data table, the binary data can be stored in a data partition or sub-partition of the data table corresponding to the device identifier corresponding to the unstructured data, thereby storing data collected by the same device in the same partition, further improving the retrieval speed and reading speed of the unstructured data and ensuring the use of the unstructured data.

[0051] In other implementations, when storing binary data in a data table, the binary data may also be stored in a corresponding partition or sub-partition of the data table according to the acquisition time of the binary data.

[0052] The above solution provided by this embodiment, when it is necessary to store the source data in the data lake, if the source data is unstructured data, such as images, videos, point clouds and other data, the unstructured data can be first parsed and binarized to obtain the label data and binary data corresponding to the source data, and the label data and binary data are stored in the data lake in different ways. Therefore, by converting the unstructured data into binary data and obtaining the label data of the unstructured data, the unstructured data and the label data are uniformly stored in the data lake, which can improve the retrieval speed and reading speed of the stored unstructured data, ensure the use of the unstructured data stored in the data lake, and speed up the response speed when using the unstructured data.

[0053] like Figure 3 As shown in the above Figure 2 Based on the illustrated embodiment, step 202 may include the following steps:

[0054] Step 221, determine the data type of the unstructured data.

[0055] Among them, the data types of unstructured data may include various types such as images, videos, point clouds, documents, reports, etc.

[0056] In this embodiment, unstructured data mainly refers to the surrounding environment data collected by each collection device, so the data type may include image data, video data, point cloud data and other types.

[0057] Step 222, based on the data type, determine a parsing method for parsing unstructured data of the data type.

[0058] In some implementations, in response to the unstructured data being image data, the image data is recognized based on an image recognition model.

[0059] The image recognition model is a deep learning model obtained by learning and analyzing a large amount of image data, which can recognize the input image and determine the label data of the image. For example, the image recognition model is a Visual Geometry Group (VGG) model, a Residual Neural Network (ResNet) model, etc.

[0060] In other implementations, in response to the unstructured data being video data, the video data may be segmented according to a preset segmentation rule to obtain at least one video segment data, and each video segment data may be parsed and processed respectively.

[0061] In this embodiment, the preset segmentation rule refers to a rule for segmenting video data into different parts. For example, the preset segmentation rule is to segment every five frames of data into a video segment. Video data collected by acquisition devices at different locations can be segmented according to different preset segmentation rules. For example, the segmentation rule for video data collected by acquisition devices at important traffic intersections is to segment every five frames of data into a video segment, while the segmentation rule for video data collected by acquisition devices on traffic roads with less traffic is to segment every 10 frames of data into a video segment. Using different segmentation rules for video data collected by different acquisition devices helps to minimize the amount of calculation while ensuring that the label data of different videos can be identified.

[0062] After the video data is divided into different video segments, each video segment can be analyzed and processed through a multimodal large model or a video content extraction model.

[0063] In some other implementations, in response to the unstructured data being point cloud data, the point cloud data is identified based on a point cloud semantic segmentation model.

[0064] Among them, the point cloud semantic segmentation model is a three-dimensional data processing model based on deep learning. It can divide point cloud data into areas with different semantic labels. By analyzing the spatial geometric structure and shape information inside the point cloud, it can automatically identify and classify different objects and surface types in the three-dimensional scene to obtain label data of the point cloud data.

[0065] In this implementation, different models or methods are used to determine label data for different data types.

[0066] Step 223: Determine the label data corresponding to the unstructured data based on the parsing method.

[0067] In this embodiment, the label data of the unstructured data can be identified through the various models and methods determined in step 222.

[0068] The technical solution provided in this embodiment adopts corresponding parsing methods for unstructured data of different data types to obtain corresponding label data, thereby achieving accurate extraction of label data and facilitating subsequent accurate retrieval of required unstructured data.

[0069] Figure 4 FIG. 1 is a flowchart of storing binary data and label data in a data lake according to an exemplary embodiment of the present disclosure. Figure 4 As shown, the following steps are included:

[0070] Step 401, determining data identification information of the unstructured data based on the collection time and collection device of the unstructured data.

[0071] Among them, when it is necessary to store unstructured data in the data lake system, in order to accurately locate each unstructured data, a data identification information can also be generated based on the collection time of the unstructured data, the device identification information of the collection device, etc. The data identification information is the unique identification information of the unstructured data. After generating each data identification information and label data, an association relationship between the label data and the data identification information can be established. One label data can correspond to one data identification information or multiple data identification information.

[0072] For example, there are currently 12 cameras collecting unstructured data, and the data collected by the 12 cameras need to be stored in the data lake system. Then, for the data collected by these 12 cameras, data identification information of each data can be generated based on the device identification information, data collection time information, etc.

[0073] It is understandable that when a video captured by the same device is stored in segments, the data identification information can be determined according to the device identification corresponding to each video segment, the video capture time, and the sequence number of the video segment.

[0074] Further, after determining the data identification information of each unstructured data, step 402 and step 403 may be performed respectively to store the binary data and label data corresponding to the unstructured data.

[0075] Step 402: Store the binary data and the corresponding data identification information in a data table in the data lake.

[0076] In this embodiment, based on the collection devices corresponding to the unstructured data, the unstructured data collected by different collection devices can be stored in different data partitions or sub-partitions of the data table, so that the data collected by the same device can be stored in the same partition, further improving the retrieval speed and reading speed of the unstructured data and ensuring the use of the unstructured data.

[0077] In a specific implementation, when storing binary data of unstructured data in each data partition, the data may be sequentially stored in each partition in the order of collection time.

[0078] Step 403: Store the label data and the corresponding data identification information in the label table of the data lake.

[0079] In this embodiment, when storing label data in the data lake, the corresponding label data can be stored in the label table of the corresponding business type based on the business type of the unstructured data; in other implementations, the corresponding label data can also be stored in the label table of the corresponding data type based on the data type of the unstructured data.

[0080] When storing, the tag data of each unstructured data and the data identification information corresponding to the tag data can be stored in the tag table, so that when the tag data is used to retrieve the unstructured data, the data identification information corresponding to the tag data can be obtained in the tag table. Among them, one tag data can correspond to multiple data identification information, but one data identification information can only correspond to one tag data.

[0081] The technical solution provided in this embodiment helps to accurately locate the unstructured data according to the label data and data identification information by generating data identification information for each unstructured data, and storing the data identification information and the binary data in the data table, and storing the data identification information and the label data in the label table. In addition, by classifying and storing the label data in different ways, and storing the unstructured data collected by different devices in different partitions, it helps to improve the retrieval speed and reading speed of the label data of the stored unstructured data, ensure the use of the unstructured data stored in the data lake, and speed up the response speed when using the unstructured data.

[0082] like Figure 5 As shown in the above Figure 4 Based on the illustrated embodiment, step 402 may include the following steps:

[0083] Step 421, obtaining the target storage partition corresponding to the unstructured data collection device.

[0084] In some implementations, the mapping relationship between each acquisition device and the storage partition can be pre-selected and stored, for example, the mapping relationship between each acquisition device and the storage partition is stored in a relational data table. When it is necessary to store unstructured data, the corresponding target storage partition can be determined based on the acquisition device corresponding to the unstructured data to be stored.

[0085] In specific implementation, the efficiency of query and maintenance is improved by dividing a large data table into multiple small tables (i.e. partitions), each small table is called a partition.

[0086] Step 422: Store the binary data and the corresponding data identification information in a data table in the target storage partition.

[0087] The technical solution provided by this embodiment helps to improve the efficiency of querying and maintaining data tables by storing unstructured data in corresponding target storage partitions. The amount of data stored in the data lake disclosed in this disclosure is very large, and using partitioned storage is more helpful to improve the retrieval efficiency of unstructured data.

[0088] Figure 6 is a schematic diagram of a query process of unstructured data provided by an exemplary embodiment of the present disclosure, such as Figure 6 As shown, the following steps are included:

[0089] Step 601, in response to receiving a query request sent by a user, parsing the query request, obtaining information of a data table to be queried and a query condition, wherein the query condition includes label data corresponding to the unstructured data.

[0090] The query request is used to indicate the operation of obtaining data from a data table by setting some query conditions and displaying the data in a table for the user to browse. In this embodiment, the query request is a structured query language (SQL) request.

[0091] In this embodiment, the data lake system supports multiple users or organizations to share resources, and multiple tenants can use different engines to access and use the data lake system, such as Doris, Spark, Trino Catalog engine, etc. After receiving query requests from different tenants, the target engine corresponding to the query request can be determined, and then the query request is parsed by the target engine to obtain the information and query conditions of the data table to be queried.

[0092] The information of the data table to be queried is used to indicate the identification information of the data table to be retrieved by the query request. The query condition is a key element used to filter and limit the query results.

[0093] In the present disclosure, the query condition includes the label data of the binary data to be retrieved, such as retrieving the data of the location of a traffic light, where "traffic light" is the label data; the query condition can also be a combination condition of two or more basic query conditions connected by logical operators (such as AND, OR, NOT), and the combination condition includes at least the label data.

[0094] Step 602: Obtain data identification information corresponding to the tag data in the tag table.

[0095] Step 603: Obtain binary data corresponding to the data identification information in the data table to be queried.

[0096] Step 604: convert the binary data into corresponding unstructured data.

[0097] In this embodiment, after obtaining binary data from the data table, the binary data can be converted back to unstructured data through a set conversion algorithm. For example, the image processing tool PIL (Python Image Library) in Python or the OpenCV library is used to convert the binary data back to image data.

[0098] In some implementations, if the binary data corresponds to video clips, after converting the binary data back into video clips, the video clips need to be further spliced ​​according to the sequence numbers or time information of the video clips to obtain a complete video data.

[0099] For example, see Figure 7 , which illustrates the query process. After receiving the query request, the query request is forwarded to the corresponding target engine, such as Doris, Spark, and Trino Catalog engines. The target engine then obtains the data identification information corresponding to the label data in the label table, and then obtains the binary data corresponding to the data identification information in the query data table.

[0100] The technical solution provided by this embodiment discloses an implementation method for retrieving unstructured data in a data lake. When receiving a query request sent by a user, the query request is parsed to obtain information and query conditions of the data table to be queried, and then the data identification information corresponding to the label data in the query condition is obtained in the label table, and then the binary data corresponding to the data identification information can be obtained in the data table to be queried, and the binary data is converted into corresponding unstructured data. The disclosed embodiment accurately retrieves unstructured data based on the label data and data identification information, realizes rapid retrieval and structured retrieval of unstructured data, and improves retrieval efficiency.

[0101] Figure 8 is a structural diagram of a data management system provided by an exemplary embodiment of the present disclosure, such as Figure 8 As shown, the data management system includes a data source 81 and a data lake system 82. The data source 81 includes various data storage systems, data acquisition devices ( Figure 8The data lake system 82 includes: a service layer 821, a gateway layer 822, and a storage layer 823. The service layer 821 is used to receive source data sent by users and stored in the data lake; the gateway layer 822 is used to obtain source data to be stored in the data lake; in response to the source data being unstructured data, the unstructured data is parsed and processed to determine the label data corresponding to the unstructured data; the unstructured data is binarized to obtain binary data corresponding to the source data; the binary data and the label data are sent to the storage layer 823; the storage layer 823 is used to store the binary data and the label data.

[0102] Among them, the data lake system 82 also includes an engine layer 824. After the gateway layer 823 obtains the data to be written or the query request to be queried, it can forward the query request and the data to be written to the corresponding engine for processing according to the engine used by the tenant.

[0103] like Figure 8 As shown, the service layer 821 includes various interfaces, a software development kit (SDK), and a data lake service module. Among them, the data lake service module can provide services for adding, deleting, and modifying data tables and functions for controlling the authority of data tables, and can also provide the function of entering the source data to be stored into the lake through the interface; the interface provided by the service layer 821 can facilitate users to access the system and submit data write requests and query requests to the data lake. The gateway layer 822 can use Kyuubi as a gateway and support multiple tenants to use different engines to access and use the data lake system through Java Database Connectivity (JDBC). The engine layer 824 can provide different engines so that different tenants can choose the appropriate engine according to their own usage scenarios to meet different needs. For example, the Spark engine helps large-scale distributed computing, the Trino engine helps small-scale ad hoc queries, and the Doris engine supports stream reading, which can meet the continuous return of large-scale binary data. The storage layer 823 is used to implement the storage of binary data and label data. The storage layer 823 supports partitioned storage of data and stores the data in an Iceberg format table and in an object storage service (OSS), a Hadoop compatible file system (HCFS), etc.

[0104] Specifically, the service layer 821 also provides permission management services, data entry services, and data management services.

[0105] The permission management service is used to provide permission control for the data lake and data tables, so that users can only read and write to the data lake and data tables within their own permission scope. In this disclosure, a unified authorization management framework (Apache Ranger) can be used to develop corresponding permission management services to complete the management of different users, different roles, and different resources, and ensure the security of data in the data lake.

[0106] The data entry service is used to provide data entry services for multiple data sources. In this disclosure, it can support source data stored in other storage systems, such as obtaining data to be stored in the data lake from data sources such as relational databases, data warehouses, distributed databases (such as Mango databases), analytical databases (such as Doris databases), etc. When storing data from data sources into the data lake, the data from multiple data sources can be imported into the data lake by calling the interface provided by the data entry service, and the unstructured data and corresponding label data therein can be stored in the data lake in the manner described in the above implementation.

[0107] The data management service is used to provide management services for the data in the data lake, such as regularly deleting orphan files, managing the life cycle of data in the data lake, and clearing expired data.

[0108] The gateway layer 822 uses Kyuubi as the gateway, which not only supports multiple tenants using different engines (such as Spark engine, Trino engine, Doris engine) to access and use the data lake system through JDBC, but also supports different levels of engine session isolation strategies. Specifically, you can set different sharing levels for the engine by configuring the parameter kyuubi.engine.share.level in the kyuubi-defaults.conf tab of the Kyuubi service configuration page.

[0109] The gateway layer 822 supports multiple tenants to use different engines to access and use the data lake system. After receiving query requests from different tenants, it can determine the target engine corresponding to the query request, such as Doris, Spark, and TrinoCatalog engines. The query request is then parsed by the target engine to obtain the information and query conditions of the data table to be queried. Then, the data identification information corresponding to the label data is obtained in the label table, and then the binary data corresponding to the data identification information is obtained in the query data table, so as to realize the data retrieval method supporting different search engines.

[0110] The engine layer 824 provides multi-engine support so that different tenants can choose the appropriate engine according to their own usage scenarios to meet different needs. For example, the Spark engine helps large-scale distributed computing and is suitable for data analysis, machine learning, and graphics processing; the Trino engine helps small-scale ad hoc queries and is used to connect to a variety of data sources, including relational databases, object storage services, etc.; and the Doris engine supports streaming reading and can meet the continuous return of large-scale binary data. The multi-engine support provided by the engine layer helps each tenant to choose the appropriate engine for the data lake according to their main needs. Through the above-mentioned data management system, the storage, query, and management of unstructured data can be realized, and the use of unstructured data stored in the data lake can be guaranteed, which speeds up the response speed when using unstructured data.

[0111] Exemplary Devices

[0112] Fig. 9 is a structural diagram of a data processing device based on a data lake provided by an exemplary embodiment of the present disclosure, such as Fig. 9 As shown, the device may include:

[0113] An acquisition unit 91 is used to acquire source data to be stored in the data lake;

[0114] The label generating unit 92 is used for analyzing the unstructured data in response to the source data being unstructured data, and determining label data corresponding to the unstructured data;

[0115] A binarization processing unit 93 is used to perform binarization processing on the unstructured data to obtain binary data corresponding to the source data;

[0116] The storage unit 94 is used to store the binary data and the label data in the data lake.

[0117] Fig.10 is a structural diagram of a data processing device based on a data lake provided by another exemplary embodiment of the present disclosure, such as Fig.10 As shown in the above Fig. 9 Based on the illustrated embodiment, in some embodiments, the label generation unit 92 includes:

[0118] The first determination subunit 921 is used to determine the data type of the unstructured data; the second determination subunit 922 is used to determine the parsing method for parsing the unstructured data of the data type based on the data type; the third determination subunit 923 is used to determine the label data corresponding to the unstructured data based on the parsing method.

[0119] In some embodiments, the data type includes image data, video data, or point cloud data;

[0120] The second determination subunit 922 is specifically used to: in response to the unstructured data being image data, identify the image data based on an image recognition model; in response to the unstructured data being video data, segment the video data according to a preset segmentation rule to obtain at least one video segment data, and parse and process each video segment data respectively; in response to the unstructured data being point cloud data, identify the point cloud data based on a point cloud semantic segmentation model.

[0121] In some embodiments, the apparatus further includes: an identification determination unit 95, configured to determine data identification information of the unstructured data based on the collection time and collection device of the unstructured data.

[0122] In some embodiments, the storage unit 94 includes: a first storage subunit 941, used to store binary data and corresponding data identification information in a data table of the data lake; a second storage subunit 942, used to store label data and corresponding data identification information in a label table of the data lake.

[0123] In some embodiments, the first storage subunit 941 is specifically used to: obtain a target storage partition corresponding to the unstructured data collection device; and store the binary data and the corresponding data identification information in a data table in the target storage partition.

[0124] In some embodiments, it also includes:

[0125] The parsing unit 96 is used to parse the query request in response to receiving the query request sent by the user, and obtain the information of the data table to be queried and the query condition, wherein the query condition includes the label data corresponding to the unstructured data;

[0126] The identification acquisition unit 97 is used to obtain data identification information corresponding to the tag data in the tag table;

[0127] A query unit 98, used to obtain binary data corresponding to the data identification information in the data table to be queried;

[0128] The conversion unit 99 is used to convert the binary data into corresponding unstructured data.

[0129] It should be pointed out that the modules in the present device can be decomposed and / or reassembled, and such decomposition and / or reassembly should be regarded as equivalent solutions of the present device.

[0130] The device in the exemplary embodiment of this device corresponds to the above exemplary method, and the relevant contents can be referenced and quoted to each other. The beneficial technical effects corresponding to the exemplary embodiment of this device can refer to the corresponding beneficial technical effects of the above exemplary method, which will not be repeated here.

[0131] Exemplary Electronic Devices

[0132] Fig.11 A structural diagram of an electronic device provided in an embodiment of the present disclosure includes at least one processor 111 and a memory 112.

[0133] The processor 111 is a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and controls other components in the electronic device 11 to perform desired functions.

[0134] The memory 112 includes one or more computer program products, and the computer program products include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory includes, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory includes, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions are stored on the computer-readable storage medium, and the processor 111 runs one or more computer program instructions to implement the data lake-based data processing method and / or other desired functions of the various embodiments of the present disclosure described above.

[0135] In one example, the electronic device 11 further includes: an input device 113 and an output device 114 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0136] The input device 113 also includes, for example, a keyboard, a mouse, and the like.

[0137] The output device 114 outputs various information to the outside, and includes, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0138] Of course, to simplify, Fig.11 Only some of the components related to the present disclosure in the electronic device 11 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 11 also includes any other appropriate components.

[0139] Exemplary computer program products and computer-readable storage media

[0140] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure also provides a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the data processing method based on a data lake of various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section.

[0141] The computer program product is written in any combination of one or more programming languages ​​to perform program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code is executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0142] In addition, an embodiment of the present disclosure is also a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the data processing method based on the data lake of various embodiments of the present disclosure described in the above "Exemplary Method" section.

[0143] Computer readable storage media adopts any combination of one or more readable media. The readable medium is a readable signal medium or a readable storage medium. The readable storage medium is, for example, but not limited to, a system, device or device including electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0144] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and cannot be considered as necessary for each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to being implemented by adopting the above specific details.

[0145] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present disclosure claims and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A data processing method based on a data lake, comprising: Get the source data to be stored in the data lake; In response to the source data being unstructured data, parsing the unstructured data to determine label data corresponding to the unstructured data; Binarization is performed on the unstructured data to obtain binary data corresponding to the source data; The binary data and the label data are stored in the data lake.

2. The method according to claim 1, wherein: The parsing of the unstructured data to determine label data corresponding to the unstructured data includes: Determining a data type of the unstructured data; Based on the data type, determining a parsing method for parsing unstructured data of the data type; Based on the parsing method, label data corresponding to the unstructured data is determined.

3. The method according to claim 2, wherein: The data type includes image data, video data or point cloud data; The step of determining a parsing method for parsing unstructured data of the data type based on the data type includes: In response to the unstructured data being image data, identifying the image data based on an image recognition model; In response to the unstructured data being video data, segmenting the video data according to a preset segmentation rule to obtain at least one video segment data, and performing analysis processing on each video segment data respectively; In response to the unstructured data being point cloud data, the point cloud data is identified based on a point cloud semantic segmentation model.

4. The method according to claim 1, further comprising: Based on the collection time and collection device of the unstructured data, data identification information of the unstructured data is determined.

5. The method according to claim 4, wherein: Storing the binary data and the label data in the data lake includes: storing the binary data and corresponding data identification information in a data table of the data lake; The label data and corresponding data identification information are stored in a label table of the data lake.

6. The method according to claim 5, wherein: The step of storing the binary data and the corresponding data identification information in a data table of the data lake includes: Acquire a target storage partition corresponding to the collection device of the unstructured data; The binary data and corresponding data identification information are stored in a data table in the target storage partition.

7. The method according to claim 5, wherein: Also includes: In response to receiving a query request sent by a user, parsing the query request to obtain information of a data table to be queried and a query condition, wherein the query condition includes label data corresponding to the unstructured data; Acquire data identification information corresponding to the tag data in the tag table; Acquire binary data corresponding to the data identification information in the to-be-queried data table; The binary data is converted into corresponding unstructured data.

8. A data processing device based on a data lake, comprising: An acquisition unit, used to acquire source data to be stored in the data lake; a label generating unit, configured to, in response to the source data being unstructured data, parse the unstructured data and determine label data corresponding to the unstructured data; A binarization processing unit, used for performing binarization processing on the unstructured data to obtain binary data corresponding to the source data; A storage unit is used to store the binary data and the label data in the data lake.

9. A data management system, comprising a service layer, a gateway layer and a storage layer; The service layer receives source data sent by users and stored in the data lake; The gateway layer obtains source data to be stored in the data lake; In response to the source data being unstructured data, the unstructured data is parsed to determine label data corresponding to the unstructured data; the unstructured data is binarized to obtain binary data corresponding to the source data; Sending the binary data and the tag data to the storage layer; The storage layer is used to store the binary data and the label data.

10. A computer-readable storage medium, wherein the storage medium stores computer program instructions, and when the computer program instructions are executed, the data processing method based on the data lake described in any one of claims 1 to 7 is implemented.

Images